Skip to content

AI crawler check:Can ChatGPT, Claude and Perplexity read and cite your site?

See what your robots.txt tells each of 18 AI crawlers, whether your llms.txt is valid, and whether your content is readable without JavaScript.

  • Free, no account
  • Results in under a minute
  • 18 AI crawlers, grouped by purpose
Example result

example-shop.hr

WordPress 6.8

Sample shop · homepage · mobile Chrome · the same check you get for your own site

Run it on your site
SEO

9 passed · 0 failed · 1 warning. Biggest issue: Canonical tag is missing.

AI crawlers in robots.txt

18 known crawlers · 2 blocked · 16 allowed
Each known AI crawler with its operator, purpose and whether robots.txt allows it
CrawlerOperatorPurposerobots.txtRule
GPTBotOpenAImodel trainingblockedits own group, line 10
OAI-SearchBotOpenAIAI search indexallowedinherits User-agent: *
ChatGPT-UserOpenAIfetches for a userallowedinherits User-agent: *
ClaudeBotAnthropicmodel trainingallowedinherits User-agent: *
Claude-SearchBotAnthropicAI search indexallowedinherits User-agent: *
Claude-UserAnthropicfetches for a userallowedinherits User-agent: *
Google-ExtendedGooglemodel trainingallowedinherits User-agent: *
PerplexityBotPerplexityAI search indexallowedinherits User-agent: *
Perplexity-UserPerplexityfetches for a userallowedinherits User-agent: *
Applebot-ExtendedApplemodel trainingallowedinherits User-agent: *
Meta-ExternalAgentMetamodel trainingallowedinherits User-agent: *
AmazonbotAmazonAI search indexallowedinherits User-agent: *
BytespiderByteDancemodel trainingallowedinherits User-agent: *
CCBotCommon Crawlmodel trainingblockedits own group, line 13
DuckAssistBotDuckDuckGoAI search indexallowedinherits User-agent: *
MistralAI-UserMistralfetches for a userallowedinherits User-agent: *
cohere-aiCoheremodel trainingallowedinherits User-agent: *
YouBotYou.comAI search indexallowedinherits User-agent: *

llms.txt

http://example-shop.hr/llms.txt → 200

Well-formed: "getReport fixtures", 4 links.

# getReport fixtures

> Deliberately broken pages used to test a website checkup tool. Each page isolates one problem.

## Pages

- [example-shop.hr](http://example-shop.hr/): the mockup shop with the findings from the report screens
- [Broken links](http://example-shop.hr/broken-links/): a page whose links answer 404, 500 and nothing at all
- [Hreflang, broken](http://example-shop.hr/hreflang-broken/): language alternates without return links

## Optional

- [Fixture index](http://example-shop.hr/): every fixture with its expected findings

What we found

0 failed1 warning2 to note9 passed
Show 9 passed checks

Fix these first

From the example above. Check your own site to get the fixes that apply to it.

  1. Warning: Canonical tag is missing

    Without a canonical URL, search engines may index duplicate versions of this page (with and without trailing slash, with UTM parameters) and split its ranking signals.

    How to fix it: Add <link rel="canonical" href="http://example-shop.hr/"> in <head>.

What this tool checks

What your site currently says to AI crawlers, and the basics an assistant needs to cite you.

  • AI crawler access

    Shows which AI assistants robots.txt blocks, so your content is in or out of AI answers by choice.

  • llms.txt file

    Checks for an optional plain-text index of your key pages, written for AI assistants to read first.

  • llms.txt format

    Checks llms.txt follows the llmstxt.org format, since tools that read it skip a file in the wrong shape.

  • AI opt-out directives

    Checks for tags that ask AI systems not to train on the page, so you know which choice the page signals.

  • robots.txt file

    Checks for the file that tells crawlers which parts of the site to skip and where your sitemap is.

  • Content that needs JavaScript

    Checks how much visible text appears only after scripts run, since it can be indexed late or not at all.

  • Meta description

    Checks for the short summary shown under your result, so Google does not pick a random sentence instead.

  • Canonical tag

    Tells search engines which address is the real one, so duplicates don't split rankings.

  • Page language

    Checks the page declares its language for search engines, translation tools and screen readers.

  • XML sitemap

    Checks for a working sitemap, the list of pages you want indexed, so new and deep pages are found faster.

  • Structured data

    Checks for JSON-LD that lets Google show prices, ratings and breadcrumbs under your listing.

  • Structured data types

    Lists the entity types the page declares, so you know which rich results it can earn.

Show 8 more checks

  • robots.txt file

    Checks for the file that tells crawlers which parts of the site to skip and where your sitemap is.

  • Content that needs JavaScript

    Checks how much visible text appears only after scripts run, since it can be indexed late or not at all.

  • Meta description

    Checks for the short summary shown under your result, so Google does not pick a random sentence instead.

  • Canonical tag

    Tells search engines which address is the real one, so duplicates don't split rankings.

  • Page language

    Checks the page declares its language for search engines, translation tools and screen readers.

  • XML sitemap

    Checks for a working sitemap, the list of pages you want indexed, so new and deep pages are found faster.

  • Structured data

    Checks for JSON-LD that lets Google show prices, ratings and breadcrumbs under your listing.

  • Structured data types

    Lists the entity types the page declares, so you know which rich results it can earn.

How it works

  1. Read robots.txt

    robots.txt is parsed for 18 AI user agents from OpenAI, Anthropic, Google, Perplexity and others, grouped by purpose, training, AI search, or fetching for a user.

  2. Check llms.txt and opt-outs

    /llms.txt and /llms-full.txt are fetched and validated against the llmstxt.org format, and the page is checked for noai and TDM opt-out directives.

  3. Check what they can read

    We look for text that only appears after JavaScript, which AI crawlers do not run, and for a description, canonical, language, sitemap and structured data.

The check shows what your site tells each crawler. It does not tell you what to choose, and it cannot confirm how a crawler actually behaves.

Questions

Is this really free?

Yes. getReport is funded by donations, not plans. This tool runs the full report and shows you the part it is about; the complete report with all seven modules is one click away, also free.

Should I block AI crawlers?

It depends on what you sell. A publisher may block training bots and allow search bots that send visitors; a shop usually wants to be cited everywhere. The finding lists each crawler with its purpose so you can decide per bot.

Is llms.txt required?

No. It is a proposed convention (llmstxt.org) that some assistants read first. It is cheap to add and the check only warns when the file exists but is malformed.

Do you store my results?

The report is kept for 12 months at its shareable link so you can come back to it. Reports are not listed publicly and carry a noindex tag; nothing about your visitors is collected.

Why do AI crawlers need a decision?

ChatGPT, Claude, Perplexity and Google AI Overviews answer questions with content they fetched from the web, and cite the pages they used. Whether your site is in those answers depends on a few robots.txt lines. Some crawlers only train models; others send visitors, so treating them as one thing is the mistake. The check shows what your site currently says to each crawler, so the answer is deliberate: allow the ones that cite and link, block the ones you do not want, and keep the page readable for all of them.

Related free tools

All 41 tools →

Free, funded by the people who use it

€0 of €75 this month. At €75, site crawl up to 500 pages + weekly re-check switches on for everyone.

Chip in