AI crawlers: see which ones can read and cite your site
See which AI crawlers your robots.txt lets in, 18 of them grouped by purpose, whether your llms.txt is valid, and whether your content is readable without JavaScript.
- Free, no account
- Results in under a minute
- 18 AI crawlers, grouped by purpose
Example result:example-shop.hr
WordPress 6.8Sample shop · homepage · mobile Chrome · the same check you get for your own site
AI crawlers in robots.txt
18 known crawlers · 2 blocked · 16 allowed| Crawler | Operator | Purpose | robots.txt | Rule |
|---|---|---|---|---|
| GPTBot | OpenAI | model training | blocked | its own group, line 10 |
| OAI-SearchBot | OpenAI | AI search index | allowed | inherits User-agent: * |
| ChatGPT-User | OpenAI | fetches for a user | allowed | inherits User-agent: * |
| ClaudeBot | Anthropic | model training | allowed | inherits User-agent: * |
| Claude-SearchBot | Anthropic | AI search index | allowed | inherits User-agent: * |
| Claude-User | Anthropic | fetches for a user | allowed | inherits User-agent: * |
| Google-Extended | model training | allowed | inherits User-agent: * | |
| PerplexityBot | Perplexity | AI search index | allowed | inherits User-agent: * |
| Perplexity-User | Perplexity | fetches for a user | allowed | inherits User-agent: * |
| Applebot-Extended | Apple | model training | allowed | inherits User-agent: * |
| Meta-ExternalAgent | Meta | model training | allowed | inherits User-agent: * |
| Amazonbot | Amazon | AI search index | allowed | inherits User-agent: * |
| Bytespider | ByteDance | model training | allowed | inherits User-agent: * |
| CCBot | Common Crawl | model training | blocked | its own group, line 13 |
| DuckAssistBot | DuckDuckGo | AI search index | allowed | inherits User-agent: * |
| MistralAI-User | Mistral | fetches for a user | allowed | inherits User-agent: * |
| cohere-ai | Cohere | model training | allowed | inherits User-agent: * |
| YouBot | You.com | AI search index | allowed | inherits User-agent: * |
llms.txt
http://example-shop.hr/llms.txt → 200Well-formed: "getReport fixtures", 4 links.
# getReport fixtures > Deliberately broken pages used to test a website checkup tool. Each page isolates one problem. ## Pages - [example-shop.hr](http://example-shop.hr/): the mockup shop with the findings from the report screens - [Broken links](http://example-shop.hr/broken-links/): a page whose links answer 404, 500 and nothing at all - [Hreflang, broken](http://example-shop.hr/hreflang-broken/): language alternates without return links ## Optional - [Fixture index](http://example-shop.hr/): every fixture with its expected findings
What we found
Show 9 passed checks
Fix these first
From the example above. Check your own site to get the fixes that apply to it.
Warning: Canonical tag is missing
Without a canonical URL, search engines may index duplicate versions of this page (with and without trailing slash, with UTM parameters) and split its ranking signals.
How to fix it: Add <link rel="canonical" href="http://example-shop.hr/"> in <head>.
What AI crawlers see on your site
What your site currently says to AI crawlers, and the basics an assistant needs to cite you.
AI crawler access
Shows which AI assistants robots.txt blocks, so your content is in or out of AI answers by choice.
llms.txt file
Checks for an optional plain-text index of your key pages, written for AI assistants to read first.
llms.txt format
Checks llms.txt follows the llmstxt.org format, since tools that read it skip a file in the wrong shape.
AI opt-out directives
Checks for tags that ask AI systems not to train on the page, so you know which choice the page signals.
robots.txt file
Checks for the file that tells crawlers which parts of the site to skip and where your sitemap is.
Content that needs JavaScript
Checks how much visible text appears only after scripts run, since it can be indexed late or not at all.
Meta description
Checks for the short summary shown under your result, so Google does not pick a random sentence instead.
Canonical tag
Tells search engines which address is the real one, so duplicates don't split rankings.
Page language
Checks the page declares its language for search engines, translation tools and screen readers.
XML sitemap
Checks for a working sitemap, the list of pages you want indexed, so new and deep pages are found faster.
Structured data
Checks for JSON-LD that lets Google show prices, ratings and breadcrumbs under your listing.
Structured data types
Lists the entity types the page declares, so you know which rich results it can earn.
Show 8 more checks
robots.txt file
Checks for the file that tells crawlers which parts of the site to skip and where your sitemap is.
Content that needs JavaScript
Checks how much visible text appears only after scripts run, since it can be indexed late or not at all.
Meta description
Checks for the short summary shown under your result, so Google does not pick a random sentence instead.
Canonical tag
Tells search engines which address is the real one, so duplicates don't split rankings.
Page language
Checks the page declares its language for search engines, translation tools and screen readers.
XML sitemap
Checks for a working sitemap, the list of pages you want indexed, so new and deep pages are found faster.
Structured data
Checks for JSON-LD that lets Google show prices, ratings and breadcrumbs under your listing.
Structured data types
Lists the entity types the page declares, so you know which rich results it can earn.
How we check what AI crawlers can read
Read robots.txt
robots.txt is parsed for 18 AI user agents from OpenAI, Anthropic, Google, Perplexity and others, grouped by purpose, training, AI search, or fetching for a user.
Check llms.txt and opt-outs
/llms.txt and /llms-full.txt are fetched and validated against the llmstxt.org format, and the page is checked for noai and TDM opt-out directives.
Check what they can read
We look for text that only appears after JavaScript, which AI crawlers do not run, and for a description, canonical, language, sitemap and structured data.
The check shows what your site tells each crawler. It does not tell you what to choose, and it cannot confirm how a crawler actually behaves.
AI crawler and AI visibility questions
Is this an AI visibility checker?
Partly. It checks access and readability: whether AI crawlers may fetch your pages and can read what is on them. It does not track whether ChatGPT, Perplexity or Google AI Overviews mention or cite you, or for which questions; that needs a prompt-tracking tool. Access comes first, though: a site that blocks AI search bots or hides its text behind JavaScript cannot be cited at all.
How can I improve my brand's visibility in AI search?
Start with access: allow the AI search and user-fetch bots, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot, even if you block training bots. Serve your main text in the HTML, not only after JavaScript. Give every page a clear title, description and structured data, and answer the questions buyers ask in plain sentences. Being mentioned on other trusted sites matters as much as your own pages.
Which AI crawlers does the check cover?
18 user agents from OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon, ByteDance, Common Crawl and others, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and Applebot-Extended. They are grouped by purpose: training models, AI search, or fetching a page when a user asks. Each shows whether your robots.txt allows or blocks it.
How do I block GPTBot but stay in ChatGPT search?
Disallow GPTBot and leave OAI-SearchBot allowed in robots.txt: a group with User-agent: GPTBot and Disallow: /, and a separate group with User-agent: OAI-SearchBot and Allow: /. OpenAI uses GPTBot for training and OAI-SearchBot to show sites in ChatGPT search. Anthropic splits ClaudeBot and Claude-SearchBot the same way. Run the check again to confirm each group reads as you meant.
Does blocking AI crawlers affect my Google rankings?
No, as long as you leave Googlebot alone. Google-Extended is a robots.txt token that only controls whether Gemini models may train on your content; blocking it changes nothing about crawling, indexing or ranking in Google Search, and does not remove you from AI Overviews, which use Search's index. Blocking Googlebot itself would take you out of Google Search entirely.
Should I block AI crawlers?
It depends on what you sell. A publisher may block training bots and allow the search bots that send visitors; a shop usually wants to be cited everywhere. Treating all AI bots as one thing is the common mistake. The check lists each of the 18 crawlers with its company and purpose, so you can decide per bot and then verify the result.
How do I optimize my website content for AI search crawlers?
Put it in the HTML your server sends and make it easy to quote. Most AI crawlers do not run JavaScript, so main text, headings, prices and structured data must be in the first response. Give each page one clear H1, headings that match the questions people ask, answer-first paragraphs, a description, canonical and JSON-LD. Allow the AI search bots in robots.txt. This check flags text that needs JavaScript and a missing description, canonical or structured data.
How do I detect AI crawlers on my website?
Read your server access logs, because robots.txt only states the rules; it does not show who came. Drop a log file into our server log analyser: it runs in your browser, nothing is uploaded, and it counts the requests from each AI crawler, search engine and SEO bot, with the pages they fetched and the status codes they got.
Related free tools
All 59 tools →Free, funded by the people who use it
No account, no paid plan. Donations keep it running.