AI crawlers and robots.txt: who fetches your site and how to decide
GPTBot, ClaudeBot, PerplexityBot and friends each have a user-agent token. Allow the ones that send visitors, block the ones you do not want, and write it down in robots.txt.
Step by step, with screenshots: How to detect AI crawlers on your website using server logs →
Check your own site
Runs these 2 checks and the other 191, free, in about 45 seconds.
Or run only this check with the free AI crawlers check, the free AI visibility checker or the free Passage quotability checker.
What it is
AI crawlers are bots that AI companies use to fetch web pages, either to train language models or to answer questions with live results. Each identifies itself with a user-agent token you can address in robots.txt, such as GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity) and CCBot (Common Crawl).
They do different jobs: training crawlers feed future models, search crawlers power answers that link back to you, and user-triggered fetchers load a page because someone asked about it. Google-Extended is only a robots.txt token for Gemini training; Google's AI Overviews use normal Googlebot, so blocking them means leaving Google Search.
OK looks like this: your robots.txt allows or blocks each AI crawler on purpose, and the result matches what you want: citations and visits from AI answers, or your content kept out. getReport lists which known AI crawlers your robots.txt blocks.
What a passing site looks like
- AI crawler access: passesinfo onlyeffort S
- No AI opt-out directive on the pageinfo onlyeffort S
1. AI crawler access
Why it matters. AI assistants (ChatGPT, Claude, Perplexity) fetch pages with their own user agents. Google's AI Overviews use normal Googlebot, so only blocking Googlebot removes you from them, and that removes you from Search too. Blocking them keeps your content out of AI answers; allowing them can bring citations and visitors. Either is a valid choice, as long as it is the one you meant.
- Decide per crawler: training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) feed models; search bots (OAI-SearchBot, PerplexityBot) feed answers that link back; user bots (ChatGPT-User, Claude-User) fetch a page because someone asked. Google-Extended is a robots.txt token, not a crawler: it controls use in Gemini training, not AI Overviews.
- Edit robots.txt: a "User-agent: GPTBot" group with "Disallow: /" blocks it; leave it out of robots.txt (or "Allow: /") to permit it. Rules for "*" apply to any crawler without its own group.
2. AI opt-out directives
Why it matters. A "noai" or "noimageai" robots directive or a TDM reservation tag asks AI systems not to use the page for training. Support varies by operator; it is a signal, not a lock, but it documents your choice.
- Keep it if it is deliberate. To opt back in, remove the noai/noimageai tokens from the robots meta tag or X-Robots-Tag header and the tdm-reservation meta tag.
- For a real block, pair it with robots.txt rules for the crawlers you want out.
Questions people ask
What is GPTBot?
GPTBot is OpenAI's web crawler that collects publicly available pages to train its models. It respects robots.txt: a group starting User-agent: GPTBot with Disallow: / blocks it. Blocking GPTBot does not remove you from ChatGPT search, which uses a separate crawler called OAI-SearchBot.
What is ClaudeBot?
ClaudeBot is Anthropic's crawler that collects public web pages for training its Claude models. It follows robots.txt, so User-agent: ClaudeBot with Disallow: / blocks it. Anthropic also uses separate user agents for fetching pages when a user asks Claude about them, which you can allow or block on their own.
How do you block GPTBot in robots.txt?
Add a group to robots.txt with User-agent: GPTBot on one line and Disallow: / on the next. To block only part of the site, disallow that path instead. OpenAI's other agents, OAI-SearchBot and ChatGPT-User, need their own groups if you want to block them too.
What is PerplexityBot?
PerplexityBot is the crawler Perplexity uses to find and index pages for its AI search answers, which cite and link to their sources. Perplexity says it is not used to train AI models, and it follows robots.txt, so User-agent: PerplexityBot with Disallow: / blocks it. Pages fetched when a user asks about a specific URL come from a separate agent, Perplexity-User, which Perplexity says generally does not follow robots.txt.