AI crawlers and robots.txt — who fetches your site and how to decide
GPTBot, ClaudeBot, PerplexityBot and friends each have a user-agent token. Allow the ones that send visitors, block the ones you do not want, and write it down in robots.txt.
Step by step, with screenshots: AI crawlers and robots.txt: who fetches your site and how to decide →
Check your own site
Runs these 2 checks and the other 185, free, in about 45 seconds.
What a passing site looks like
- AI crawler access: passesinfo onlyeffort S
- No AI opt-out directive on the pageinfo onlyeffort S
1. AI crawler access
Why it matters. AI assistants (ChatGPT, Claude, Perplexity) fetch pages with their own user agents. Google's AI Overviews use normal Googlebot, so only blocking Googlebot removes you from them, and that removes you from Search too. Blocking them keeps your content out of AI answers; allowing them can bring citations and visitors. Either is a valid choice, as long as it is the one you meant.
- Decide per crawler: training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) feed models; search bots (OAI-SearchBot, PerplexityBot) feed answers that link back; user bots (ChatGPT-User, Claude-User) fetch a page because someone asked. Google-Extended is a robots.txt token, not a crawler: it controls use in Gemini training, not AI Overviews.
- Edit robots.txt: a "User-agent: GPTBot" group with "Disallow: /" blocks it; leave it out of robots.txt (or "Allow: /") to permit it. Rules for "*" apply to any crawler without its own group.
2. AI opt-out directives
Why it matters. A "noai" or "noimageai" robots directive or a TDM reservation tag asks AI systems not to use the page for training. Support varies by operator; it is a signal, not a lock, but it documents your choice.
- Keep it if it is deliberate. To opt back in, remove the noai/noimageai tokens from the robots meta tag or X-Robots-Tag header and the tdm-reservation meta tag.
- For a real block, pair it with robots.txt rules for the crawlers you want out.