# AI crawlers and robots.txt — who fetches your site and how to decide

> GPTBot, ClaudeBot, PerplexityBot and friends each have a user-agent token. Allow the ones that send visitors, block the ones you do not want, and write it down in robots.txt.

Technical SEO · HTML version: https://getreport.app/learn/ai-crawlers

## AI crawler access

Passing looks like: AI crawler access passes.

**Why it matters.** AI assistants (ChatGPT, Claude, Perplexity) fetch pages with their own user agents. Google's AI Overviews use normal Googlebot, so only blocking Googlebot removes you from them, and that removes you from Search too. Blocking them keeps your content out of AI answers; allowing them can bring citations and visitors. Either is a valid choice, as long as it is the one you meant.

**How to fix it.**

1. Decide per crawler: training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) feed models; search bots (OAI-SearchBot, PerplexityBot) feed answers that link back; user bots (ChatGPT-User, Claude-User) fetch a page because someone asked. Google-Extended is a robots.txt token, not a crawler: it controls use in Gemini training, not AI Overviews.
2. Edit robots.txt: a "User-agent: GPTBot" group with "Disallow: /" blocks it; leave it out of robots.txt (or "Allow: /") to permit it. Rules for "*" apply to any crawler without its own group.

## AI opt-out directives

Passing looks like: No AI opt-out directive on the page.

**Why it matters.** A "noai" or "noimageai" robots directive or a TDM reservation tag asks AI systems not to use the page for training. Support varies by operator; it is a signal, not a lock, but it documents your choice.

**How to fix it.**

1. Keep it if it is deliberate. To opt back in, remove the noai/noimageai tokens from the robots meta tag or X-Robots-Tag header and the tdm-reservation meta tag.
2. For a real block, pair it with robots.txt rules for the crawlers you want out.

Check your own page: https://getreport.app/