Skip to content

robots.txt AI rules: choose which AI bots may read your site

This robots.txt AI rules generator lists 13 AI bots and tokens with what each one is for, lets you allow or block each, and writes the robots.txt block to copy.

Start from a preset
  • GPTBot (OpenAI)
    GPTBot OpenAI, trainingCollects pages to train the company’s AI models.Blocking it keeps your pages out of future training data. It does not remove you from that company’s search answers, which use a separate bot.
  • OAI-SearchBot (OpenAI)
    OAI-SearchBot OpenAI, ai searchBuilds the index the assistant searches when it answers a question and cites sources.Blocking it can remove your pages from that assistant’s answers and the links under them.
  • ChatGPT-User (OpenAI)
    ChatGPT-User OpenAI, fetch on requestOpens a page when a person asks the assistant about it or pastes its link.Blocking it means the assistant cannot read your page when someone asks for it by name.
  • ClaudeBot (Anthropic)
    ClaudeBot Anthropic, trainingCollects pages to train the company’s AI models.Blocking it keeps your pages out of future training data. It does not remove you from that company’s search answers, which use a separate bot.
  • Claude-SearchBot (Anthropic)
    Claude-SearchBot Anthropic, ai searchBuilds the index the assistant searches when it answers a question and cites sources.Blocking it can remove your pages from that assistant’s answers and the links under them.
  • Claude-User (Anthropic)
    Claude-User Anthropic, fetch on requestOpens a page when a person asks the assistant about it or pastes its link.Blocking it means the assistant cannot read your page when someone asks for it by name.
  • PerplexityBot (Perplexity)
    PerplexityBot Perplexity, ai searchBuilds the index the assistant searches when it answers a question and cites sources.Blocking it can remove your pages from that assistant’s answers and the links under them.
  • Perplexity-User (Perplexity)
    Perplexity-User Perplexity, fetch on requestOpens a page when a person asks the assistant about it or pastes its link.Blocking it means the assistant cannot read your page when someone asks for it by name.Perplexity says this bot may not follow robots.txt for fetches a person asks for.
  • Googlebot (Google)
    Googlebot Google, search engineBing’s search crawler. Microsoft Copilot and other assistants answer from the Bing index.Blocking it removes you from Bing Search too, and from the assistants built on it. Presets never block it.
  • Google-Extended (Google)
    Google-Extended Google, training and groundingA robots.txt token, not a crawler: Googlebot does the fetching, and this token decides whether Gemini may train on your pages and use them to ground its answers.Blocking it keeps your pages out of Gemini training and grounding. Google Search and AI Overviews are not affected.
  • bingbot (Microsoft)
    bingbot Microsoft, search engineBing’s search crawler. Microsoft Copilot and other assistants answer from the Bing index.Blocking it removes you from Bing Search too, and from the assistants built on it. Presets never block it.
  • Applebot-Extended (Apple)
    Applebot-Extended Apple, trainingCollects pages to train the company’s AI models.Blocking it keeps your pages out of future training data. It does not remove you from that company’s search answers, which use a separate bot.A robots.txt token, not a crawler: Applebot does the fetching. Blocking it opts you out of training Apple’s AI models; your pages stay in Spotlight, Siri and Safari results.
  • CCBot (Common Crawl)
    CCBot Common Crawl, open training datasetCommon Crawl’s open web archive, which many AI models are trained on.Blocking it keeps new pages out of the next Common Crawl archives. It does not affect any assistant’s search answers.

Paste it to see which AI bots it already blocks at the site root. Open yoursite.com/robots.txt and copy everything.

Built in your browser. Nothing is uploaded.

Your AI rules

5 bots blocked. Paste the block at the end of your robots.txt.

  • robots.txt is a request, not a lock: the large operators follow it, but it cannot stop a bot that ignores it. A firewall rule can.

robots.txt block to copy

Add it to the robots.txt at the root of your domain. Each bot picks up the change the next time it fetches the file.

robots-ai-rules.txt
# AI bot rules, generated 2026-10-27 with getreport.app/tools/robots-ai-rules

# OpenAI GPTBot: training
User-agent: GPTBot
Disallow: /

# Anthropic ClaudeBot: training
User-agent: ClaudeBot
Disallow: /

# Google Google-Extended: training and grounding
User-agent: Google-Extended
Disallow: /

# Apple Applebot-Extended: training
User-agent: Applebot-Extended
Disallow: /

# Common Crawl CCBot: open training dataset
User-agent: CCBot
Disallow: /
  • Free, no account
  • Runs in your browser
  • 13 AI bots and robots.txt tokens

What the robots.txt AI rules generator covers

Each bot's purpose in plain words and what blocking it costs you. Built in your browser; nothing is uploaded.

  • Every bot explained

    Training, AI search, fetch on request or search engine, with the trade-off of blocking each one in one sentence.

  • Three presets

    Allow all, block training only, or block all AI. No preset blocks bingbot, because that would also take you out of Bing.

  • Tokens without a crawler

    Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers. Blocking them opts you out of AI training, not search.

  • Your current file read

    Paste your robots.txt to see which AI bots it already blocks, and on which line, with the same parser our report uses.

  • Blocked bots let back in

    A bot your current file blocks but you now allow gets an Allow group, with a note on the private paths to copy into it.

  • Ready to paste

    One commented group per bot, to add at the end of the robots.txt at the root of your domain.

Show 2 more checks

  • Blocked bots let back in

    A bot your current file blocks but you now allow gets an Allow group, with a note on the private paths to copy into it.

  • Ready to paste

    One commented group per bot, to add at the end of the robots.txt at the root of your domain.

How the robots.txt AI rules generator works

  1. Pick a preset

    Start from allow all, block training only or block all AI, then switch any bot between Allow and Block.

  2. Paste your file

    Optional. Paste your current robots.txt to see what it does to each AI bot today, and get warnings about groups that would clash.

  3. Copy the block

    Copy or download the rules and add them to your robots.txt. Bots read the change the next time they fetch the file.

robots.txt is a request, not a lock. The large operators follow it, but a bot that ignores it needs a firewall rule, and Perplexity says Perplexity-User may not follow it.

robots.txt AI rules questions

How do I block AI bots with robots.txt?

Add a group per bot: a User-agent line with its token, such as GPTBot or ClaudeBot, then Disallow: /. The generator writes those groups for the bots you pick. Bots you leave out follow your User-agent: * rules, so you only list the ones to block.

What is Google-Extended?

A robots.txt token Google uses to decide whether Gemini may train on your pages and use them to ground its answers. It has no crawler of its own: Googlebot does the fetching. Google says blocking it does not affect your inclusion or ranking in Google Search.

What is Applebot-Extended?

A robots.txt token that opts your pages out of training Apple's AI models. Apple says it does not crawl pages itself, and that pages that disallow it stay discoverable in Spotlight, Siri and Safari. Applebot, which is not affected, does the crawling.

Should I block training bots or search bots?

Training bots, if you want to keep your text out of AI models, and keep the search bots allowed. Search bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the index assistants cite from; blocking them can take you out of those answers. Blocking GPTBot or ClaudeBot does not.

Why does an allowed bot get an Allow group?

Because your current file blocks it. A bot follows the most specific group that names it and ignores the User-agent: * group, so a group with Allow: / lets it back in. That also lifts your other Disallow lines for that bot, so copy any private paths into the group.

Do AI bots really follow robots.txt?

The large ones say they do: OpenAI, Anthropic, Google, Apple and Common Crawl document their tokens. It is still a request, not a lock. Perplexity says its Perplexity-User fetcher may not follow it for fetches a person asks for, and unknown scrapers can ignore it, which only a firewall rule stops.

Is the robots.txt AI rules generator free?

Yes, with no sign-up, and it runs in your browser. getReport is funded by donations, not plans.

Related free tools

All 59 tools →

Free, funded by the people who use it

No account, no paid plan. Donations keep it running.

Chip in