When a site runs through Cloudflare, the robots.txt that crawlers read may not be the file on your server. Cloudflare's managed robots.txt setting, available on all plans, adds its own block to the top of your file: Disallow groups for known AI training crawlers such as GPTBot, ClaudeBot and Google-Extended, and a Content-Signal line that states how your content may be used. If you have no file, Cloudflare serves one for you. This guide shows what the Cloudflare robots.txt adds, how it combines with your own rules, how to turn it on or off, and how it differs from actually blocking bots. For how robots.txt works in general, start with the robots.txt guide.
Quick answer
- What it is: a setting that makes Cloudflare prepend a managed block to your robots.txt, or serve one when your origin has none.
- What it adds: a
User-Agent: *group withContent-signal: search=yes, ai-train=noandAllow: /, thenDisallow: /groups for a list of AI training crawlers. - Where it lives: Security → Settings, filtered by Bot traffic, in the Cloudflare dashboard.
- Your rules still apply. Cloudflare's
Allow: /and your ownUser-agent: *rules are merged, and your longerDisallowrules win. - Search is not blocked. Googlebot and Bingbot are not in the managed list.
- It is a request, not a wall. To enforce a block, use AI Crawl Control or a WAF rule.
- Check the live file, not the one on your server. The robots.txt tester fetches exactly what crawlers get.
What the Cloudflare managed robots.txt adds
Cloudflare's documentation shows the managed content. It starts with a long comment block explaining the Content Signals Policy, then:
# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
# END Cloudflare Managed ContentAfter the END line comes your own file, unchanged. Cloudflare maintains the list of crawlers, so it can change over time; open your live /robots.txt to see the current one.
The Content-Signal line
Content-signal is not part of the robots.txt standard (RFC 9309). It is Cloudflare's Content Signals Policy, a machine-readable statement of three uses:
| Signal | Meaning in the policy |
|---|---|
search | Building a search index and showing links and short excerpts. Does not include AI-generated summaries. |
ai-input | Feeding content into an AI model at answer time, such as retrieval or grounding for AI search answers. |
ai-train | Training or fine-tuning AI models. |
The managed default says yes to search, no to training and leaves ai-input unset, which under the policy neither grants nor refuses it. Cloudflare is also testing a use= value, such as use=reference for "index, excerpt and link back". The comment block frames these signals as a reservation of rights under Article 4 of the EU Directive on Copyright in the Digital Single Market.
Crawlers that do not understand the line ignore it, as they ignore any unknown line in robots.txt. Cloudflare's own documentation notes that Google Search Console may report "Syntax not understood" for Content-Signal lines, and says it has seen no effect on crawling or SEO from those warnings.
How it combines with your own robots.txt
Cloudflare checks whether your origin answers /robots.txt with a 200:
- You have a file: Cloudflare puts its managed block in front of it and serves both as one response.
- You have no file: Cloudflare serves the managed block on its own.
The combined file has two User-agent: * groups: Cloudflare's, with Allow: /, and yours, with your Disallow rules. That looks contradictory, but it is not. Under RFC 9309, and in Google's documentation, when several groups name the same user agent, a crawler combines them into one group. Then the longest matching rule wins, so your Disallow: /cart/ still beats Cloudflare's Allow: / for /cart/ URLs, because it is longer.
Two things can still surprise you:
- Your own group for a listed crawler. If your file already has a
User-agent: GPTBotgroup that allows something, the crawler merges it with Cloudflare'sDisallow: /. For each URL the longest matching rule decides, so your file and Cloudflare's block can give a different answer than you expect. Test the URL. - Your
Sitemap:lines stay where they were and keep working;Sitemapapplies to the whole file, not to a group.
How to turn the Cloudflare robots.txt setting on or off
In the Cloudflare dashboard, for the domain:
- Go to Security → Settings.
- Filter by Bot traffic.
- Find Set your preference to block training in robots.txt and switch it on or off.
To keep the managed block but drop the long policy comments, Cloudflare's documentation points to a Display Content Signals Policy checkbox under Control AI Crawlers in the zone overview. Dashboard labels change from time to time, so if a name differs, search the dashboard for "robots.txt".
You cannot edit the managed list itself. If you want a different set, for example blocking training crawlers but allowing ClaudeBot, turn the setting off and write those groups in your own robots.txt at the origin. AI crawlers and robots.txt has the tokens and files to copy for each choice.
One more case: Cloudflare says that domains on the Free plan with no robots.txt of their own, and without the managed setting, show the Content Signals Policy comments when a crawler asks for /robots.txt. Those are comments only, with no rules, so they allow everything.
robots.txt setting vs blocking AI bots
Cloudflare has several AI-crawler features, and they do different things:
| Feature | What it does | Enforced? |
|---|---|---|
| Managed robots.txt | Adds Disallow rules and content signals to the file | No: a request that compliant crawlers follow |
| AI bot policies (Security → Settings) | Blocks crawlers Cloudflare classifies as Training, Agent or Search, on all pages or only on pages with ads | Yes: requests are blocked |
| AI Crawl Control | Per-crawler allow or block, with a list of which crawlers ignored your robots.txt | Yes: creates a WAF rule |
robots.txt is honoured by the operators who say they follow it, such as OpenAI, Anthropic, Google and Common Crawl, and ignored by those who do not. Cloudflare's own documentation says the file expresses preferences but does not stop access. If you need a real block, use the AI bot policies or AI Crawl Control; if you use both, keep them consistent.
Cloudflare's documentation also announces new defaults for domains added from 15 September 2026, blocking bots classified as Training or Agent on pages that display ads. If you manage a recently added domain, check Security → Settings to see what is already active.
This matters for measuring. A firewall block answers 403 before robots.txt is ever read, so a robots.txt tester, getReport's included, reports what the file says, not what the firewall does. A site that allows a crawler in robots.txt but blocks it at Cloudflare still never gets crawled by it.
How to check what your Cloudflare robots.txt really says
Open https://yourdomain/robots.txt in a private window: that is the file crawlers get, with the managed block if it is on. Then test pages against it:
The tester fetches the live file through Cloudflare, parses it the way Google does, merging the two * groups, and says whether the page is allowed for Googlebot and for all other agents. It also fetches the sitemaps the file declares and checks noindex on the page. The full getReport report lists which known AI crawlers your robots.txt blocks, so you can see the effect of the managed block at a glance:
For the Cloudflare side, the Directives tab in AI Crawl Control shows how often crawlers request robots.txt on each hostname, the status code they got, whether content signals are present, and which crawlers ignored your rules. The Cloudflare settings guide covers the other dashboard settings worth checking.
Common mistakes
- Editing robots.txt on the server and wondering why the live file is longer. The managed block comes from Cloudflare, not from your origin.
- Assuming the managed setting blocks Googlebot. It does not; search crawlers are not in the list, and
search=yesis the default. - Blocking a crawler in robots.txt and expecting it to stop. Only compliant crawlers stop. Use a Cloudflare block for the rest.
- Allowing a crawler in robots.txt while a Cloudflare AI bot block refuses it. Decide once, apply it in both places.
- Treating "Syntax not understood" for Content-Signal as an error. Google ignores the line; the rest of the file still works.
- Forgetting subdomains. Each hostname has its own robots.txt; check
shop.andblog.hosts too.
Questions people ask
Why does my robots.txt show rules I did not write?
If your site is behind Cloudflare, the managed robots.txt setting is probably on. It puts a block marked "BEGIN Cloudflare Managed content" in front of your file, with AI-crawler Disallow groups and a Content-signal line. Turn it off under Security → Settings, filtered by Bot traffic, if you want only your own rules.
Does Cloudflare's managed robots.txt affect Google rankings?
No. The managed block does not name Googlebot, and its User-Agent: * group allows everything, with search=yes. It does block Google-Extended, which controls whether Google may use your content for Gemini training and grounding; Google says that token does not affect Search ranking or inclusion. Your own rules keep working as before.
Can I edit the robots.txt Cloudflare adds?
No, the managed block is all or nothing. You can switch it on or off and hide the policy comments, but not change the list of crawlers. For a custom mix, such as allowing some AI crawlers and blocking others, turn the setting off and write the groups in the robots.txt on your own server.
What does Content-Signal mean in robots.txt?
Content-Signal is a line from Cloudflare's Content Signals Policy that states how your content may be used: search for search indexes, ai-input for feeding AI answers, ai-train for model training, each set to yes or no. It is not part of the robots.txt standard, and crawlers that do not support it ignore it.