Cloudflare AI bot settings: get cited, or block AI bots, on purpose

Your Cloudflare settings, step by step

6 steps, 2 to change.

Checked against Cloudflare’s docs on 8 October 2026. Cloudflare renames settings often; if a name differs in your dashboard, follow the docs link on the step.

1. 1. Set the AI bot policies (Change)

   They apply even if you never opened them: since 15 September 2026 Cloudflare blocks Training and Agent bots on pages with ads by default. Agent bots are the ones that fetch a page when someone asks an assistant about it.

   1. In the Cloudflare dashboard, open your site, go to Security Settings and filter by Bot traffic.
   2. Select Configure AI bot policies (the older setting was called Block AI bots).
   3. Search: Allow (do not block). Agent: Allow (do not block). Training: Block (on all pages).
   4. Training also covers crawlers that collect for both training and search. Check AI Crawl Control to see which crawlers that is for your site.

   Cloudflare docs for this step

2. 2. Decide on the managed robots.txt (Check)

   It disallows training crawlers only and adds the Content Signals line search=yes, ai-train=no, so it fits your goal. It does not touch search bots.

   1. In the Cloudflare dashboard, open your site, go to Security Settings and filter by Bot traffic.
   2. Turn on the preference to block training in robots.txt if you want training crawlers asked to stay out.
   3. Open yoursite/robots.txt to see the result: Cloudflare puts its lines before yours, in one file.

   Cloudflare docs for this step

3. 3. Test Bot Fight Mode, and turn it off if it challenges AI bots (Check)

   Cloudflare says no WAF custom rule or Page Rule can skip Bot Fight Mode, and its docs do not say whether verified AI bots get through. If they are challenged, turning it off is the only fix on the Free plan.

   1. In the Cloudflare dashboard, open your site and go to AI Crawl Control to see which AI crawlers reach your site and which are blocked.
   2. If the search bots you want are missing or blocked: In the Cloudflare dashboard, open your site, go to Security Settings and filter by Bot traffic. Go to Bot fight mode and turn it off.

   Cloudflare docs for this step

4. 4. Add one WAF rule that lets verified AI bots through (Change)

   Super Bot Fight Mode, Browser Integrity Check, Under Attack mode, rate limiting and your own rules can all challenge AI bots, and most AI bots run no JavaScript, so they cannot pass a challenge. One Skip rule placed first lets verified AI bots past all of them.

   1. In the Cloudflare dashboard, open your site and go to the Security rules page.
   2. Select Create rule, then Custom rules.
   3. Name the rule, select Edit expression and paste the expression below.
   4. Choose the action Skip and tick what to skip.
   5. Deploy it and move it to the top of the list. The Free plan has 5 custom rules, Pro 20, Business 100.
   Rule name: Let verified AI bots throughCopy expression

   (cf.verified_bot_category in {"AI Search" "AI Assistant"})

   Action: Skip. Tick: All remaining custom rules, Super Bot Fight Mode rules, Browser Integrity Check, Security Level and Rate limiting rules (only the ones offered on your plan). Place it first.

   cf.verified_bot_category only matches bots Cloudflare has verified, so a fake GPTBot user agent does not get through.

   Cloudflare docs for this step

5. 5. Keep Browser Integrity Check, skipped for AI bots (Keep)

   It challenges visitors with no user agent or an unusual one. The Skip rule above already passes verified AI bots, so you can leave it on for everyone else.

   1. Nothing to change if you added the Skip rule with Browser Integrity Check ticked.
   2. To turn it off for one path instead, use a Configuration Rule.

   Cloudflare docs for this step

6. 6. Optional: a WAF rule that blocks training bots (Check)

   The AI bot policies above already do this. Use a rule instead when you need exceptions, such as allowing training bots on your blog only.

   1. In the Cloudflare dashboard, open your site and go to the Security rules page.
   2. Select Create rule, then Custom rules.
   3. Name the rule, select Edit expression and paste the expression below.
   4. Choose the action Block and deploy it.
   Rule name: Block AI training botsCopy expression

   (cf.verified_bot_category in {"AI Crawler"}) or (http.user_agent contains "GPTBot") or (http.user_agent contains "ClaudeBot") or (http.user_agent contains "CCBot")

   Action: Block

   Cloudflare docs for this step

Then check the result: the AI crawler check shows what your robots.txt tells each AI bot, and the robots.txt AI rules generator writes the robots.txt side.