# LLM SEO: the two ways language models use your site > A language model can know your site from its training data or read it live when someone asks. Which bots feed each route, how fast changes show, and what to do. Updated 2026-11-05 · Technical SEO · HTML version: https://getreport.app/guides/llm-seo LLM SEO is the work of getting large language models to use your content correctly, and a model can use it by exactly two routes: what it learned from your pages during training, or what it reads from them live, through a search index, when someone asks a question. The routes are fed by different bots, obey different robots.txt lines, update at very different speeds, and only one of them sends a link back to you. Our [guide to Google AI Mode](https://getreport.app/guides/google-ai-mode) covers the most visible retrieval product in detail; this chapter separates the two routes so you spend effort where it shows, and ends with a robots.txt we tested against our own fixture. ![Two routes from your page into an LLM answer: training crawlers feed the model's memory, which changes only at the next model release and gives no link, while search crawlers feed a live index that is read when someone asks and cites your page](https://getreport.app/guides/img/llm-seo/diagram.svg "Training data is slow and unlinked; retrieval is fast and cites you. robots.txt sets each route separately.") ## Quick answer - **Route 1, training data:** training crawlers (GPTBot, ClaudeBot, CCBot, and Google's Google-Extended token) collect pages that may end up in a future model's memory. Changes show months later, at the next model release, and answers from memory carry no link. - **Route 2, retrieval:** search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot for AI Overviews and AI Mode) build the indexes assistants read at answer time. Changes show in hours to days, and the answer cites your page. - **Almost all the LLM SEO work you control pays off on route 2:** let the search crawlers in, serve your text in plain HTML, be indexed in Google and Bing, and write pages that state facts plainly. - **You can block training and keep retrieval**, because each operator gives the two routes separate robots.txt tokens. Blocking ChatGPT-style training does not remove you from ChatGPT search. - **A partial block is the common mistake:** blocking two training bots, as our fixture does, leaves the others inheriting `User-agent: *` and still allowed. - **Check what your robots.txt says to each bot** with the free [AI crawler check](https://getreport.app/tools/ai-crawler-check), then write the rules with the [robots.txt AI rules generator](https://getreport.app/tools/robots-ai-rules). ## What does an LLM actually know about your site? A language model answers from two kinds of knowledge, and it helps to keep them apart before you change anything. **Parametric memory** is what the model absorbed during training: patterns and facts compressed into its weights from a huge corpus of text, much of it crawled from the open web. Every model has a training data cutoff, which OpenAI and Anthropic list in their model documentation. After that date the model has learned nothing new. Your pages may be in that corpus because a training crawler fetched them, or because they were in a public archive such as Common Crawl, which many model builders use; OpenAI's GPT-3 paper (Brown et al., 2020) names filtered Common Crawl as the largest part of that model's training mix. **Retrieval** is what the assistant reads at answer time. When a question needs current or specific information, the product searches an index, fetches a handful of pages, and has the model write from them, citing the ones it used. ChatGPT search, Claude's web search, Perplexity, Microsoft Copilot and Google's AI Overviews and AI Mode all work this way. Google describes AI Mode running a "query fan-out": several related searches at once, each pulling pages from Google's index. The practical difference is control. You cannot edit a model's memory; you can only stop new copies of your pages entering future training. Retrieval reads your current page, so you can change it by tomorrow. That is why our pillar on [generative engine optimization](https://getreport.app/guides/generative-engine-optimization) treats training data as the lever you influence least. ## Which bots feed which route? Every major operator now publishes separate user agents for training, for its search index, and for pages fetched because a user asked. The table is the reference for the rest of this guide; every token comes from the operator's own crawler documentation, checked in October 2026. | Route | Bots and tokens | robots.txt line | How fast a change shows | Link back to you? | What you control | | --- | --- | --- | --- | --- | --- | | Training data | GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended (Google token, no crawler of its own) | `User-agent: GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended` | Only at the next model trained after the crawl: months, and never retroactive | No, unless the answer also searches | Whether future crawls may collect your pages | | Retrieval: search index | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), Googlebot (Google Search, AI Overviews, AI Mode) | `User-agent: OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot` | OpenAI and Perplexity say robots.txt changes apply in about 24 hours; content changes show after the next recrawl, usually days | Yes, as a citation | Access, what the page says, snippet limits (`nosnippet`, `max-snippet`) | | Retrieval: user-triggered fetch | ChatGPT-User, Claude-User, Perplexity-User | `User-agent: ChatGPT-User`, `Claude-User`, `Perplexity-User` | Immediate: the page is fetched at the moment someone asks | Usually, in the answer | Limited: OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it; Anthropic says Claude-User respects it | Three details matter more than they look. **Google-Extended is not a crawler.** Googlebot does all of Google's fetching, and Google-Extended is a token that tells Google whether those pages may be used for Gemini training and grounding. Google says it does not affect inclusion or ranking in Google Search, so it also does not take you out of AI Overviews or AI Mode, which are part of Search. To be eligible there, Google says a page only needs to be indexed and eligible for a snippet. **OpenAI may share one crawl between routes.** Its documentation says that if a site allows both GPTBot and OAI-SearchBot, it may use the results of one crawl for both purposes. The robots.txt decision is still per token. **Common Crawl is training data for many companies at once.** CCBot builds a public archive anyone can download, so blocking it affects future archives used by many model builders, not one product. Pages already in published archives stay there. ## What LLM SEO work pays off on the retrieval route? Retrieval is where you get cited, linked and measured, and where a fix shows within days. In order of impact: 1. **Let the search crawlers in, at every layer.** robots.txt must allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, and so must your CDN and firewall. OpenAI documents that sites which disallow OAI-SearchBot are not shown in ChatGPT search answers. A bot blocked at the edge never reads your robots.txt at all. 2. **Be indexed where assistants search.** AI Overviews and AI Mode use Google's index. Copilot grounds its answers in Bing, and OpenAI says ChatGPT search uses third-party search providers as well as its own crawler, so submit your sitemap in Bing Webmaster Tools as well as Google Search Console. Our tour of [the AI search engines](https://getreport.app/guides/ai-search-engines) shows which index each one searches. 3. **Serve your main text in the HTML.** Most AI crawlers do not run JavaScript, so text, prices and answers that appear only after scripts are invisible to them. Our guide to [whether AI crawlers render JavaScript](https://getreport.app/guides/can-ai-crawlers-render-javascript) has the crawler-by-crawler detail and the curl test. 4. **Do not over-restrict snippets.** `nosnippet` and a very low `max-snippet` limit what Google can quote, in AI features as much as in classic results. Use `data-nosnippet` on the parts you genuinely want kept out. 5. **Write pages that state facts plainly.** One question per section, the answer in the first sentence, numbers with units, an author and an updated date. No operator documents this as a ranking rule, but it is the text that ends up quoted. 6. **Keep pages fresh and recrawled.** Accurate sitemap `lastmod` dates, and IndexNow where your CMS supports it, get changes picked up sooner. None of this is special to language models: it is ordinary technical SEO aimed at a few more crawlers, which is why Google says optimising for its AI features is still SEO. ## What LLM SEO work affects training data? Training data is slow and unlinked, but it is what a model says about you when it answers without searching. The work here is smaller and mostly indirect: 1. **Decide per training token.** Allowing GPTBot, ClaudeBot, CCBot and Google-Extended lets future models learn from your pages; blocking them keeps future copies out. Neither choice changes a model already released. 2. **Be described consistently across the web.** A model learns about you from every page in its corpus, and most of those pages are not yours: directories, reviews, forums, news, reference sites. One consistent name, description and set of facts everywhere is the best you can do for memory. 3. **Publish stable, unambiguous facts on your own site.** What you do, where, since when, for whom. Facts that do not change between crawls survive into training intact. 4. **Fix wrong memory through retrieval, not training.** If an assistant states something outdated about you from memory, you cannot correct the model. You can publish a clear, current page that search crawlers index, so answers that search pick up the right fact and cite it. ## Should you block training bots but allow search bots? For many sites this is the sensible default: you keep citations and referral visits, and stop handing future training sets a free copy of your work. Make the trade-off knowingly. **What you keep:** every citation and link from ChatGPT search, Claude's search, Perplexity and Google's AI features, because those come from route 2. Anthropic says blocking ClaudeBot does not affect Claude-SearchBot or Claude-User, and OpenAI treats GPTBot and OAI-SearchBot as independent settings. **What you give up:** future models will know less about you natively, so answers given without a search (and some assistants answer many questions that way) are less likely to mention you, or will rely on what others wrote. Blocking Google-Extended also affects grounding in Gemini apps, not only training, so it is not a pure training switch. Publishers whose text is the product usually block training. Shops, local businesses and documentation sites usually allow everything: they gain nothing from keeping their product facts out of future models. ## How do you check which robots.txt group each bot obeys? A bot uses the one group whose `User-agent` line names it, and falls back to `User-agent: *` only when no group does (RFC 9309, section 2.2.1). That rule is behind most broken AI opt-outs, so we tested it. Our fixture server serves this file at `http://localhost:4100/robots.txt`; it blocks two training bots and lets every other bot inherit `*`: ```text # Fixture robots.txt, served by the getReport fixture server on localhost:4100 User-agent: * Disallow: /pages/noindex-meta/ Disallow: /private/ User-agent: GetReportBot Disallow: /opted-out/ Allow: / # AI crawlers (tool 28): two training bots blocked, everything else inherits the * group User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / Sitemap: http://localhost:4100/sitemap.xml ``` First, robots.txt is the same file whoever asks, and the server does not enforce it. Sending the fixture homepage a request as GPTBot and as OAI-SearchBot gets the same answer: ```bash # Terminal: same page, two user agents $ curl -s -o /dev/null -w "%{http_code} %{size_download} bytes\n" -A "GPTBot" http://localhost:4100/ 200 7573 bytes $ curl -s -o /dev/null -w "%{http_code} %{size_download} bytes\n" -A "OAI-SearchBot" http://localhost:4100/ 200 7573 bytes ``` A blocked bot stays away by its own choice, not the server's, so a block you need enforced belongs at the firewall or CDN, as our guide to [AI crawlers and robots.txt](https://getreport.app/guides/ai-crawlers-and-robots-txt) shows. Second, which group does each bot obey? This short script (Python 3 standard library only) finds the group per bot under the RFC 9309 rule and cross-checks the verdict for `/` with Python's built-in `urllib.robotparser`: ```python # which_group.py: usage python3 -I which_group.py ... import sys, urllib.request from urllib import robotparser url = sys.argv[1] text = open(url[7:]).read() if url.startswith("file://") else urllib.request.urlopen(url).read().decode() groups, cur, last_ua = [], None, False for n, line in enumerate(text.splitlines(), 1): line = line.split('#', 1)[0].strip() if ':' not in line: continue key, val = (s.strip() for s in line.split(':', 1)) if key.lower() == 'user-agent': if not last_ua: cur = {'agents': [], 'line': n, 'rules': []} groups.append(cur) cur['agents'].append(val.lower()) last_ua = True elif key.lower() in ('allow', 'disallow') and cur: cur['rules'].append(f'{key} {val}') last_ua = False rp = robotparser.RobotFileParser(); rp.parse(text.splitlines()) for bot in sys.argv[2:]: g = next((g for g in groups if bot.lower() in g['agents']), None) or \ next(g for g in groups if '*' in g['agents']) home = 'allowed' if rp.can_fetch(bot, '/') else 'blocked' name = bot if bot.lower() in g['agents'] else '*' print(f"{bot:<17} line {g['line']:<3} User-agent: {name:<16} / {home:<8} {'; '.join(g['rules'])}") ``` Run against the fixture, it prints: ```text $ python3 -I which_group.py http://localhost:4100/robots.txt GPTBot CCBot ClaudeBot Google-Extended OAI-SearchBot ChatGPT-User Claude-SearchBot PerplexityBot Googlebot GPTBot line 10 User-agent: GPTBot / blocked Disallow / CCBot line 13 User-agent: CCBot / blocked Disallow / ClaudeBot line 1 User-agent: * / allowed Disallow /pages/noindex-meta/; Disallow /private/ Google-Extended line 1 User-agent: * / allowed Disallow /pages/noindex-meta/; Disallow /private/ OAI-SearchBot line 1 User-agent: * / allowed Disallow /pages/noindex-meta/; Disallow /private/ ChatGPT-User line 1 User-agent: * / allowed Disallow /pages/noindex-meta/; Disallow /private/ Claude-SearchBot line 1 User-agent: * / allowed Disallow /pages/noindex-meta/; Disallow /private/ PerplexityBot line 1 User-agent: * / allowed Disallow /pages/noindex-meta/; Disallow /private/ Googlebot line 1 User-agent: * / allowed Disallow /pages/noindex-meta/; Disallow /private/ ``` Read it route by route. GPTBot and CCBot match their own groups and are blocked from the whole site, so OpenAI's training crawler and Common Crawl are out of route 1. Every search and user bot matches `*`, so route 2 is open everywhere except `/private/` and the noindex test page: exactly what you want for citations. But ClaudeBot and Google-Extended, both training tokens, also match `*` and are allowed. If the goal was "no training", this file does half the job, and nothing in it looks wrong. Third, the fix. This version names all four training tokens in one group and leaves the search bots to `*`. We ran the same script on it: ```text # robots.txt at the root of your domain: out of future training, open to AI search User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: / User-agent: * Disallow: /private/ Sitemap: https://example-shop.hr/sitemap.xml ``` ```text $ python3 -I which_group.py file://$PWD/robots-training-off.txt GPTBot ClaudeBot Google-Extended CCBot OAI-SearchBot Claude-SearchBot PerplexityBot Googlebot GPTBot line 2 User-agent: GPTBot / blocked Disallow / ClaudeBot line 2 User-agent: ClaudeBot / blocked Disallow / Google-Extended line 2 User-agent: Google-Extended / blocked Disallow / CCBot line 2 User-agent: CCBot / blocked Disallow / OAI-SearchBot line 9 User-agent: * / allowed Disallow /private/ Claude-SearchBot line 9 User-agent: * / allowed Disallow /private/ PerplexityBot line 9 User-agent: * / allowed Disallow /private/ Googlebot line 9 User-agent: * / allowed Disallow /private/ ``` All four training tokens now share the group on line 2 and are blocked; every search bot and Googlebot falls to `*` and keeps access, with `/private/` still closed. One caution if you later give a search bot its own group with `Allow: /`: that group replaces `*` for that bot entirely, so copy any private `Disallow` lines into it. ### Check your own site For your own domain you do not need the script. The AI crawler check reads your live robots.txt for 18 AI user agents, grouped by purpose (training, AI search, fetches for a user), and shows for each whether its own group or `User-agent: *` decided the verdict: the same distinction the script prints. It also flags text that only appears after JavaScript runs. > **Free tool:** [AI crawlers check: GPTBot, ClaudeBot and llms.txt](https://getreport.app/tools/ai-crawler-check): See what your robots.txt tells 18 AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot, whether your llms.txt is valid and what needs JavaScript. Free. The same finding appears in every getReport report: > **Check: AI crawler access.** AI assistants (ChatGPT, Claude, Perplexity) fetch pages with their own user agents. Google's AI Overviews use normal Googlebot, so only blocking Googlebot removes you from them, and that removes you from Search too. Blocking them keeps your content out of AI answers; allowing them can bring citations and visitors. Either is a valid choice, as long as it is the one you meant. > > 1. Decide per crawler: training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) feed models; search bots (OAI-SearchBot, PerplexityBot) feed answers that link back; user bots (ChatGPT-User, Claude-User) fetch a page because someone asked. Google-Extended is a robots.txt token, not a crawler: it controls use in Gemini training, not AI Overviews. > 2. Edit robots.txt: a "User-agent: GPTBot" group with "Disallow: /" blocks it; leave it out of robots.txt (or "Allow: /") to permit it. Rules for "*" apply to any crawler without its own group. To write the rules, the robots.txt AI rules generator runs in your browser and nothing is uploaded. It lists 13 AI bots and tokens with what blocking each one costs, offers a "block training only" preset, and if you paste your current file it shows which AI bots it already blocks and on which line. > **Free tool:** [robots.txt AI rules generator: allow or block each bot](https://getreport.app/tools/robots-ai-rules): Free robots.txt AI rules generator: allow or block GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 8 more AI bots, each explained, then copy the rules. **What passing looks like:** in the AI crawler check, the training bots you meant to block show as blocked by their own group, and OAI-SearchBot, Claude-SearchBot and PerplexityBot show as allowed. If a training bot you meant to block shows "allowed" via `*`, you have the fixture's half-block; add its token to your training group and run the check again. Then look at the JavaScript finding, because an allowed bot that sees an empty page cannot cite it either: > **Check: The visible text is present without JavaScript.** Google renders JavaScript later and with a budget, so text that only appears after scripts run can be indexed late or not at all. Other search engines and link previews may never see it. > > 1. Serve the main content in the HTML (server-side rendering or static generation) and use JavaScript only to enhance it. > 2. Check the difference in the technical detail; menus and widgets are fine, headlines and body copy are not. ## How we tested The getReport team ran these tests on 16 October 2026 with curl 8.5.0 and Python 3.11.17, against the robots.txt served by our local fixture server (`http://localhost:4100/robots.txt`, the file shown above) and a copy of the recommended file saved locally as `robots-training-off.txt`. The group matching follows RFC 9309; the allowed or blocked verdicts come from Python's standard `urllib.robotparser`, which agreed with the group matching on every bot we tested. Bot tokens and their documented behaviour were checked against OpenAI's, Anthropic's, Google's, Perplexity's and Common Crawl's crawler pages in October 2026; our sandbox could not open them directly, so we read them through search results. What we did not test: how any real crawler behaves when it meets these files, how long any operator takes to act on a change, whether any model's training data contains a given page, and any AI engine's answers. We sent no queries to ChatGPT, Claude, Perplexity or Google. ## Questions people ask ### How do LLMs get information about my website? By two routes. A model may have learned about your site during training, from pages a training crawler such as GPTBot or ClaudeBot fetched, or from public archives such as Common Crawl, before its knowledge cutoff. Or the assistant reads your site live at answer time, through a search index built by bots such as OAI-SearchBot, Claude-SearchBot, PerplexityBot or Googlebot, and cites the page it used. Only the second route reads your current content. ### How long does it take for an LLM to learn about new content? For answers that search, days: once a search crawler has recrawled the page and the index has updated, the new text can be retrieved and cited. For the model's own memory, months at best, because new content only enters the next model trained after it was crawled, and older models never learn it. OpenAI and Perplexity say robots.txt changes for their crawlers take effect in about 24 hours. ### Can I remove my content from an AI model that was already trained? Not through robots.txt. Blocking GPTBot, ClaudeBot, CCBot or Google-Extended only keeps your pages out of future crawls and future training runs; a model already released keeps what it learned, and pages in published Common Crawl archives stay there. Some operators offer privacy or opt-out request forms for personal data. For outdated facts, the faster route is a clear current page that answer-time search can find and cite. ### Does Common Crawl data end up in AI models? Often, yes. Common Crawl publishes a free public archive of the web collected by its CCBot, and many model builders train on filtered versions of it: OpenAI's GPT-3 paper named filtered Common Crawl as the largest share of that model's training mix. Blocking CCBot in robots.txt keeps your pages out of future archives, which reaches many model builders at once, but it does not remove copies already published. ### What is a knowledge cutoff in an LLM? It is the date after which a model's training data stops, so the model has no built-in knowledge of anything published later. OpenAI and Anthropic list it for each model in their documentation. Questions about anything newer can only be answered if the assistant searches the web, which is why fresh content reaches AI answers through retrieval and search crawlers, not through training. ### Should I block AI training bots if I want to be cited? You can block them without losing citations. Citations come from search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, which have their own robots.txt tokens, so a group that disallows GPTBot, ClaudeBot, CCBot and Google-Extended leaves them untouched. The cost is that future models will know less about you without searching. Shops and documentation sites usually allow training; publishers often block it. ### Is LLM SEO the same as optimising for Google AI Mode? Partly. AI Mode is one retrieval product: it reads pages from Google's index, fetched by ordinary Googlebot, and Google says a page needs only to be indexed and eligible for a snippet to appear. LLM SEO covers more engines, ChatGPT, Claude and Perplexity among them, each with its own search crawler, plus the training-data route. The retrieval basics are the same everywhere, which is why one set of fixes serves all of them. ## Sources - [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) (accessed October 2026) - [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (accessed October 2026) - [Google Search Central: Google's common crawlers, including Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) (accessed October 2026) - [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) (accessed October 2026) - [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) (accessed October 2026) - [Common Crawl: Frequently asked questions (CCBot)](https://commoncrawl.org/faq) (accessed October 2026) - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (September 2022) - [Brown et al.: Language Models are Few-Shot Learners (GPT-3)](https://arxiv.org/abs/2005.14165) (2020)