# LLM visibility: see which pages AI answers use, from your own data > LLM visibility, page by page: which of your pages ChatGPT, Claude and Perplexity read, fetch for live answers, cite and send visitors to, measured from your server log and analytics. Updated 2026-11-06 · Technical SEO · HTML version: https://getreport.app/guides/llm-visibility LLM visibility is how much of your site the products built on large language models (ChatGPT, Claude, Perplexity, Copilot and Google's AI Mode) actually use: which pages they may read, which they fetch for a live answer, which they cite and which send you visitors. Most of it can be measured page by page from data you already own, your server log and your analytics, without tracking a single prompt. Google's newest surface works differently from the rest, and our [guide to Google AI Mode](https://getreport.app/guides/google-ai-mode) explains how it builds answers from Google's own index. This chapter shows how to read the four traces a page leaves on its way into an AI answer, with a script you can run on your own log today and its real output on our example shop. ![A page's five steps into an AI answer (access, index crawl, live fetch, citation, visit) with where each one shows: robots.txt and raw HTML, the server log twice, nowhere unless you ask, and analytics. Training crawls branch off with no per-page trace](https://getreport.app/guides/img/llm-visibility/diagram.svg "Each step leaves its own trace. The first step with no trace is where a page drops out.") ## Quick answer - **Measure LLM visibility per page, not per site.** A page passes through access, an index crawl, sometimes a live fetch, a citation and a click. Each step leaves a different trace. - **Your server log separates three kinds of AI request**: training crawlers (GPTBot, ClaudeBot, CCBot), search index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and fetches triggered by a person's question (ChatGPT-User, Claude-User, Perplexity-User). - **A user-triggered fetch is the strongest log evidence** that a page was pulled into a conversation. Training hits prove only that a page was copied. - **Analytics shows the visits**: referrers such as chatgpt.com and perplexity.ai, plus the `utm_source=chatgpt.com` tag ChatGPT adds. Expect a floor, because some visits arrive as Direct. - **Citations without clicks leave no trace at all.** You see those only by asking the question yourself, so treat it as a spot check. - **Start with access**: the free [AI crawler check](https://getreport.app/tools/ai-crawler-check) shows whether a page is eligible at all. ## What does LLM visibility mean for one page? Site-level numbers hide the useful part: a monthly total of ChatGPT visits says nothing about which page did the work. [What AI visibility is and how to measure it](https://getreport.app/guides/what-is-ai-visibility) sets out the three layers (access, presence, traffic) for a whole site. This page goes one level down: the same layers, read for each URL, so you can tell which pages AI products rely on and where a page drops out. Follow one page through the chain in the diagram. First it must be **eligible**: the operator's crawler may fetch it and the text is in the HTML. Then a search crawler **indexes** it, so the product can retrieve it later. When someone asks a question, the product either answers from that index or **fetches the page live** on the user's behalf. The answer may **cite** it, and the user may **click**. Training crawls sit off to the side: they copy the page into a future model's training data, and no later answer will ever tell you it was used. Each step is recorded in a different place, or not at all. Line up the traces per page, and the first missing one tells you which step failed. ## Which signals show that an AI answer used a page? Four signals are in your own data and one is visible only from outside. The table lists what each one proves and, just as important, what it cannot tell you. | Signal | Where you see it | What it proves | What it misses | | --- | --- | --- | --- | | Access: robots.txt allows the bot, text is in the HTML | robots.txt, raw HTML, the [AI crawler check](https://getreport.app/tools/ai-crawler-check) | The page is eligible to be read by that operator | Whether anyone ever read it | | Training crawl: GPTBot, ClaudeBot, CCBot | Server log, by user agent | The page was copied, possibly for model training | Any link to a later answer; there is none | | Search index crawl: OAI-SearchBot, Claude-SearchBot, PerplexityBot | Server log, by user agent | The product can retrieve the page for answers | Whether it was retrieved for any question | | User-triggered fetch: ChatGPT-User, Claude-User, Perplexity-User | Server log, by user agent | A person's conversation opened the page, at that time | Answers built from the index without a live fetch | | AI referral visit | Analytics (source chatgpt.com, perplexity.ai, claude.ai and others) or the referrer field in your log | The page was linked in an answer and someone clicked | Citations nobody clicked, and visits that arrive with no referrer | | Citation you can see | The answer itself, when you ask | The page is a source for that question, that day | Every other question, user, location and day | Two details from the operators' own documentation shape how you read the log. OpenAI says ChatGPT-User is not used to crawl the web automatically and plays no part in whether content appears in ChatGPT search; that is OAI-SearchBot's job. So a citation in ChatGPT search often comes from the index with no ChatGPT-User request at all, which is the dashed arrow in the diagram. Anthropic documents the same split for Claude: ClaudeBot for training, Claude-SearchBot for search quality and Claude-User for requests people trigger by asking. Perplexity describes Perplexity-User as the agent that visits a page when a user's question needs it, with the link included in the response. How each engine searches and cites is in our [tour of the AI search engines](https://getreport.app/guides/ai-search-engines). Google is the exception. AI Overviews and AI Mode answer from Google's search index, so there is no separate AI Mode fetcher in your log: Googlebot's hits look the same whether a page later appears in a classic result or an AI answer. Google-Extended is a robots.txt token, not a user agent, so it never shows up in a log line either. For Google, the per-page data lives in Search Console, which the pillar covers. ## How to measure LLM visibility page by page Five steps, from the one you control fully to the one you can only sample. ### 1. Check that each page is eligible A page an AI crawler may not fetch, or one whose text appears only after JavaScript runs, cannot show up in any later step. Check this first, because every other number is capped by it. > **Free tool:** [AI crawlers check: GPTBot, ClaudeBot and llms.txt](https://getreport.app/tools/ai-crawler-check): See what your robots.txt tells 18 AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot, whether your llms.txt is valid and what needs JavaScript. Free. The check reads your robots.txt verdict for 18 AI crawlers grouped by purpose (training, AI search, user fetches), validates llms.txt and compares the raw HTML with the rendered page. It tells you what your site says to each crawler; it cannot confirm how a crawler behaves. The same findings appear in every getReport report: > **Check: AI crawler access.** AI assistants (ChatGPT, Claude, Perplexity) fetch pages with their own user agents. Google's AI Overviews use normal Googlebot, so only blocking Googlebot removes you from them, and that removes you from Search too. Blocking them keeps your content out of AI answers; allowing them can bring citations and visitors. Either is a valid choice, as long as it is the one you meant. > > 1. Decide per crawler: training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) feed models; search bots (OAI-SearchBot, PerplexityBot) feed answers that link back; user bots (ChatGPT-User, Claude-User) fetch a page because someone asked. Google-Extended is a robots.txt token, not a crawler: it controls use in Gemini training, not AI Overviews. > 2. Edit robots.txt: a "User-agent: GPTBot" group with "Disallow: /" blocks it; leave it out of robots.txt (or "Allow: /") to permit it. Rules for "*" apply to any crawler without its own group. > **Check: The visible text is present without JavaScript.** Google renders JavaScript later and with a budget, so text that only appears after scripts run can be indexed late or not at all. Other search engines and link previews may never see it. > > 1. Serve the main content in the HTML (server-side rendering or static generation) and use JavaScript only to enhance it. > 2. Check the difference in the technical detail; menus and widgets are fine, headlines and body copy are not. You can also ask your server directly what it sends each agent. This is our fixture of example-shop.hr, served locally, fetched with the user-triggered and search agents (the robots.txt has one `User-agent: *` group that blocks only `/cart/`): ```text # Terminal: one request per AI user agent to the fixture's homepage for ua in "ChatGPT-User/1.0" "Claude-User/1.0" "Perplexity-User/1.0" "OAI-SearchBot/1.0" "GPTBot/1.2"; do printf "%-22s " "$ua" curl -s -o /dev/null -w "%{http_code} %{size_download} bytes\n" -A "Mozilla/5.0 (compatible; $ua)" http://localhost:4101/ done ChatGPT-User/1.0 200 1039 bytes Claude-User/1.0 200 1039 bytes Perplexity-User/1.0 200 1039 bytes OAI-SearchBot/1.0 200 1039 bytes GPTBot/1.2 200 1039 bytes ``` Every agent gets a 200 and the same bytes, which is what passing looks like. A 403, a challenge page or a much smaller body for one agent means a firewall or bot-protection rule treats it differently from browsers, and robots.txt is not the whole story. ### 2. Split the AI requests in your log by purpose Counting "AI bot hits" in one number mixes copying with answering. Split them into training, search index and user-triggered, per page, and add the AI referral visits from the same log. This awk script does all four counts for a standard combined-format log (Apache or nginx): ```text # ai-per-page.awk: AI evidence per page in a combined-format access log BEGIN { FS = "\"" } { split($2, req, " "); url = req[2]; page = url; sub(/\?.*/, "", page); ref = $4; ua = $6 } ua ~ /ChatGPT-User|Claude-User|Perplexity-User|MistralAI-User/ { user[page]++; seen[page]; next } ua ~ /OAI-SearchBot|Claude-SearchBot|PerplexityBot/ { search[page]++; seen[page]; next } ua ~ /GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent/ { train[page]++; seen[page]; next } ref ~ /chatgpt\.com|perplexity\.ai|claude\.ai|copilot\.microsoft\.com|gemini\.google\.com/ || url ~ /utm_source=chatgpt\.com/ { visit[page]++; seen[page] } END { for (p in seen) printf "%-32s %5d %6d %5d %6d\n", p, train[p], search[p], user[p], visit[p] } ``` Here it is on the example-shop.hr fixture log, 3,000 requests from 1 to 14 September 2026: ```text # Terminal, from the repository root printf "%-32s %5s %6s %5s %6s\n" page train search user visits awk -f ai-per-page.awk fixtures/logs/access.log | sort -k2,2nr -k1,1 | head -6 page train search user visits /blog/ 14 3 0 0 /blog/kako-odabrati-cipele/ 10 2 0 0 /proizvodi/jakne/ 7 4 0 0 /proizvodi/ruksaci/ 6 3 0 0 /proizvodi/trekking-cipele/ 6 3 0 0 / 5 3 0 0 # Totals over all 15 pages AI agents touched pages 15 train 80 search 40 user 0 visits 0 ``` That is a sobering picture: 120 AI requests (4% of the log) across 15 pages, 80 of them training copies, 40 from PerplexityBot's search index, and not one user-triggered fetch or AI referral in two weeks. The pages are indexed by one AI search engine and used, as far as the log can show, by nobody. A crawler-hit dashboard would have reported 120 "AI visits" and called it visibility. To prove the script catches the signals that matter, we appended three hand-written lines to a copy of the log: a ChatGPT-User fetch of `/dostava/`, a visit to `/dostava/?utm_source=chatgpt.com` and a visit to `/proizvodi/jakne/` with a perplexity.ai referrer. The rerun counts each one on the right page (the query string is stripped so the tagged visit lands on `/dostava/`): ```text /dostava/ 2 4 1 1 /proizvodi/jakne/ 7 4 0 1 ``` If you would rather not use a terminal, the [log analyser](https://getreport.app/tools/log-analyser) reads the same log in your browser and names ChatGPT-User, Claude-User, Perplexity-User and the other AI agents separately, with the URLs each fetched most. The file is never uploaded. > **Free tool:** [Log file analysis for SEO: free log analyzer](https://getreport.app/tools/log-analyser): Free log file analysis for SEO in your browser: see what Googlebot and AI crawlers fetch, the 404s they hit and which bots are fake. Nothing is uploaded. ### 3. Make sure the bots are real Any scraper can write "ChatGPT-User" in its user agent, so a surprising count deserves a check on the IP address. OpenAI publishes separate address lists for GPTBot, OAI-SearchBot and ChatGPT-User; the [bot IP verifier](https://getreport.app/tools/ai-bot-ip-verifier) checks an address against the lists operators publish and, for Google and Bing, the reverse DNS test they document. One caution when you read user-triggered agents: OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the request, so a robots.txt block is not proof the agent stayed away. > **Free tool:** [Bot IP verifier: real GPTBot, ClaudeBot or Googlebot?](https://getreport.app/tools/ai-bot-ip-verifier): Free bot IP verifier: paste an address and see if it is really GPTBot, ClaudeBot, PerplexityBot or Googlebot, with the published range as proof. ### 4. Count AI referral visits by landing page The visit step lives in your analytics. In GA4, put visits whose session source is chatgpt.com, perplexity.ai, claude.ai, copilot.microsoft.com or gemini.google.com in a channel of their own, then break it down by landing page. OpenAI's publishers FAQ says ChatGPT adds `utm_source=chatgpt.com` to the links it shows, so those visits carry the source even when the referrer is stripped. The [AI referral tracking](https://getreport.app/tools/ai-referral-tracking) generator writes the channel group rule, the Exploration filter and the Looker Studio formulas for you, in your browser. > **Free tool:** [AI referral tracking for GA4: a ready channel group](https://getreport.app/tools/ai-referral-tracking): Free AI referral tracking for GA4: a channel group regex for ChatGPT, Perplexity, Claude, Gemini and Copilot visits, plus Looker Studio formulas to copy. Read this number as a floor. A visit that reaches you with no referrer and no tag is counted as Direct, and no rule recovers it. Clicks from Google's AI Overviews and AI Mode arrive with an ordinary google.com referrer, so they hide among classic search clicks. ### 5. Look at the citations when you ask The citation step leaves no trace in your data when nobody clicks, so the only way to see it is to ask. Keep this to a handful of questions per important page, asked the way a customer would, and treat each answer as one sample of something that changes between runs, users and locations. The [AI citation checker](https://getreport.app/tools/ai-citation-checker) does this for one page at a time. It fetches your page once through the same guarded fetcher our reports use, turns its title into a question, asks Claude with web search for the country you pick, and lists every cited source with your page's place marked. Only Claude is asked for now. Answers are cached and shared for the calendar week, so the same question asked again that week returns the same answer, and each connection gets 3 asks a day, shared with the AI brand mention checker. The answers are paid for from donations, under a daily spending cap. > **Free tool:** [AI citation checker: is your page cited by AI?](https://getreport.app/tools/ai-citation-checker): Free AI citation checker: we turn your page title into the question it answers, ask Claude with web search, and show if your page is cited and who is instead. The tool page's example is a recorded fixture result, not a live answer: for `https://www.example-shop.hr/vrt`, titled "Garden tools & seeds | Example Shop", the checker asked "What should I know about garden tools & seeds?" on 8 October 2026. Claude cited two sources, a fictional competitor first and the example-shop.hr page second, and named Example Shop in the text. In the log terms above, that answer would leave a trace only if someone clicked the second link. ## How to read the signals together Put the per-page counts from steps 2 and 4 side by side and the pattern points at the step that fails. These are ordered by how often you will see them on a small or medium site. | Pattern for a page | Most likely meaning | What to do | | --- | --- | --- | | Training hits only, no search crawls | Copied for training, not in any AI search index you can see | Check robots.txt and firewall for OAI-SearchBot, Claude-SearchBot and PerplexityBot | | Search crawls, no user fetches, no visits | Indexed but not used, or used without clicks | Ask the page's question once; check that the first lines answer it | | AI referral visits, no user fetches | Cited from the index (normal for ChatGPT search) | Nothing to fix; watch the landing pages and what visitors do | | User fetches, no visits | Read during conversations, the link not clicked or not shown | Make the page's unique facts (prices, times, numbers) easy to quote | | No AI requests of any kind | Not eligible, or not discovered | Run the AI crawler check; confirm the page is in your sitemap and indexed in Bing and Google | A page that keeps getting Perplexity-User or ChatGPT-User fetches is one people ask about through AI. That is demand you can see nowhere else, and it tells you which pages to keep accurate first. ## What passing looks like For each page you care about: every AI user agent gets a 200 with the full text (step 1), the search crawlers of the products your customers use appear in the log within a few weeks (step 2), the addresses check out (step 3), and at least some AI referral visits land on it (step 4). User-triggered fetches are a bonus, not a requirement, since an index citation needs none. Rerun the script monthly over the same log window and compare page by page; one page moving from "search crawls only" to "visits" is the result you want to see. ## How we tested The getReport team ran every command in this guide on 16 October 2026, with mawk 1.3.4 and curl 8.5.0 on Linux, against our fixtures: the example-shop.hr access log (3,000 generated requests, 1 to 14 September 2026, in nginx combined format with request time) and the demo shop fixture of example-shop.hr served locally on port 4101. The positive control used three hand-written log lines appended to a copy of that log, as described in step 2. The citation example is the recorded fixture behind the AI citation checker's tool page (`fixtures/examples/ai-citation.json`), not a fresh call. Tool behaviour (caching, limits, engines) was read from the tool pages and the engine code in our repository. What we did not test: we did not query ChatGPT, Claude, Perplexity or Google, did not run the citation checker live, and did not verify the fixture's IP addresses against the operators' published lists. The fixture log is synthetic and contains no user-triggered fetches or AI referrals; real logs from real sites will differ. User agent names and robots.txt behaviour were checked against the operators' documentation as of October 2026 and change often. ## Questions people ask ### What is LLM visibility? LLM visibility is how much products built on large language models, such as ChatGPT, Claude, Perplexity and Google's AI Mode, use your site: which pages they may read, fetch, cite and send visitors to. It is measured with your server log (AI crawler and user-triggered requests), your analytics (AI referral visits) and the citations you see when you ask. Reading it per page shows where each page drops out. ### What does ChatGPT-User mean in my access log? It means a ChatGPT conversation opened that page on a person's behalf at that moment, for example because they asked about it or pasted the link. OpenAI says the agent does not crawl automatically and does not decide whether a page appears in ChatGPT search, which is OAI-SearchBot's role. Verify the IP address against OpenAI's published list before trusting a large count. ### Can robots.txt block ChatGPT-User or Perplexity-User? Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person starts the request, and Perplexity describes Perplexity-User as fetching on a user's behalf. Anthropic says Claude-User follows robots.txt. If you must stop a user-triggered agent, block it at the firewall or CDN, and accept that the page then cannot be read in those conversations. ### Why do I see GPTBot in my logs but no visits from ChatGPT? Because GPTBot collects pages for model training and has nothing to do with answers or links. ChatGPT search relies on OAI-SearchBot, and visits come only when an answer links your page and someone clicks. Check that OAI-SearchBot is allowed and appears in your log, then look for `utm_source=chatgpt.com` visits by landing page in your analytics. ### Does Google AI Mode show up in my server logs? Not as a separate agent. AI Mode and AI Overviews answer from Google's search index, so the only Google requests in your log are ordinary Googlebot crawls, identical for classic results and AI answers. Google-Extended is a robots.txt token without its own user agent. Per-page data for Google's AI features comes from Search Console instead of your log. ### Is LLM visibility the same as AI visibility? They describe the same thing from different angles, and people use the two terms interchangeably. LLM visibility stresses the products built on language models, AI visibility the wider set of AI features, including Google's. What matters more than the label is the level: a site total hides which pages do the work, so measure each layer per page. ## Sources - [OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)](https://developers.openai.com/api/docs/bots) (accessed October 2026) - [OpenAI Help Center: Publishers and developers FAQ (utm_source=chatgpt.com)](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq) (accessed October 2026) - [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (ClaudeBot, Claude-SearchBot, Claude-User)](https://support.claude.com/en/articles/8896518) (accessed October 2026) - [Perplexity: Perplexity crawlers (PerplexityBot, Perplexity-User)](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) (accessed October 2026) - [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) (accessed October 2026) - [Google: Google's common crawlers (Google-Extended token)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) (accessed October 2026)