How to detect AI crawlers on your website using server logs

To detect AI crawlers on your website, read your server's access log: every request from GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and the rest is there, with the page it fetched and the status it got, identified by a name in the user agent. Analytics tools that run in the browser will never show them, because crawlers do not run your tracking script. This guide lists the AI crawler names to look for, shows how to count them and measure their share of your traffic, how to check that they are genuine, and what the numbers tell you. It is part of the guide to Googlebot and crawling, which covers search crawlers too.

Quick answer

• Use the access log, not analytics. JavaScript analytics only records visitors whose browser runs the tag; crawlers do not.
• Search the user agent for the documented names: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot and others.
• Split them by purpose: training crawlers, AI search crawlers and user-triggered fetchers mean different things for your site.
• Measure share two ways: requests and bytes, each as a percentage of the total.
• Verify before you conclude. User agents can be faked; OpenAI, Perplexity and others publish IP ranges to check against.
• Or drop the log into the free log file analyser, which identifies AI crawlers in your browser without uploading the file.

Why analytics does not show AI crawlers

Google Analytics and most privacy-friendly alternatives count a visit when a script runs in the visitor's browser. AI crawlers download your HTML and leave; they do not execute the tag, so they never appear. Many analytics tools also filter known bots on purpose.

Your web server, CDN or host logs every request before any script runs. That is where AI crawlers show up, and it is the only complete record. If you have not opened an access log before, reading Apache and nginx access logs shows where it lives and what each field means.

The AI crawlers to look for

Each operator documents the name its crawler sends. Match these tokens in the user agent, case-insensitively:

Two tokens never appear in a log. Google-Extended and Applebot-Extended are robots.txt switches, not crawlers: Googlebot and Applebot do the fetching, and the token only decides whether the content may be used for training. Google's AI Overviews and AI Mode come from ordinary Googlebot crawling, so they have no separate crawler to count.

Lists like this go out of date. The operators' own crawler pages are the reference, and new names appear every few months; watch your log's unknown bots for them.

What each kind of visit tells you

• Training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider) collect pages for future models. They send no visitors back.
• AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the indexes AI answers cite. Their visits are a precondition for being cited.
• User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) load a page because a person asked an assistant something that led there. They are the closest thing to a signal of real interest, and OpenAI and Perplexity say their fetchers may not follow robots.txt, because a person started them.

How to count AI crawlers from the shell

With a combined-format log, these commands answer the basic questions. Adjust the file name; for rotated logs, combine them first or use zgrep.

BOTS='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Meta-ExternalAgent|Amazonbot|Bytespider|CCBot|DuckAssistBot|MistralAI-User'

# Requests per AI crawler
grep -oE "$BOTS" access.log | sort | uniq -c | sort -rn

# Requests per day for one crawler
grep 'GPTBot' access.log | awk '{print substr($4, 2, 11)}' | uniq -c

# The pages a crawler fetched most
grep 'ClaudeBot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# Status codes a crawler received
grep 'PerplexityBot' access.log | awk '{print $9}' | sort | uniq -c | sort -rn

Measuring their share of your traffic

"Share" can mean requests or bandwidth, and they differ: a crawler fetching large pages or images costs more than its request count suggests. Compute both:

# Share of all requests
total=$(wc -l < access.log)
ai=$(grep -cE "$BOTS" access.log)
echo "AI crawlers: $ai of $total requests ($((100 * ai / total))%)"

# Share of bytes sent (field 10 in the combined format)
awk -v re="$BOTS" '{b = ($10 == "-") ? 0 : $10; all += b; if ($0 ~ re) ai += b}
  END {printf "AI crawlers: %.1f%% of bytes\n", 100 * ai / all}' access.log

Compare the result with the same figures for Googlebot and Bingbot. Behind a CDN that caches pages, remember that cached answers never reach your server, so the origin log undercounts every crawler.

Detect them without the shell

Drop the access log into the log file analyser. It recognises 17 AI crawlers, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent, Amazonbot, Bytespider and CCBot, alongside search engines, SEO tools and monitors. For each it shows hits, days active, distinct URLs and the pages it fetched most, plus the status codes it received. User agents that look like bots but match no known name are listed as unknown, so new crawlers stand out. The file is read in your browser and never uploaded.

Then compare what you found with what you allow. The AI crawler check reads your robots.txt the way each of 18 AI crawlers does:

Check that they are real

A request that says "GPTBot" may not be from OpenAI. Scrapers borrow well-known names to avoid blocks, just as they borrow Googlebot's. Verify by address:

• Published IP lists. OpenAI publishes a JSON file of addresses for each of its bots; Anthropic, Perplexity, Common Crawl, DuckDuckGo and Mistral publish lists too. Check the addresses from your log against them.
• Reverse DNS, for operators that document it: Amazonbot resolves under crawl.amazonbot.amazon, Applebot under applebot.apple.com. Always confirm with a forward lookup, as described in how to verify Googlebot.
• No list published, as for Meta and ByteDance: treat volumes as claims, and look at the addresses' network owner if the numbers matter.

Be aware of the reverse problem too: crawlers that do not announce themselves. A client that sends an ordinary browser user agent but requests hundreds of pages a minute, loads no images or scripts and sends no referrer is behaving like a crawler. The log shows it as a "human"; its pattern gives it away.

If you use a CDN

CDNs see requests before your server does, including the ones they answer from cache. Cloudflare's AI Crawl Control dashboard (formerly AI Audit) shows AI crawler requests by operator, with status codes and popular paths, and lets you block individual crawlers. Other CDNs offer bot analytics on some plans. Where available, these numbers are more complete than your origin log.

What to do with the numbers

1. Compare with your policy. Decide per purpose which AI crawlers you want, as described in AI crawlers and robots.txt, and write it into robots.txt.
2. Look for disobedience. A crawler you disallowed that still gets 200 responses weeks later is ignoring robots.txt, or is an impostor. A crawler that only fetches /robots.txt is obeying you.
3. Check what the search crawlers get. OAI-SearchBot or PerplexityBot receiving 403s or challenge pages means a firewall is blocking the bots that could cite you.
4. Watch the user-triggered fetchers. Rising ChatGPT-User or Perplexity-User hits on a page mean people are asking assistants about it.
5. Rate-limit, don't guess. If a crawler overloads your server, rate-limit or block it at the CDN or firewall; robots.txt is a request, not a lock.

For the full SEO workflow, including Googlebot's wasted crawl, see SEO log file analysis.

Common mistakes

• Looking in analytics. Crawlers do not run the tag; the numbers there are zero or filtered.
• Counting Google-Extended. It is a robots.txt token with no user agent.
• Treating all AI bots as one number. A spike in ChatGPT-User and a spike in Bytespider mean opposite things.
• Believing every user agent. Verify the heavy hitters against published IP lists.
• Reading a CDN-cached site's origin log as complete. Cached requests are missing.

Questions people ask

Why doesn't Google Analytics show AI crawler visits?

Because Google Analytics counts a visit only when its JavaScript tag runs in the visitor's browser, and AI crawlers do not run scripts; they download the HTML and move on. Analytics also filters known bots. The requests are recorded in your server's access log and your CDN's logs, which is where you need to look for GPTBot, ClaudeBot and the others.

How can I tell if ChatGPT is reading my website?

Search your access log for OpenAI's user agents: GPTBot for training, OAI-SearchBot for the ChatGPT search index, and ChatGPT-User for pages fetched because a user asked. ChatGPT-User visits are the clearest sign that someone's question led ChatGPT to your page. Check the addresses against OpenAI's published IP lists, since the names can be faked.

How much of my traffic comes from AI bots?

Only your logs can tell you, and it varies widely by site. Count AI crawler requests as a share of all requests, and the bytes they received as a share of all bytes sent, because the two can differ a lot. Compare the figures with Googlebot's. Behind a caching CDN, use the CDN's analytics, as your origin log misses cached responses.

Can AI crawlers hide from my server logs?

They cannot avoid being logged, because every request reaches the log, but they can avoid being identified. Documented AI crawlers announce themselves in the user agent. Scrapers sometimes send an ordinary browser user agent instead, so they look like visitors. Their pattern gives them away: many pages per minute, no images or scripts, and no referrer.