Log file analysis: see what Googlebot and AI crawlers do on your site

Check the IP address, not the user agent, which anyone can send. Google's documented test: the IP must reverse-resolve to a googlebot.com or google.com name, and that name must resolve back to the same IP. Bing documents the same with search.msn.com. This analyser runs the check for the Googlebot and Bingbot addresses in your log with one click and marks each verified, fake or unknown.

Drop your access log above: the analyser counts Googlebot's hits per day and the status codes it received, separately from Googlebot Image, AdsBot and the other Google crawlers. Verify the addresses with the fake Googlebot check, since many scrapers borrow its name. Search Console's Crawl stats report shows the same trend for the last 90 days, without your log.

GPTBot, ClaudeBot, PerplexityBot, Amazonbot, Bytespider, CCBot and other AI crawlers, each with its hits, the URLs it fetched most and how much of your content it reads. User agents that match no known bot are listed as unknown, so new crawlers stand out. User agents can be faked, so treat large volumes from an unfamiliar name with care.

There is no fixed limit, but a useful rule of thumb is to act when more than a tenth of Googlebot's hits go to 404s, redirects or parameter URLs, especially on large sites. The analyser shows each bot's share of wasted crawl and the 404s it keeps hitting. Fix internal links to them, redirect removed pages that have links, and stop generating endless parameter URLs.

Googlebot Smartphone sends a Chrome on Android user agent that contains "compatible; Googlebot/2.1; +http://www.google.com/bot.html", and the desktop crawler sends a similar string with a desktop browser. The Chrome version in it changes over time. Because any scraper can copy the string, a user agent alone does not prove a visit is from Google; use the fake Googlebot check to verify the IP.

Get the raw access logs from your host, CDN or server, ideally a few weeks' worth, then filter the requests by crawler and look at what each fetches and which status codes it gets. Drop the file here and this analyser does that in your browser: hits per bot and per day, top URLs, 404s and redirects, AI crawlers and fake Googlebots. Search Console's Crawl stats cover only Google, with example URLs rather than every request.

No, and the tool is free with no limit. The file is read by a Web Worker in your browser and the page only receives the totals. The single exception is the optional bot verification: when you click it, the IP addresses that claimed to be Googlebot or Bingbot, at most 200 per bot, go to our API for the reverse-DNS check, nothing else.

Apache and Nginx "common" and "combined" formats (with or without a virtual host in front, with extra fields such as request time after the user agent), JSON lines with the usual field names, and W3C extended logs from IIS. Gzipped files (.gz) are unpacked in the browser. Lines that do not parse are counted, not silently skipped.