Log analyser:What do Googlebot and AI crawlers really do on your site?
Drop an access log and see which bots crawl you, what they fetch, how much of their crawl goes to 404s and redirects, and whether your Googlebot is the real one.
Processed in your browser. Your file is never uploaded. Apache, Nginx, IIS, JSON lines, .gz · up to about 500 MB
- Free, no account
- Runs in your browser
- Apache, Nginx, IIS and JSON logs
What this tool checks
Everything is counted in your browser, in a background thread. The file is never uploaded.
Every bot by name
Google (Googlebot, Image, Video, AdsBot, InspectionTool, GoogleOther), Bing, Apple, DuckDuckGo, Yandex, Baidu, SEO tools, link previews and uptime monitors. Other bots are listed as unknown.
AI crawlers
GPTBot, ClaudeBot, PerplexityBot, Amazonbot, Bytespider, CCBot and more, with how much of your content each one reads.
Hits, days and status codes
Hits and share per bot, hits per day, and the status codes each bot received.
Wasted crawl
The 404s a bot keeps hitting and how much of its crawl went to redirects, parameter URLs and static files.
Top URLs and response time
The URLs each bot fetched most, and the average response time when the log carries it.
Fake Googlebot check
One click checks whether Googlebot and Bingbot addresses are genuine with the reverse-DNS lookup Google and Microsoft document: verified, fake or unknown.
The common log formats
Apache and Nginx common and combined, JSON lines (Nginx, Cloudflare, most log shippers) and W3C extended (IIS). Gzipped files are unpacked on the fly.
Show 3 more checks
Top URLs and response time
The URLs each bot fetched most, and the average response time when the log carries it.
Fake Googlebot check
One click checks whether Googlebot and Bingbot addresses are genuine with the reverse-DNS lookup Google and Microsoft document: verified, fake or unknown.
The common log formats
Apache and Nginx common and combined, JSON lines (Nginx, Cloudflare, most log shippers) and W3C extended (IIS). Gzipped files are unpacked on the fly.
How it works
Drop your log
Drop an access log file, plain or .gz. The format is recognised automatically.
Read in your browser
The file is read line by line in a background thread, so a large log does not freeze the page. Every hit is attributed to a bot by its user agent.
Verify the bots
Optionally, only the IP addresses that claimed to be Googlebot or Bingbot go to our server for the reverse-DNS check. Nothing else is sent.
Files of a few hundred megabytes work in a current desktop browser. Very large logs are best split by day or month first.
Questions
Is this really free?
Yes. getReport is funded by donations, not plans. This tool runs entirely in your browser, so there is no limit and nothing to pay for.
Is my log really not uploaded?
Correct. The file is read by a Web Worker in your browser and the page only receives the totals. The single exception is the optional bot verification: when you click it, the IP addresses that claimed to be Googlebot or Bingbot (at most 200 per bot) are sent to our API for the reverse-DNS check, nothing else.
Which log formats work?
Apache and Nginx "common" and "combined" formats (with or without a virtual host in front, with extra fields such as request time after the user agent), JSON lines with the usual field names, and W3C extended logs from IIS. Gzipped files (.gz) are unpacked in the browser. Lines that do not parse are counted, not silently skipped.
How big a file can I analyse?
Files of a few hundred megabytes work in a current desktop browser; the analysis streams the file and keeps only bounded counters, so memory does not grow with the log. Very large logs are best split by day or month first.
Why read the log and not just Search Console?
Search Console shows you a sample of what Google did, days later, for one property. The access log is the complete record: every crawler, every URL, every status, the same day. It is the only place you can see crawl budget wasted on redirect chains, faceted URLs and long-dead pages, see which AI crawlers read your content and how much, and catch scrapers that pretend to be Googlebot. Your browser can do the job: the file stays where it is, and the numbers that matter fit on one screen.
What does "fake Googlebot" mean?
Anyone can send the Googlebot user agent. Google publishes how to tell the real one: the IP address must reverse-resolve to a googlebot.com or google.com name, and that name must resolve back to the same IP. Bing documents the same for search.msn.com. Hits that fail this check are scrapers or testing tools; block or rate-limit them, and never build robots.txt decisions on them.
Related free tools
All 41 tools →Free, funded by the people who use it
€0 of €75 this month. At €75, site crawl up to 500 pages + weekly re-check switches on for everyone.