SEO log file analysis means reading your web server's access log to see what Googlebot, Bingbot and AI crawlers actually requested, and what your server answered. Search Console shows what Google did, sampled and days later; the log has every request as it happened, from every crawler, with the real status code. In about twenty minutes it answers three questions: where Googlebot wastes its visits, which "Googlebots" are fake, and which AI crawlers read your content. This guide walks through it step by step. For how Google's crawler works in general, start with our guide to Googlebot and crawling.
Quick answer
- Get the log, including the rotated
.gzfiles: at least two weeks, ideally a month. - Drop it into the log file analyser. It is parsed in your browser; nothing is uploaded.
- Verify Googlebot and Bingbot. Anything marked fake is a scraper; block it at the firewall or CDN, not in robots.txt.
- Fix the crawl waste in order: 404s (410 and out of the sitemap), redirects (link to the final URL), parameter URLs (canonical or
Disallow), 5xx bursts. - Check AI crawlers against the policy in your robots.txt.
- Export the CSVs, fix, and compare next month's log.
What log file analysis shows that nothing else does
Every other SEO tool estimates. A crawler you run simulates a bot, Search Console samples, and analytics only sees visitors who run JavaScript, which bots do not. The access log is the record: one line per request. Three things only it can show:
- Crawl waste. Every fetch that ends in a 404, a redirect hop or a
?orderby=priceduplicate is not spent on a page you want ranked. On a 300-page site this hardly matters; on a shop with 20,000 products and faceted filters it decides whether new products get crawled this week or next month. Our guide to crawl budget explains when that limit is real. - Impostors. "Googlebot" is just a string in a request header, and scrapers use it because many sites let Googlebot through their rate limits. Only the IP address can be verified.
- AI crawlers. GPTBot, ClaudeBot, PerplexityBot and others fetch pages for training, for AI search, or because a user asked an assistant to open your URL. robots.txt says what you allow; the log shows whether they listened.
What a log line contains
Apache and nginx write one line per request in the combined format unless someone changed it:
66.249.66.1 - - [25/Sep/2026:06:12:44 +0200] "GET /shoes/alpina-pro/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.6668.100 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"For SEO, four fields matter: the client IP (the only part you can verify), the requested URL with its query string, the status code, and the user agent (who the client claims to be). Many servers can add the response time at the end. Reading Apache and nginx access logs explains every field and how to add response time.
How getReport checks it
The tool is a page in your browser, not a server-side job. A Web Worker streams the file line by line, so a 300 MB log does not freeze the tab, and only totals are kept; under the drop zone it says "Processed in your browser. Your file is never uploaded." A Try the sample log button shows the result before you find your own file. It reads Apache and Nginx common and combined formats (vhost prefix and request-time fields included), JSON lines and W3C extended logs with a #Fields: header (IIS); gzipped files are unpacked in the browser.
Every line is attributed by user agent: the Googlebot family and the other search engines, 17 AI crawlers from GPTBot and ClaudeBot to CCBot, SEO tools, social link previews and uptime monitors. Anything else with "bot", "crawl", "spider" or a library name in its user agent is listed as an unknown bot; the rest is human.
Below the table: per-day charts, the Where Googlebot's hits went panel, per-bot tables of most fetched URLs and 404s, an AI crawlers panel and CSV downloads. The log shows what crawlers did; these report checks show what your site asks them to do:
Step by step
1. Get the log, including the rotated files
| Hosting | Where |
|---|---|
| cPanel | Metrics → Raw Access |
| Plesk | Websites & Domains → Logs |
| Kinsta, WP Engine, SiteGround | A logs screen in the host dashboard |
| Your own nginx | /var/log/nginx/access.log plus access.log.1, access.log.2.gz, … |
| Your own Apache | /var/log/apache2/access.log (Debian, Ubuntu) or /var/log/httpd/access_log (RHEL) |
| Cloudflare | Edge logs need Logpush on a paid plan; the origin log is still yours |
Logs are rotated daily or weekly and many hosts keep only a few weeks, so download now. Combine a month into one file:
cat access.log access.log.1 > /tmp/month.log
zcat access.log.*.gz >> /tmp/month.log
grep Googlebot /tmp/month.log | awk '{print $1}' | sort | uniq -c | sort -rn | headThe last line prints the ten addresses that claimed to be Googlebot most often.
2. Drop the file in and read the summary
Open the log file analyser and drop the file in; gzipped files work as they are. The summary says how many lines were read and how many did not parse: a handful is normal, thousands mean a custom log format.
Six tiles follow: human visits, bot hits, Googlebot hits with distinct URLs, Googlebot 404s (amber above 5% of its hits), AI crawler hits, and 5xx answers (red if any). Bots outnumbering humans is normal for a small site; one unknown bot outnumbering Googlebot is not. In the bots table, a high share of parameter URLs means a crawler is stuck in your filters, and a slow average response time means your server is holding the crawl back: Google crawls less when answers take long.
3. Verify Googlebot and Bingbot
Click Verify Googlebot & Bingbot addresses. The tool sends only the IP addresses (at most 200 per bot, ranked by hits) to our API, which runs the check Google and Microsoft document: a reverse DNS lookup, then a forward lookup of the returned name, which must resolve back to the same address. Real Googlebot resolves under googlebot.com or google.com, real Bingbot under search.msn.com. Verification is rate limited to 20 requests per 10 minutes.
The verification column then reads real, fake and unknown hits per bot. Unknown means no reverse record, which is common for cloud ranges and proves nothing. Fake addresses are listed in a red panel:
Block or rate-limit them at the firewall or CDN, never in robots.txt: a fake bot ignores it, and a User-agent: Googlebot block there hides the site from the real one. For a firewall allow-list, Google publishes its ranges as JSON under developers.google.com/static/crawling/ipranges/: common-crawlers.json (renamed from googlebot.json in 2026), special-crawlers.json and the user-triggered fetcher files. How to spot a fake Googlebot covers the manual checks.
4. Open "Where Googlebot's hits went" and fix the waste
Bars turn amber above 3% and red above 10%, except static files, which stay grey because Googlebot needs your CSS and JavaScript to render pages. Take the wastes in order:
404 and 410 pages. Select Googlebot in the URL tables and read "404s hit by Googlebot". Paste the list into the bulk URL checker to see which still answer 404. Pages that had links or traffic get a 301 to the closest live equivalent. Pages gone for good answer 410, which Google retries less often, and come out of the sitemap:
# Nginx server block: gone for good
location = /summer-sale-2023/ { return 410; }Redirects. A crawler sent from /product to /product/ has spent two fetches on one page. Point internal links and sitemap entries at the final URL, and collapse stacked rules (http → https → www → trailing slash) into one hop, as in redirects without chains.
Parameter URLs. Sort, filter and tracking parameters multiply one category into hundreds of URLs. Set the canonical tag on every variant to the clean URL, then block the parameters that never produce a page worth indexing:
# robots.txt at the site root
User-agent: *
Disallow: /*?orderby=
Disallow: /*&orderby=
Disallow: /*?replytocom=Blocking in robots.txt stops crawling, not the indexing of URLs already known; keep the canonical for those. The robots.txt learn page covers the matching rules.
5xx answers. The panel shows the share; the daily chart shows when. A burst at the same hour every night is a backup or cron job competing with the crawler, and Google crawls less for days afterwards.
5. Check the AI crawlers against your policy
The AI crawlers panel shows each one with hits, days active, distinct URLs and its most fetched paths. Three kinds show up, and each deserves its own decision:
- Training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider) feed models.
- Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) feed answer engines that cite and link.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) open one page because a person asked.
Write your decision into robots.txt and confirm it with the AI crawler check. A crawler you blocked that still gets 200 answers is not obeying, and needs a firewall rule; one that fetches only /robots.txt is respecting it. Detecting AI crawlers in server logs goes further, including how to measure their share of traffic.
6. Export, fix, compare next month
Download the bots CSV and the selected bot's URLs CSV: they are your baseline and the list you hand to the developer. In four weeks, run the new log and compare: Googlebot's 404 and redirect shares should fall and its distinct URL count rise while hits hold. That is crawl activity moving from waste to pages.
Platform notes
WordPress. Two parameters are almost always near the top of a WordPress log: ?replytocom= on comment reply links and ?add-to-cart= on WooCommerce listings; both belong in robots.txt as in step 4. Page caches such as WP Rocket or LiteSpeed Cache do not change the log, since a cached page is still a request to your server.
Shopify. Shopify gives merchants no raw access logs. The closest source is Search Console → Settings → Crawl stats, which shows Googlebot's requests by response, file type and purpose for the last 90 days, but neither AI crawlers nor fake Googlebots.
Behind a proxy or CDN. Log the visitor's address rather than the proxy's, or every Googlebot verifies as unknown: on nginx, real_ip_header X-Forwarded-For; with set_real_ip_from for the proxy's range. A CDN that caches HTML answers many bot requests at the edge, so the origin log undercounts them; use the CDN's log export where your plan allows it.
Verify
- Search Console's Crawl stats report agrees in shape: the same daily pattern and a similar split between 200, 301 and 404. Exact numbers differ, because Google counts all its crawlers together.
- Next month's log shows a lower 404 and redirect share in Where Googlebot's hits went, and the 404 tile drops below 5%.
- Fake addresses now get
403instead of200.
Common mistakes
- Trusting the user agent string. "Googlebot" from an address that also probes
/wp-login.phpand/.envis not Google. Verify, then block at the firewall. - Blocking Google's own ranges by accident. Crawl stats collapse the day after a firewall change. Allow-list Google's published ranges before any user-agent rule.
- Forgetting the rotated
.gzfiles. The charts cover three days and every share is noise. - Analysing a single day. Googlebot's behaviour varies with the weekday and your deploys; read at least two weeks and compare the same span month to month.
Questions people ask
What is log file analysis?
Log file analysis is reading the access log your web server writes, one line per request, to see who requested what and what the server answered. In SEO it means filtering those lines to search engine and AI crawlers, so you can see which URLs Googlebot actually fetches, which errors and redirects it hits, and whether bots claiming to be Google really are. Nothing is estimated or sampled.
Which log files do I need for an SEO log analysis?
The raw access log of the web server that answers your domain, in its original format, for at least two weeks and ideally a month, including the rotated and compressed older files. Error logs are not needed. If a CDN caches your HTML, its edge logs complete the picture, because those requests never reach your server. Analytics exports cannot replace logs, since bots do not run JavaScript.
How do I filter Googlebot in a log file?
Match "Googlebot" in the user agent field, for example with grep Googlebot access.log, and then verify the IP addresses, because anyone can send that string. The user agent tells you which crawler variant claims to be asking; only a reverse and forward DNS lookup, or Google's published IP ranges, confirm it is Google. The log file analyser does both steps and keeps the rest of the file in your browser.
Can log file analysis show pages Googlebot never crawls?
Yes, by comparison. Export the list of URLs Googlebot fetched over a month and compare it with your XML sitemap: sitemap URLs that never appear in the log are pages Google is not visiting. They are usually deep in the site, weakly linked or new. Link them from pages Googlebot fetches often, and check they answer 200 and are not blocked by robots.txt.
How do I count Googlebot visits per page from my logs?
Filter verified Googlebot lines, then count requests per URL and the days between them. In a shell, grep Googlebot, extract the request path and pipe it through sort and uniq -c. The log file analyser shows the same as a most-fetched-URLs table per bot. Pages fetched daily are the ones Google considers important or fresh; pages missing for weeks need better internal links.
How long should I keep log files for SEO analysis?
Keep at least 30 days and ideally 90, so you can compare crawling across releases instead of one week. Many hosts rotate logs after 7-14 days, so download them regularly. Logs contain IP addresses, which are personal data under the GDPR, so set a retention period, limit who can read them and delete them once the analysis is done.