# Googlebot and crawling: how search and AI bots crawl your site

> Googlebot is the crawler Google Search uses to fetch web pages. How it finds, fetches and renders your pages, how Bingbot and AI crawlers differ, and what your server logs show about all of them.

Updated 2026-09-26 · Technical SEO · HTML version: https://getreport.app/guides/googlebot-and-crawling

Googlebot is the web crawler Google Search uses to find and download pages: a program that follows links and sitemaps, fetches each URL, renders it like a browser and hands the result to Google's index. If Googlebot cannot reach a page, or only sees an empty shell of it, that page cannot rank. This guide is the map of the whole topic for site owners and SEOs: how Googlebot works, how Bingbot and the AI crawlers differ, how to tell a real Googlebot from a fake, and how to see in your own server logs what every bot actually fetched.

## Quick answer

- **Googlebot is two crawlers:** Googlebot Smartphone, which does most of the crawling because Google indexes the mobile version of your pages, and Googlebot Desktop. Both obey the `Googlebot` rules in robots.txt.
- **Crawling is not indexing.** Googlebot fetches a page; Google then decides whether to index it. A crawled page can still be left out of search.
- **It reads only the first 2 MB** of an HTML file (64 MB of a PDF), runs JavaScript in an up-to-date Chromium, and starts every page with no cookies.
- **Speed and errors set the pace.** A fast, error-free server gets crawled more; 5xx errors and 429 responses make Google slow down.
- **Anyone can pretend to be Googlebot.** Only a reverse-DNS check of the IP address (or Google's published IP ranges) proves a visit is real.
- **Your access log is the only complete record** of what Googlebot, Bingbot and AI crawlers requested. Drop it into the free [log file analyser](https://getreport.app/tools/log-analyser); it runs in your browser and the file never leaves your device.

## What is Googlebot?

Googlebot is the generic name for Google Search's main web crawler. Google's documentation describes it as two crawlers that share one robots.txt token:

- **Googlebot Smartphone**, a mobile crawler that identifies itself as Chrome on an Android phone.
- **Googlebot Desktop**, a desktop crawler that identifies itself as desktop Chrome.

Both send `Googlebot/2.1` in their user agent and follow the rules you write for `User-agent: Googlebot`. You cannot give them different robots.txt rules.

Around Googlebot sits a family of other Google crawlers, and they show up in your logs under their own names. Google groups them in three kinds:

| Kind | Examples | Obeys robots.txt? | What it is for |
| --- | --- | --- | --- |
| Common crawlers | Googlebot, Googlebot-Image, Googlebot-Video, Storebot-Google, GoogleOther, Google-InspectionTool | Yes | Search, Images, Video, Shopping, research crawls, the URL Inspection tool |
| Special-case crawlers | AdsBot-Google, AdsBot-Google-Mobile, Mediapartners-Google | Yes, but some ignore the `*` group | Ads landing-page quality, AdSense |
| User-triggered fetchers | Google Site Verifier, Google-Read-Aloud, Feedfetcher | Generally no | A fetch a person asked for, such as verifying a site |

`Google-Extended` is on that list too, but it is only a robots.txt token, not a crawler. It controls whether content Googlebot already fetched may be used for Gemini model training and grounding; it has no effect on Search or AI Overviews.

> **Note:**
> The full list, with every user agent string and robots.txt token, is on Google Search Central under "Google's common crawlers", "Google's special-case crawlers" and "Google's user-triggered fetchers". Google updates it when crawlers are added or renamed, so treat any list you copy, including this one, as a snapshot.

## How does Googlebot work?

Crawling is a loop of four steps. Knowing which step went wrong tells you which fix you need.

### 1. Discovery

Googlebot does not know your pages exist until it finds a URL for them. It finds URLs in links on pages it has already crawled, in XML sitemaps you submit or list in robots.txt, in redirects, and in URLs you submit through the URL Inspection tool. A page nothing links to and no sitemap lists is invisible to it.

### 2. Scheduling

Every discovered URL goes into a queue. Google decides when to fetch each one by weighing two things, which it calls crawl capacity and crawl demand. **Capacity** is how much your server can take without slowing down, measured continuously from its response times and errors. **Demand** is how much Google wants the URL: popular pages and pages that change often are fetched more, low-value duplicates less. For most sites, Google's documentation says Googlebot "shouldn't access your site more than once every few seconds on average".

### 3. Fetching

Before crawling a host, Googlebot requests `/robots.txt` and caches it for up to 24 hours. If a URL is disallowed, it stops there. Otherwise it fetches the page over HTTP/1.1 or HTTP/2, accepting gzip and Brotli compression. Two limits matter:

- It reads the **first 2 MB** of an HTML or other supported text file, and the **first 64 MB** of a PDF. Content after the cut-off is not considered for indexing.
- It follows up to **10 redirect hops** in a chain; after that, the URL errors in Search Console.

The status code decides what happens next. A 200 goes on to rendering; a 301 or 308 passes the crawl to the target; a 404 or 410 eventually drops the URL; a 429 or 5xx tells Google your server is struggling, and it crawls more slowly until errors stop.

### 4. Rendering

Google's Web Rendering Service opens the fetched HTML in a headless, evergreen Chromium, runs the JavaScript and reads the resulting page, including links that only exist after scripts run. Rendering can happen later than the fetch, and Googlebot starts each page fresh: no cookies, local storage or session storage from earlier pages, and no clicks, scrolling or permission prompts. Content that needs any of those is not seen. The [guide to JavaScript-rendered content and Google](https://getreport.app/guides/javascript-rendered-content-and-google) covers what breaks and how to fix it.

After rendering, the page goes to indexing, where Google decides whether to store it, which URL is canonical and what the page is about. That step is separate from crawling: pages that were fetched fine can still show "Crawled – currently not indexed". If that is your problem, start with [why a page is not in Google](https://getreport.app/guides/why-is-my-page-not-in-google).

## What is Googlebot Smartphone?

Googlebot Smartphone is the crawler that does most of Google's crawling. Google indexes the mobile version of your content (mobile-first indexing), and since July 2024 it crawls and indexes essentially every site with the smartphone crawler. Googlebot Desktop still visits, but as a minority of requests.

Its user agent looks like this, with `W.X.Y.Z` replaced by the current Chrome version:

```text
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
```

What this means in practice:

- The mobile page is the page Google judges. Text, links, images, structured data and meta tags that exist only on your desktop layout do not count.
- Hiding content behind "Read more" buttons is fine as long as it is in the HTML; content loaded only after a tap is not seen.
- Separate mobile sites (`m.example.com`) need the same content and correct `rel="alternate"` and canonical links; a responsive site avoids the problem altogether.

## Googlebot user agents in your logs

Every Google crawler identifies itself with a user agent string, and a log line tells you which one visited. The two main Googlebot strings contain `Googlebot/2.1`; the others carry their own names, such as `Googlebot-Image/1.0`, `Storebot-Google/1.0`, `AdsBot-Google` or `Google-InspectionTool/1.0`, the last being you or someone else testing a URL in Search Console. Google groups them as common crawlers, which always obey robots.txt, special-case crawlers such as AdsBot, some of which ignore the `*` group, and user-triggered fetchers, which generally ignore robots.txt because a person asked for the page.

The Chrome version inside the Googlebot strings changes as Google updates its renderer, so match on the `Googlebot` token, never on the full string. And since any script can send the same string, a user agent is a claim, not proof. The next section is how you check the claim.

The guide to [Googlebot user agent strings](https://getreport.app/guides/googlebot-user-agents) lists every string with the robots.txt token it obeys, shows the shell commands to split Smartphone from Desktop hits in a log, and gives the Bingbot strings for comparison.

## How to tell a real Googlebot from a fake one

Scrapers, vulnerability scanners and spam bots routinely send Googlebot's user agent, because many sites wave it through firewalls and rate limits. Blocking a fake Googlebot is good housekeeping; blocking the real one removes you from Google. The difference is in the IP address.

Google documents two ways to verify a visitor:

1. **Reverse and forward DNS.** Look up the host name of the IP, check that it ends in `googlebot.com`, `google.com` or `googleusercontent.com`, then look up that host name and check it points back to the same IP.

```bash
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1
```

2. **Published IP ranges.** Google publishes its crawler address ranges as JSON files under `developers.google.com/static/crawling/ipranges/`, split by kind: `common-crawlers.json` (formerly `googlebot.json`) for Googlebot and the other common crawlers, `special-crawlers.json`, and separate files for user-triggered fetchers. Match the IP against the right file.

Bing documents the same test for Bingbot, with host names ending in `search.msn.com`.

The step most people skip is the forward lookup: anyone who controls an IP block can point its reverse record at a Google-looking name, but only Google can make that name resolve back. The guide to [spotting a fake Googlebot](https://getreport.app/guides/verify-googlebot-fake-googlebot) covers both methods in detail, where the IP range files moved in 2026, a script to check addresses against them, and how to block impostors at the firewall or CDN without catching the real crawler. The log file analyser below runs the DNS check for every Googlebot address in a log.

## What controls how much Google crawls

"Crawl budget" is the name for the number of URLs Google can and wants to crawl on your site in a given time, the capacity and demand from the scheduling step above. It is the most over-diagnosed problem in SEO. Google's own guidance on managing crawl budget is written for sites with more than a million pages that change weekly, sites with more than 10,000 pages that change daily, and sites where a large share of URLs sit in "Discovered – currently not indexed". Below that, Googlebot can fetch your whole site easily.

What small sites experience as a budget problem is usually one of three other things: Google does not think a page is worth crawling yet, the server was slow or erroring when Googlebot came, or the crawls went to junk URLs such as redirect chains, filter combinations and old 404s. Only the last one shows up as waste, and only your log shows it clearly.

The guide to [crawl budget and when it matters for your site](https://getreport.app/guides/crawl-budget-for-sites-under-10000-pages) walks through proving whether you are limited at all, and then fixing the waste in order: sitemap first, then redirect chains, then parameter URLs, then internal links to the pages you want crawled. It also covers what not to do, such as adding `Crawl-delay` (Google ignores it) or blocking CSS and JavaScript (Google then cannot render the page).

## How often does Google crawl a site?

There is no fixed schedule. Google crawls each URL as often as its demand justifies and your server allows: a busy homepage may be fetched many times a day, an old blog post once in several months. Two places show you the real pattern. **Search Console → Settings → Crawl stats** shows Google's requests per day over the last 90 days, broken down by response code, file type, purpose (discovery or refresh) and Googlebot type, with host status for robots.txt, DNS and server connectivity. **Your access log** shows every request per URL, so you can count the days between visits to any page.

To get new or changed pages crawled sooner, link them from pages Google already crawls often, list them in your XML sitemap with an accurate `lastmod`, and use **Request indexing** in URL Inspection for a handful of important URLs. Search Console's crawl-rate setting was retired in January 2024; if Googlebot overloads your server, return 503 or 429 for a day or two at most and it backs off automatically.

The guide to [how often Google crawls a site](https://getreport.app/guides/how-often-does-google-crawl) walks through the Crawl stats report row by row, the log commands for per-page crawl intervals, and what actually speeds crawling up or slows it down.

## What is Bingbot, and how do other search crawlers differ?

Bingbot is Microsoft Bing's web crawler, and it does for Bing what Googlebot does for Google. It matters beyond Bing itself: Bing's index also feeds Yahoo, DuckDuckGo's traditional results and several AI assistants that search the web through Bing.

Bingbot sends user agents containing `bingbot/2.0`, in a desktop and a mobile variant, and behaves differently from Googlebot in a few ways:

| | Googlebot | Bingbot |
| --- | --- | --- |
| `Crawl-delay` in robots.txt | Ignored | Honoured |
| Instant URL submission | URL Inspection, one URL at a time | IndexNow API, many URLs at once |
| Crawl rate control | None (retired January 2024) | Crawl control by hour in Bing Webmaster Tools |
| Verification | Reverse DNS to `googlebot.com` / `google.com` | Reverse DNS to `search.msn.com` |

IndexNow is an open protocol Microsoft Bing and Yandex launched in 2021: you ping it with changed URLs and participating engines crawl them soon after. Google does not use it, so it is an addition to your sitemap, not a replacement.

Other search crawlers you will see in a typical log include Applebot (Siri and Spotlight suggestions), DuckDuckBot, YandexBot and Baiduspider. All of them publish their user agents, and most publish IP ranges or a DNS verification method.

## How do AI crawlers fit in?

AI companies run their own crawlers, and on many sites they now make more requests than Bingbot. They fall into three groups with different consequences:

- **Training crawlers** collect pages to train models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance) and others.
- **Search crawlers** build an index an AI assistant searches and cites with links: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- **User-triggered fetchers** load a page because someone asked an assistant about it: ChatGPT-User, Claude-User, Perplexity-User. Some vendors say these may not follow robots.txt, because a person requested the page.

Two things surprise most site owners. First, Google's AI Overviews and AI Mode are built from normal Googlebot crawling, so there is no separate Google AI crawler to allow or block; `Google-Extended` only governs Gemini training and grounding. Second, blocking an AI crawler in robots.txt does not remove content it already collected, and a crawler that ignores robots.txt has to be stopped at the server or CDN.

The guide to [AI crawlers and robots.txt](https://getreport.app/guides/ai-crawlers-and-robots-txt) explains which crawler does what, how to decide per purpose rather than per company, and gives a robots.txt you can copy. The report's AI crawler check reads your robots.txt the same way:

> **Check: AI crawler access.** AI assistants (ChatGPT, Claude, Perplexity) fetch pages with their own user agents. Google's AI Overviews use normal Googlebot, so only blocking Googlebot removes you from them, and that removes you from Search too. Blocking them keeps your content out of AI answers; allowing them can bring citations and visitors. Either is a valid choice, as long as it is the one you meant.
>
> 1. Decide per crawler: training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) feed models; search bots (OAI-SearchBot, PerplexityBot) feed answers that link back; user bots (ChatGPT-User, Claude-User) fetch a page because someone asked. Google-Extended is a robots.txt token, not a crawler: it controls use in Gemini training, not AI Overviews.
> 2. Edit robots.txt: a "User-agent: GPTBot" group with "Disallow: /" blocks it; leave it out of robots.txt (or "Allow: /") to permit it. Rules for "*" apply to any crawler without its own group.

To see which AI crawlers actually visit, look in your access log rather than analytics: crawlers do not run tracking scripts, so browser-based analytics never records them. Each documented crawler names itself in the user agent, and the purpose matters when you read the counts. Training crawlers send nothing back, search crawlers are a precondition for being cited, and a rise in user-triggered fetches such as ChatGPT-User means people are asking assistants about your pages.

Names can be borrowed, so check heavy hitters against the IP lists OpenAI, Anthropic, Perplexity and others publish. The guide to [detecting AI crawlers on your website](https://getreport.app/guides/detect-ai-crawlers-in-server-logs) lists the tokens to search for, the shell commands to count requests and bandwidth per crawler, and what to do when a crawler ignores your robots.txt.

## How do you control what crawlers do?

Four controls, each with a different job. Mixing them up is the most common crawling mistake.

| You want to… | Use | Why not the others |
| --- | --- | --- |
| Stop crawlers fetching a section (cart, internal search, admin) | `Disallow` in robots.txt | It saves crawling but does not remove pages from Google |
| Keep a page out of Google's index | `noindex` meta tag or `X-Robots-Tag` header | Googlebot must be able to crawl the page to see it, so do not also disallow it |
| Consolidate duplicates | `rel="canonical"` or a 301 redirect | Blocking duplicates hides the canonical signal |
| Stop a bot that ignores robots.txt | Firewall, CDN bot rules or rate limits | robots.txt is a request, not a lock |

A URL blocked in robots.txt can still appear in Google if other sites link to it, shown without a description ("Indexed, though blocked by robots.txt"). And a robots.txt file that answers with a server error makes Google pause crawling your site, so it must return 200 (or 404 if you have none).

The guide [robots.txt: test it before it hides your site](https://getreport.app/guides/robots-txt-before-it-hides-your-site) covers what to put in the file and how to test URLs against it the way Googlebot does; the [robots.txt fix page](https://getreport.app/learn/robots-txt) summarises what "OK" looks like. Every getReport report checks the basics:

> **Check: robots.txt is present.** robots.txt tells crawlers which parts of the site to skip and where the sitemap is. Without one, crawlers use their own defaults and may waste time on search results or admin pages.
>
> 1. Publish a plain-text robots.txt at the site root with at least "User-agent: *", "Allow: /" and a "Sitemap:" line.
> 2. Most CMSs generate one; in WordPress it is under the SEO plugin's Tools section.

> **Check: robots.txt allows this page.** A Disallow rule stops search engines from crawling this page, so they cannot read its content or see any updates. The page can still appear in results as a bare URL with no description.
>
> 1. Remove or narrow the Disallow rule in robots.txt so this URL is allowed for all crawlers.
> 2. If the page should stay out of search, allow crawling and use a noindex tag instead; a blocked page cannot be de-indexed.

> **Check: robots.txt declares a sitemap.** A "Sitemap:" line in robots.txt is how crawlers find your XML sitemap without you registering it anywhere. Without it, new pages are discovered later.
>
> 1. Add a line to robots.txt: Sitemap: https://your-site.com/sitemap.xml (use the full URL).

For bots that ignore the rules, the guide to [rate limiting and bot protection on a small site](https://getreport.app/guides/rate-limiting-and-bot-protection-on-a-small-site) shows how to slow them down without ever blocking the real Googlebot.

## What your server logs show about crawlers

Search Console shows Google's view, delayed and sampled. Your web server's access log shows every request from every bot as it happened: the IP address, the time, the URL, the status code, the response size and the user agent. A typical line in the Apache or nginx "combined" format looks like this:

```text
66.249.66.1 - - [26/Sep/2026:08:14:03 +0000] "GET /products/blue-mug HTTP/1.1" 200 18432 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
```

Read left to right: client IP, two unused identity fields, timestamp, request line, status code, bytes sent, referrer, user agent. This is the "combined log format", the default in both Apache and nginx.

Where the file lives depends on the server: `/var/log/nginx/access.log` for nginx, `/var/log/apache2/access.log` or `/var/log/httpd/access_log` for Apache, and a Raw Access or Logs screen on cPanel and Plesk. Old logs are rotated into numbered and gzipped files, so a month usually spans several. Behind Cloudflare or a load balancer, the log records the proxy's address unless you restore the real client IP, which you need before you can verify any crawler. The guide to [nginx and Apache access logs](https://getreport.app/guides/reading-access-logs-apache-nginx) covers finding the file on each setup, every field, adding response times to the format and the shell commands that answer quick questions.

Read in bulk, the log answers questions nothing else can:

- **Where Googlebot's visits go.** The share of its hits on 404s, redirects, parameter URLs and static files, against the pages you want ranked.
- **Whether important pages are crawled at all.** A product that Googlebot has not fetched in 60 days will not show new prices in search.
- **Which AI crawlers read your content,** and how much, measured instead of guessed.
- **Which "Googlebots" are fake.** Scrapers in a Googlebot costume, often the heaviest users of a small server.

The step-by-step guide to [log file analysis for SEO](https://getreport.app/guides/server-log-analysis-googlebot-and-ai-bots) takes you from downloading the log from cPanel, Plesk or your server to a list of fixes in about twenty minutes.

## How to check which bots crawl your site

> **Free tool:** [Log file analysis for SEO: free log analyzer](https://getreport.app/tools/log-analyser): Free log file analysis for SEO in your browser: see what Googlebot and AI crawlers fetch, the 404s they hit and which bots are fake. Nothing is uploaded.

Drop an access log, plain or gzipped, into the log file analyser. It recognises Apache and nginx common and combined formats, JSON lines and IIS W3C logs, and reads the file in a background thread in your browser. **The log file never leaves your device**; the page only receives the totals.

You get:

- every bot by name, from Googlebot, Googlebot-Image, AdsBot and GoogleOther to Bingbot, Applebot, SEO tools and uptime monitors, with hits, hits per day and the status codes each received;
- the AI crawlers, including GPTBot, ClaudeBot, PerplexityBot, Amazonbot, Bytespider and CCBot, with how much of your content each one reads;
- wasted crawl: the 404s a bot keeps hitting and the share of its hits spent on redirects, parameter URLs and static files;
- the top URLs per bot, and average response time when the log records it.

The fake Googlebot check is optional: when you click it, only the IP addresses that claimed to be Googlebot or Bingbot, at most 200 per bot, go to getReport's server for the reverse-DNS lookup Google and Microsoft document. Each comes back verified, fake or unknown. Nothing else from the file is sent.

Very large logs of several hundred megabytes work in a current desktop browser; beyond that, split the file by day or month first.

## Common mistakes

- **Blocking Googlebot to stop a scraper.** The scraper was using Googlebot's name. Verify the IP first, then block the fake at the firewall or CDN, never with a robots.txt rule for `Googlebot`.
- **Using robots.txt to remove pages from Google.** A disallowed page cannot show Googlebot its `noindex`, so it can stay in the index. Allow crawling and add `noindex`; disallow later if you must.
- **Blocking CSS, JavaScript or image folders.** Googlebot renders pages; without those files it sees a broken layout and may miss content and links.
- **Relying on `Crawl-delay` for Google.** Googlebot ignores it. Fix server speed or, in an emergency, return 503 briefly.
- **Letting staging rules reach production.** A `Disallow: /` copied from a staging site is the classic way to disappear from search within days.
- **Treating a crawled page as an indexed one.** Crawl stats going up does not mean more pages rank. Check indexing in Search Console's Pages report.
- **Serving different content to Googlebot.** Showing crawlers something visitors do not see is cloaking, a spam-policy violation, and a common sign of a hacked site.

## Questions people ask

### What is Googlebot in simple terms?

Googlebot is Google's web crawler: a program that visits web pages, downloads them and passes them to Google so they can appear in search results. It finds pages by following links and reading sitemaps, then renders them in a Chromium browser to see the content. It comes in a smartphone and a desktop version, and the smartphone version does most of the work.

### How does Googlebot work step by step?

Googlebot works in four steps: it discovers URLs through links and sitemaps, schedules them by how important they look and how much your server can take, fetches each one after checking robots.txt, and renders the page in Chromium to run its JavaScript. The result goes to Google's indexing systems, which decide separately whether the page is stored and shown in search.

### What is Bingbot?

Bingbot is Microsoft Bing's web crawler, the equivalent of Googlebot for Bing. It crawls pages for Bing's index, which also feeds Yahoo, DuckDuckGo and AI assistants that search through Bing. Unlike Googlebot, it honours `Crawl-delay` in robots.txt and accepts instant URL submissions through IndexNow. Verify it with a reverse-DNS lookup that should end in `search.msn.com`.

### What is the difference between Googlebot Smartphone and Googlebot Desktop?

Googlebot Smartphone crawls your pages as a mobile browser and Googlebot Desktop as a desktop browser. Because Google uses mobile-first indexing, the smartphone crawler makes most requests and its view of the page is the one Google indexes. Both obey the same `Googlebot` rules in robots.txt, so you cannot block one without the other.

### Does Googlebot run JavaScript?

Yes. Googlebot renders pages in an up-to-date headless Chromium and runs their JavaScript, so content and links added by scripts can be indexed. Rendering can happen after the first fetch, though, and Googlebot does not click, scroll or keep cookies between pages. Put important content and links in the server-rendered HTML to be safe.

### Does blocking Googlebot in robots.txt remove my page from Google?

No. A `Disallow` rule stops Googlebot fetching the page, but the URL can stay indexed, without a description, if other pages link to it. To remove a page, let Googlebot crawl it and return a `noindex` tag or header, or a 404 or 410 status. Once it has dropped out, you can disallow it if you want.

### Is Google-Extended a separate crawler from Googlebot?

No. Google-Extended is only a robots.txt token; the crawling is still done by Googlebot. Disallowing `Google-Extended` tells Google not to use your content for Gemini model training and grounding. It does not affect Google Search, rankings or AI Overviews, which are built from normal Googlebot crawling.

### Why does Googlebot keep crawling pages that return 404?

Because Google rechecks URLs it once knew, in case they come back, especially when links or sitemap entries still point to them. Occasional 404 hits are normal and do not hurt rankings. If Googlebot spends a large share of its visits on them, remove the internal links and sitemap entries, and return 410 for pages that are gone for good.