How often does Google crawl a site? Crawl rate and the Crawl stats report

How often Google crawls a site has no fixed answer: Google decides per URL, fetching pages it considers important or fast-changing often, sometimes several times a day, and quiet pages rarely, sometimes once in several months. The pace is set by two things Google calls crawl demand (how much it wants your URLs) and crawl capacity (how much your server can take). This guide shows how to see your real crawl rate in Search Console's Crawl stats report and in your server logs, what speeds crawling up or slows it down, and how to get a new or changed page crawled sooner. It is part of the guide to Googlebot and crawling.

Quick answer

• There is no schedule. Popular, frequently updated pages are crawled often; deep, unchanged pages may wait weeks or months.
• See the real numbers in Search Console → Settings → Crawl stats: requests per day for the last 90 days, by response code, file type, purpose and Googlebot type.
• Your access log shows every fetch per URL, so you can count days between visits to a specific page.
• Get pages crawled sooner by linking them from pages Google already visits often, listing them in your XML sitemap with an accurate lastmod, and using Request indexing in URL Inspection for a few key URLs.
• A slow or erroring server slows Google down. 5xx and 429 responses make Googlebot back off.
• You cannot raise the crawl rate directly. Search Console's crawl-rate setting was retired in January 2024.

What decides how often Google crawls

Google's documentation on crawl budget describes two forces.

Crawl capacity limit. Googlebot works out how many parallel connections it can use and how long to wait between fetches without slowing your server. It raises the limit when your site answers quickly and lowers it when responses slow down or errors appear. Google's guidance says Googlebot "shouldn't access your site more than once every few seconds on average" for most sites.

Crawl demand. Within that limit, Google fetches what it wants most:

• Popularity. URLs that are linked to widely and get traffic are refreshed more often.
• Staleness. Google tries to keep its copy of a page from going out of date, so pages that change often are revisited often.
• Site events. A migration or a large batch of new URLs raises demand for a while.

So a homepage and a busy category page can be crawled daily, while an old blog post that has not changed in years is checked rarely. "How often does Google crawl my site" is really a set of answers, one per URL.

How to read the Crawl stats report

Open Search Console, choose your property, then Settings → Crawl stats → Open report. The report is available only for root-level properties: a domain property such as example.com, or a URL-prefix property at the root such as https://www.example.com/.

It covers the last 90 days:

Click any row to see example URLs. Things to look for:

• A drop in requests with a rise in response time. Your server slowed down and Google throttled itself. Fix the server, and crawling recovers on its own.
• Host status warnings. A robots.txt fetch that fails with a server error makes Google pause crawling; DNS or connectivity problems do the same.
• A large share of 404s or redirects. Crawls spent on dead or moved URLs are crawls not spent on your pages.
• Discovery near zero on a site that publishes regularly. Google is not finding the new URLs; check your sitemap and internal links.

The report counts all Google's crawlers together and shows examples rather than every URL. For page-level answers, you need the log.

How to see crawl frequency per page in your logs

Your server's access log has a line for every request Googlebot made. Count the days on which it fetched a specific URL:

# Days on which Googlebot fetched one URL in this log
grep 'Googlebot' access.log | grep '"GET /products/blue-mug ' \
  | awk '{print substr($4, 2, 11)}' | sort -u

# Googlebot requests per day across the whole site
grep 'Googlebot' access.log | awk '{print substr($4, 2, 11)}' | uniq -c

The first command prints dates such as 26/Sep/2026; the gap between them is that page's crawl interval. Remember that a user agent can be faked, so check that the addresses are really Google's first; how to verify Googlebot shows how.

For a whole site, the log file analyser does the counting:

It shows Googlebot's hits per day, the status codes it received, its most fetched URLs and how much of its crawl went to 404s, redirects and parameter URLs. It runs in your browser; the file is not uploaded.

How to get Google to crawl your site sooner

You cannot order a crawl, but you can make Google find and want your pages faster. In order of effort:

1. Link new pages from pages Google already crawls often

Your homepage, main categories and a "latest" block are crawled most. A new page linked from them is discovered on the next visit. A page nothing links to, an orphan, may never be crawled at all.

2. Keep the XML sitemap current, with honest lastmod

List every page you want indexed, and set <lastmod> to the date the content last changed meaningfully. Google uses lastmod when it is consistently accurate, and ignores changefreq and priority. A sitemap where every date is today teaches Google to ignore the dates. Declare the sitemap in robots.txt and submit it in Search Console. When to set sitemap lastmod covers the details.

The old sitemap "ping" URL no longer works; Google retired it in 2023. Submitting in Search Console and listing the sitemap in robots.txt are what remain.

3. Request indexing for a few important URLs

In Search Console, paste the URL into URL Inspection, then click Request indexing. It puts that one URL in a priority queue. There is a daily quota, and Google says crawling can still take anywhere from a few days to a few weeks. Requesting the same URL repeatedly does not speed it up.

Google's Indexing API is not an alternative for normal pages: Google limits it to job postings and livestream video pages.

4. Make the server fast and reliable

Capacity is half the equation. A server that answers in 200 ms with no errors gets more parallel fetches than one taking 2 seconds. Check response time in the Crawl stats report and in the report's server response time finding:

5. Stop wasting crawls

Redirect chains, endless filter and sort parameters, and old URLs that return 404 all use requests. On a small site it rarely matters; on a large shop it can decide whether new products are crawled this week or next month. Crawl budget for sites under 10,000 pages explains when this is worth your time.

How to slow Google down if it overloads your server

Search Console's crawl-rate limiter was retired on 8 January 2024. Google's advice for an emergency is to return 503, 500 or 429 status codes to Googlebot for a short time; it will reduce its crawl rate automatically. Do not keep that up for more than a day or two, or Google may start dropping URLs from the index. If the problem persists, Google has a form in its help centre to report Googlebot overcrawling.

Crawl-delay in robots.txt does nothing for Google, though Bing honours it.

Common mistakes

• Expecting a fixed crawl schedule. Google crawls per URL, by demand. Weekly patterns in the report are a result, not a setting.
• Resubmitting the sitemap every day. Google refetches a known sitemap on its own; submit it again when the location changes.
• Setting every lastmod to today. It makes the field useless and may slow the discovery of real changes.
• Reading more crawling as better ranking. Crawling is a precondition, not a ranking factor. A page can be crawled daily and still not be indexed.
• Blocking CSS and JavaScript to "save crawl". Googlebot then cannot render the page properly.

Questions people ask

How often does Googlebot visit a typical page?

It varies from several times a day to once in several months. Google recrawls each URL according to how popular it is and how often it changes, within what your server can handle. Homepages and busy category pages are usually fetched daily; old, unchanged pages much less often. Your access log shows the exact dates for any page, and Search Console's Crawl stats report shows the site-wide trend.

How do I get Google to crawl a new page quickly?

Link it from your homepage or another page Google crawls often, add it to your XML sitemap with an accurate lastmod date, and use Request indexing in Search Console's URL Inspection tool. That combination usually gets a page crawled within days, though Google says it can take a few weeks. Repeated requests for the same URL do not help.

What is a normal crawl rate in the Crawl stats report?

There is no normal number; it scales with your site's size, popularity and how often you publish. A 50-page brochure site may see a few dozen requests a day, a large shop tens of thousands. What matters is the trend and the mix: steady or rising requests, low response times, few 404s and 5xx errors, and discovery requests when you add pages.

Why did Google's crawl requests suddenly drop?

Usually because your server slowed down or returned errors, and Googlebot reduced its rate to protect it. Check average response time and host status in the Crawl stats report for the same days. Other causes are a robots.txt change that blocks more than intended, or fewer new URLs to discover. Fix the cause and the rate recovers without any action in Search Console.

Can I change the crawl rate in Google Search Console?

No, not any more. Google retired the crawl-rate limiter setting on 8 January 2024, saying its crawlers now adjust automatically to how your server responds. To slow Googlebot in an emergency, return 503 or 429 responses for a day or two at most. To be crawled more, make the server faster and give Google better reasons to visit.