Crawl budget: what it is, when it matters and how to optimise it

Crawl budget is the set of URLs Google can and wants to crawl on your site in a given time. It is set by two things: how much crawling your server can take (the crawl capacity limit) and how much Google wants to fetch (crawl demand). Google's own guidance says it is a real concern for sites with over a million pages that change weekly, sites with over 10,000 pages that change daily, and sites with many URLs stuck in "Discovered - currently not indexed". For a site with a few hundred or a few thousand pages it is almost never why a page is missing. This guide shows how to tell which case you are, and what to fix in which order. It builds on our guide to Googlebot and crawling.

Quick answer

• Check first: Search Console → Settings → Crawl stats. If Google requests more URLs per day than your site has pages, budget is not your problem.
• Small sites: slow indexing is about demand (links, quality) or a slow server, not budget.
• What wastes crawling on any site: sitemap entries that 404 or redirect, redirect chains, parameter and filter URLs, archives, internal search pages, soft 404s and staging copies.
• Fix in this order: sitemap, redirect chains, parameter URLs, infinite spaces, internal links, server speed.
• Do not add Crawl-delay, block CSS or JavaScript, or install a "crawl budget" plugin.
• Measure with the sitemap checker and the log file analyser.

What is crawl budget?

Google describes crawl budget in its crawl budget management guide, which moved to its new Crawling Infrastructure documentation and was rewritten on 22 July 2026. Two parts decide it:

The July 2026 rewrite added three clarifications worth knowing. Every site starts with the same conservative default capacity limit, which grows over time if there is demand and the server copes. The capacity limit is shared across all of Google's crawlers, so heavy crawling by one leaves less for the others. And faster responses and HTTP caching (answering 304 Not Modified to conditional requests) make each unit of capacity go further.

Google is also clear that the only ways to increase crawl budget are more serving capacity and, more importantly, more valuable content. There is no setting for it.

Does crawl budget matter for your site?

What small sites experience as a budget problem is nearly always one of three other things:

• Google does not think the page is worth crawling yet. It is only in the sitemap, nothing links to it, or it looks like a hundred others. That is demand, not capacity; see Discovered - currently not indexed.
• The server was slow or erroring when Googlebot came. Google backs off when responses take long or return 5xx.
• The crawls went to junk. Redirect chains, filter combinations, ?replytocom= links, calendar pages. Plenty of URLs fetched, the wrong ones.

The third is the only real budget issue on a small site, and it is visible in your server log. How often Google crawls a site explains how to read the Crawl stats report in detail.

How getReport checks it

The validator finds the sitemaps declared in robots.txt and at /sitemap.xml, parses them, and requests up to 25 listed URLs, spread across the file, to see whether each answers 200 and lands on itself. Every entry that does not is a fetch Google wastes too. When the file is a sitemap index, as on most WordPress sites, the sampled rows are the child sitemaps; declare a child sitemap directly to sample its pages.

The redirect finding warns at two or more hops and lists each one; the internal-links finding warns when a page has fewer than 3 or more than 300 same-host links. Neither crawls the whole site, so run them on a few representative pages.

The log file analyser is the other half. It reads an access log in your browser, with nothing uploaded, and splits Googlebot's hits into pages it could index and answers it could not:

Static files are normal, since Googlebot needs CSS and JavaScript to render pages; the red and amber bars are the ones to work on. SEO log file analysis walks through the whole report.

Crawl budget optimisation, step by step

1. Measure: pages versus requests

Compare total crawl requests per day in Crawl stats with your page count (the sitemap validator prints the URLs per file). If Google fetches 1,500 URLs a day and you have 900 pages, it can crawl the whole site daily; the rest of this list is about making those fetches count. If it fetches far fewer than you have pages and "By response" shows server errors or rising response times, the server is holding the crawl back.

2. Find the waste in a month of logs

Each is a near-infinite space: a few templates generating thousands of URLs that Googlebot keeps rechecking.

3. Fix the sitemap

The sitemap is your list of what matters. Remove deleted pages, list final URLs instead of redirected ones, and keep lastmod honest: a date that changes on every request teaches Google to ignore it, while a real date lets it skip unchanged pages. Sitemap lastmod: when to set it has the rules.

4. Collapse redirect chains

Update internal links, menus and the sitemap to the final URL, and make every old address redirect once, straight to the end:

# nginx: one hop from every old address to the final one
location = /old-slug/ { return 301 https://example.com/new-slug/; }

Redirect chains after a migration shows how to find every hop.

5. Canonicalise or block parameter URLs

A canonical on every variant tells Google which URL to keep, but Google still fetches the variant to read it. For parameters that never produce a page worth indexing, stop the fetch in robots.txt:

User-agent: *
Disallow: /*?*sort=
Disallow: /*?*replytocom=
Disallow: /*?*add-to-cart=
Disallow: /?s=

Never block a URL whose noindex or canonical you still need Google to read, and do not block a filter page with search demand of its own. Faceted navigation and parameter URLs sorts parameters into keep, canonicalise, noindex and block.

6. Link to the pages you want crawled

Crawl demand follows links. A page that exists only in the sitemap gets fetched eventually; one linked from the home page and three related articles gets fetched on the next visit.

7. Keep the server fast

Google lowers the crawl rate when responses slow down or 5xx errors appear, and raises it again slowly. A first byte over 800 ms on a page that should be cached comes before anything else here; check it with the TTFB test.

Crawl budget for large sites

On sites past Google's thresholds, the same steps apply at scale, with three additions from Google's guide:

• Manage the URL inventory. Consolidate duplicates, return 404 or 410 for pages that are gone (not a soft 404 that answers 200), and keep faceted navigation from generating unlimited combinations.
• Support conditional requests. Answer If-Modified-Since and If-None-Match with 304 Not Modified when nothing changed, so Google spends less capacity on unchanged pages.
• Watch capacity during launches. A large release of new URLs needs a server that can take the extra crawling; if Googlebot genuinely overloads it, Google's documented emergency lever is answering 503 or 429 for a short time, not a robots.txt rule.

What not to do

• Crawl-delay in robots.txt. Googlebot ignores it.
• Blocking CSS, JavaScript or images "to save budget". Google renders pages; without them it sees a broken page.
• Blocking paginated category pages. Page 2 onward is how the products listed there get found.
• "Crawl budget optimiser" plugins. They flip switches your SEO plugin already has, and some block resources you need.
• Deleting content to shrink the site. A 200-page site has no more budget per page than a 2,000-page one, just fewer pages that can rank.

Platform notes

WordPress. The defaults generate most of the infinite spaces above: ?replytocom= on comment reply links, comment feeds, date and author archives, attachment pages and ?s= search. Yoast SEO and Rank Math have switches for archives and attachment redirects; ?replytocom= and WooCommerce's ?add-to-cart= and ?orderby= belong in robots.txt.

Shopify. Shopify's default robots.txt already blocks /search, /cart, /checkout and sorted collection variants. Crawl waste is mostly deleted products; keep their URLs redirected under URL redirects.

Static sites. No infinite spaces unless you build them. 404s in the sitemap mean the build and the deploy disagree.

Verify

• The sitemap validator's sampled URLs all answer 200 and land on themselves.
• Next month's log: Googlebot's 404, redirect and parameter shares fall and its distinct URL count rises while total hits hold.
• Crawl stats: response time flat or falling, no 5xx, and requests per day comfortably above your page count.

Common mistakes

• Treating "Discovered - currently not indexed" as a budget problem on a small site. Links and content fix it; robots.txt rules do not.
• Blocking a parameter in robots.txt after adding a canonical or noindex to it. The tag is never read.
• A sitemap that lists everything the CMS knows, such as one-post tags and attachment pages.
• Looking at one day of logs. Googlebot's pattern shows over weeks.

Questions people ask

What is crawl budget?

Crawl budget is how many URLs Google is willing and able to fetch on your site in a given time. Google describes it as two parts: the crawl capacity limit (how hard it can crawl without slowing your server) and crawl demand (how popular and fresh your URLs are). It only becomes a real constraint on large or fast-changing sites; smaller sites should focus on links and content.

Does crawl budget matter for small websites?

Rarely. Google's guidance says crawl budget matters for sites with over a million pages changing weekly, over 10,000 pages changing daily, or many URLs stuck in Discovered - currently not indexed. A site with a few thousand pages is usually crawled in full every few days. If its pages are slow to appear, the cause is weak internal links, thin or duplicate content, or a slow server.

How do I increase my crawl budget?

By making your server faster and your content more worth crawling; there is no setting for it. Google raises the crawl capacity limit on its own when responses stay fast and error-free, and raises demand for pages that are linked, popular and genuinely updated. Removing waste, such as redirect chains, 404s in the sitemap and endless parameter URLs, lets the existing budget reach the pages that matter.

Does page speed affect crawl budget?

Yes, server speed does. Google measures how quickly and reliably your server answers and lowers the crawl rate when responses slow down or return 5xx errors. Faster time to first byte and support for 304 Not Modified responses let Google fetch more URLs with the same capacity. Front-end metrics such as Largest Contentful Paint matter for users, not for crawl rate.

Does blocking pages in robots.txt save crawl budget?

Yes, for URLs Google would otherwise fetch, such as endless filter combinations or internal search pages. A Disallow rule stops the fetch entirely. It does not remove URLs already indexed, and Google cannot read a noindex or canonical on a blocked page, so block only URLs that should never be fetched, and use canonicals or noindex for the rest.