"Blocked by robots.txt" in Search Console: what it means and how to fix it

"Blocked by robots.txt" in Google Search Console means Google found a URL but did not crawl it, because a Disallow rule in your robots.txt covers it. "Indexed, though blocked by robots.txt" means Google put the URL in its index anyway, from links, without reading the page. Neither is an error in itself: both are fine for URLs you meant to block, and both need a fix when the URL is a page you want in Google, or one you want out of it. This guide shows how to tell which case you have, how to find the rule, and the exact order of steps for each fix. For how robots.txt works in general, read the robots.txt guide.

Quick answer

• Blocked by robots.txt, page should rank: find the Disallow rule that matches, remove or narrow it, then ask Google to recrawl.
• Blocked by robots.txt, page should not rank: nothing to do. This is the report confirming your rule works.
• Indexed, though blocked, page should rank: remove the block. Google then reads the page and shows it with a proper title and description.
• Indexed, though blocked, page should disappear: remove the block and add noindex. Google can only see noindex on a page it may crawl.
• Test before and after with the robots.txt tester, which names the rule that blocks a URL.
• Expect days, not minutes. Google caches robots.txt for up to 24 hours, then has to recrawl each URL.

What the two statuses mean

Both appear in Search Console under Indexing → Pages, the Page indexing report.

Blocked by robots.txt is listed under "Why pages aren't indexed". Google knows the URL, from a link, your sitemap or an old crawl, checked it against your robots.txt and did not fetch it. The page is not in the index.

Indexed, though blocked by robots.txt appears among the indexed pages, in the section about improving how pages appear. Google did not fetch the page either, but it indexed the address because other pages link to it. Such a result has no description; Google shows the URL, a title it made from link text, and a note that no information is available for the page.

That second case surprises people, because they expected robots.txt to keep a page out of Google. It does not. robots.txt controls crawling, not indexing: it asks crawlers not to fetch a URL, and says nothing about whether the address may be listed. The only reliable way to keep a page out of results is noindex, and that only works on a page Google may crawl. Why that is, and why a Noindex: line in robots.txt does nothing, is covered in robots.txt vs noindex.

Step 1: decide what each URL should do

Open the status in the Page indexing report and look at the example URLs. Sort them into two groups before touching anything:

Most sites have both in the report. A shop typically sees hundreds of ?sort_by= or /cart URLs there, which is expected, next to a handful of real products caught by a rule that was too broad, which is not.

Step 2: find the rule that blocks the URL

Take one URL from each group and test it:

The tester fetches the live robots.txt from that host and parses it with the same rules Google uses: the most specific user-agent group, the longest matching rule, Allow winning a tie, and the * and $ wildcards. It says whether the URL is allowed for Googlebot and for all other agents, and when a rule blocks it, the finding names the line. It also checks the page's robots meta tag and X-Robots-Tag header, so it tells you whether the page could be indexed once it is unblocked.

In Search Console, URL Inspection on the same URL shows "Crawl allowed? No: blocked by robots.txt" in the page's crawl details, and Settings → robots.txt shows which version of the file Google last fetched and when. If the file Google has differs from the one you see in your browser, Google is still working from an older copy.

The usual causes of a page being blocked by mistake:

• A prefix that matches more than you meant. Disallow: /p blocks /products/ and /pricing/; Disallow: /blog blocks /blog-news/ too.
• A wildcard that is too broad. Disallow: /*? blocks every URL with a query string, including pagination such as ?page=2.
• A staging file that reached production, with Disallow: / for everyone.
• A named group with fewer rules. A User-agent: Googlebot group makes Googlebot ignore the * group entirely.
• A platform or plugin default you never looked at, such as SEO plugin crawl settings or a CDN that adds its own rules.

The robots.txt patterns guide explains how Google matches rules, with examples of narrowing a rule so it catches only what you meant.

How to fix "Blocked by robots.txt" for pages that should rank

1. Edit the rule. Remove it, or narrow it so it no longer matches the page. To keep a folder blocked but one page open, add a longer Allow rule, which wins over the shorter Disallow.
2. Check the live file. Open https://yourdomain/robots.txt in a private window and confirm the change is there, not only on your server: caches, plugins and CDNs can serve an older copy.
3. Test the URL again with the robots.txt tester. It should now say allowed for Googlebot, and the final verdict should say the page can be indexed.
4. Ask Google to refetch the file. In Search Console, Settings → robots.txt → the file → Request a recrawl. Otherwise Google picks up the change within about 24 hours.
5. Request indexing for key URLs in URL Inspection. For many URLs, make sure they are in your XML sitemap and let Google recrawl them.
6. Validate the fix in the Page indexing report once you are done, so Search Console tracks the URLs as they move.

Popular pages are usually recrawled within days. If the whole site was blocked, give it a few weeks to return to its previous state, and watch the Pages report and your traffic.

How to fix "Indexed, though blocked by robots.txt"

The fix depends on whether the page should be in Google.

If the page should be in Google

Remove or narrow the rule exactly as above. Once Google can crawl the page, it reads the content and the result gets a real title and description. The status then moves to plain "Indexed" as Google recrawls.

If the page should not be in Google

This is the case that needs the steps in the right order:

1. Add noindex to the page: <meta name="robots" content="noindex"> in the HTML, or an X-Robots-Tag: noindex HTTP header for PDFs and other files. Most CMSs and SEO plugins have a setting per page or per page type.
2. Remove the robots.txt block for those URLs, so Google can fetch them and see the noindex.
3. Wait until the URLs drop out. They move to "Excluded by 'noindex' tag" in the Page indexing report as Google recrawls. Requesting indexing in URL Inspection for a few key URLs speeds that up.
4. Decide whether to block them again. Once they are out, you can restore the Disallow rule to save crawling. If you do, keep the noindex too, and be aware that new links to those URLs can bring the status back.

A page that is both blocked and noindex stays indexed, because Google never sees the tag. That combination is the most common reason this status never goes away.

For a faster, temporary result while you wait, the Removals tool in Search Console hides a URL from results for about six months. It does not remove the page from the index for good; the noindex does that.

If the page is private, such as a staging site, a customer area or documents, neither robots.txt nor noindex is the right tool. Put it behind a password, and read why your page is not in Google for the other indexing signals.

If the URLs are junk nobody searches for

Parameter URLs such as ?add-to-cart=, sort orders or session IDs often land in this status in their hundreds. If they rarely appear in results and nobody searches for them, leaving them blocked is a valid choice. The better long-term fix is to stop linking to them: many appear because a template links to every sort order or filter, and the faceted navigation guide covers when to block those and when to canonicalise them.

How getReport shows it

Every full report fetches your robots.txt, tests the page against it for Googlebot and names the rule if one blocks it. The indexability check combines that with the status code, noindex and the canonical into one verdict:

A page that is blocked in robots.txt and carries noindex shows both findings, which is the signal to fix them in the order above.

Common mistakes

• Adding a Disallow to remove a page from Google. It stops crawling, not indexing; the page often stays as a bare URL.
• Blocking and noindexing at the same time, and waiting for the page to drop out. It never will.
• Fixing the file on the server while a CDN or plugin serves another one. Always check the live /robots.txt.
• Treating every "Blocked by robots.txt" URL as a problem. Carts, internal search and filters belong there.
• Unblocking thousands of junk URLs at once without noindex or a canonical. Google then crawls and may index all of them.
• Validating the fix before Google has the new file. Request a recrawl of robots.txt first.

Questions people ask

How do I fix "Indexed, though blocked by robots.txt"?

Decide whether the page should be in Google. If yes, remove or narrow the Disallow rule and request indexing. If no, add a noindex meta tag or X-Robots-Tag header, then remove the robots.txt block so Google can see it, and wait for the URL to drop out. Once it is gone, you can block it again.

Is "Blocked by robots.txt" in Search Console an error?

No. It is a status, not an error: Google is telling you it respected a rule in your robots.txt. For carts, account pages, internal search and filter URLs, that is exactly what you want. It only needs a fix when the listed URLs include pages you want in Google, such as products or articles caught by a rule that is too broad.

Why does Google index pages I blocked in robots.txt?

Because robots.txt blocks crawling, not indexing. When other pages link to a blocked URL, Google can add the address to its index from those links without ever fetching the page. It then shows it without a description. To keep a page out of results, allow crawling and use noindex, or protect it with a password.

How long does it take Google to see a robots.txt change?

Google usually caches robots.txt for up to 24 hours, so a change takes effect within about a day. You can ask for a faster refetch under Settings → robots.txt in Search Console. After that, each URL is recrawled on Google's normal schedule: key pages within days, rarely visited ones over weeks.

Should I use the Removals tool for pages blocked by robots.txt?

Only as a stopgap. The Removals tool hides a URL from Google's results for about six months but does not remove it from the index. For a permanent fix, let Google crawl the page and add noindex, or delete it so it answers 404 or 410, or protect it with a password.