# "Blocked by robots.txt" in Search Console: what it means and how to fix it

> "Blocked by robots.txt" means a rule in your robots.txt stops Google from crawling a URL; "Indexed, though blocked by robots.txt" means Google indexed it anyway. How to tell whether to fix each and how.

Updated 2026-09-26 · Technical SEO · HTML version: https://getreport.app/guides/blocked-by-robots-txt

"Blocked by robots.txt" in Google Search Console means Google found a URL but did not crawl it, because a `Disallow` rule in your robots.txt covers it. "Indexed, though blocked by robots.txt" means Google put the URL in its index anyway, from links, without reading the page. Neither is an error in itself: both are fine for URLs you meant to block, and both need a fix when the URL is a page you want in Google, or one you want out of it. This guide shows how to tell which case you have, how to find the rule, and the exact order of steps for each fix. For how robots.txt works in general, read the [robots.txt guide](https://getreport.app/guides/robots-txt).

## Quick answer

- **Blocked by robots.txt, page should rank:** find the `Disallow` rule that matches, remove or narrow it, then ask Google to recrawl.
- **Blocked by robots.txt, page should not rank:** nothing to do. This is the report confirming your rule works.
- **Indexed, though blocked, page should rank:** remove the block. Google then reads the page and shows it with a proper title and description.
- **Indexed, though blocked, page should disappear:** remove the block **and** add `noindex`. Google can only see `noindex` on a page it may crawl.
- **Test before and after** with the [robots.txt tester](https://getreport.app/tools/robots-txt-tester), which names the rule that blocks a URL.
- **Expect days, not minutes.** Google caches robots.txt for up to 24 hours, then has to recrawl each URL.

## What the two statuses mean

Both appear in Search Console under **Indexing → Pages**, the Page indexing report.

**Blocked by robots.txt** is listed under "Why pages aren't indexed". Google knows the URL, from a link, your sitemap or an old crawl, checked it against your robots.txt and did not fetch it. The page is not in the index.

**Indexed, though blocked by robots.txt** appears among the indexed pages, in the section about improving how pages appear. Google did not fetch the page either, but it indexed the address because other pages link to it. Such a result has no description; Google shows the URL, a title it made from link text, and a note that no information is available for the page.

That second case surprises people, because they expected robots.txt to keep a page out of Google. It does not. robots.txt controls crawling, not indexing: it asks crawlers not to fetch a URL, and says nothing about whether the address may be listed. The only reliable way to keep a page out of results is `noindex`, and that only works on a page Google may crawl. Why that is, and why a `Noindex:` line in robots.txt does nothing, is covered in [robots.txt vs noindex](https://getreport.app/guides/robots-txt-noindex).

## Step 1: decide what each URL should do

Open the status in the Page indexing report and look at the example URLs. Sort them into two groups before touching anything:

| The URLs are | Examples | What you want |
| --- | --- | --- |
| Pages that should be in Google | Products, articles, categories, the home page | Crawled and indexed |
| Pages with no search value | Cart, checkout, account, internal search, filter and sort parameters, admin | Not crawled, or not indexed |

Most sites have both in the report. A shop typically sees hundreds of `?sort_by=` or `/cart` URLs there, which is expected, next to a handful of real products caught by a rule that was too broad, which is not.

## Step 2: find the rule that blocks the URL

Take one URL from each group and test it:

> **Free tool:** [robots.txt tester: check and validate robots.txt](https://getreport.app/tools/robots-txt-tester): Free robots.txt tester: see whether robots.txt blocks a page for Googlebot, whether your sitemap is declared and reachable, and if the page can be indexed.

The tester fetches the live robots.txt from that host and parses it with the same rules Google uses: the most specific user-agent group, the longest matching rule, `Allow` winning a tie, and the `*` and `$` wildcards. It says whether the URL is allowed for Googlebot and for all other agents, and when a rule blocks it, the finding names the line. It also checks the page's robots meta tag and `X-Robots-Tag` header, so it tells you whether the page could be indexed once it is unblocked.

> **Check: robots.txt allows this page.** A Disallow rule stops search engines from crawling this page, so they cannot read its content or see any updates. The page can still appear in results as a bare URL with no description.
>
> 1. Remove or narrow the Disallow rule in robots.txt so this URL is allowed for all crawlers.
> 2. If the page should stay out of search, allow crawling and use a noindex tag instead; a blocked page cannot be de-indexed.

In Search Console, **URL Inspection** on the same URL shows "Crawl allowed? No: blocked by robots.txt" in the page's crawl details, and **Settings → robots.txt** shows which version of the file Google last fetched and when. If the file Google has differs from the one you see in your browser, Google is still working from an older copy.

The usual causes of a page being blocked by mistake:

- **A prefix that matches more than you meant.** `Disallow: /p` blocks `/products/` and `/pricing/`; `Disallow: /blog` blocks `/blog-news/` too.
- **A wildcard that is too broad.** `Disallow: /*?` blocks every URL with a query string, including pagination such as `?page=2`.
- **A staging file that reached production,** with `Disallow: /` for everyone.
- **A named group with fewer rules.** A `User-agent: Googlebot` group makes Googlebot ignore the `*` group entirely.
- **A platform or plugin default** you never looked at, such as SEO plugin crawl settings or a CDN that adds its own rules.

The [robots.txt patterns guide](https://getreport.app/guides/robots-txt-patterns-wildcards-allow-crawl-delay) explains how Google matches rules, with examples of narrowing a rule so it catches only what you meant.

## How to fix "Blocked by robots.txt" for pages that should rank

1. **Edit the rule.** Remove it, or narrow it so it no longer matches the page. To keep a folder blocked but one page open, add a longer `Allow` rule, which wins over the shorter `Disallow`.
2. **Check the live file.** Open `https://yourdomain/robots.txt` in a private window and confirm the change is there, not only on your server: caches, plugins and CDNs can serve an older copy.
3. **Test the URL again** with the robots.txt tester. It should now say allowed for Googlebot, and the final verdict should say the page can be indexed.
4. **Ask Google to refetch the file.** In Search Console, Settings → robots.txt → the file → Request a recrawl. Otherwise Google picks up the change within about 24 hours.
5. **Request indexing for key URLs** in URL Inspection. For many URLs, make sure they are in your XML sitemap and let Google recrawl them.
6. **Validate the fix** in the Page indexing report once you are done, so Search Console tracks the URLs as they move.

Popular pages are usually recrawled within days. If the whole site was blocked, give it a few weeks to return to its previous state, and watch the Pages report and your traffic.

## How to fix "Indexed, though blocked by robots.txt"

The fix depends on whether the page should be in Google.

### If the page should be in Google

Remove or narrow the rule exactly as above. Once Google can crawl the page, it reads the content and the result gets a real title and description. The status then moves to plain "Indexed" as Google recrawls.

### If the page should not be in Google

This is the case that needs the steps in the right order:

1. **Add `noindex`** to the page: `<meta name="robots" content="noindex">` in the HTML, or an `X-Robots-Tag: noindex` HTTP header for PDFs and other files. Most CMSs and SEO plugins have a setting per page or per page type.
2. **Remove the robots.txt block** for those URLs, so Google can fetch them and see the `noindex`.
3. **Wait until the URLs drop out.** They move to "Excluded by 'noindex' tag" in the Page indexing report as Google recrawls. Requesting indexing in URL Inspection for a few key URLs speeds that up.
4. **Decide whether to block them again.** Once they are out, you can restore the `Disallow` rule to save crawling. If you do, keep the `noindex` too, and be aware that new links to those URLs can bring the status back.

A page that is both blocked and `noindex` stays indexed, because Google never sees the tag. That combination is the most common reason this status never goes away.

For a faster, temporary result while you wait, the **Removals** tool in Search Console hides a URL from results for about six months. It does not remove the page from the index for good; the `noindex` does that.

If the page is private, such as a staging site, a customer area or documents, neither robots.txt nor `noindex` is the right tool. Put it behind a password, and read [why your page is not in Google](https://getreport.app/guides/why-is-my-page-not-in-google) for the other indexing signals.

### If the URLs are junk nobody searches for

Parameter URLs such as `?add-to-cart=`, sort orders or session IDs often land in this status in their hundreds. If they rarely appear in results and nobody searches for them, leaving them blocked is a valid choice. The better long-term fix is to stop linking to them: many appear because a template links to every sort order or filter, and the [faceted navigation guide](https://getreport.app/guides/faceted-navigation-and-parameter-urls) covers when to block those and when to canonicalise them.

## How getReport shows it

Every full report fetches your robots.txt, tests the page against it for Googlebot and names the rule if one blocks it. The indexability check combines that with the status code, `noindex` and the canonical into one verdict:

> **Check: The page can be indexed.** To appear in search, a page must return HTTP 200, carry no noindex directive and be allowed in robots.txt. If any of the three fails, the page is invisible to searchers.
>
> 1. Open the technical detail to see which condition fails, then fix that one; the individual findings above have the steps.
> 2. After the fix, request indexing in Google Search Console to speed things up.

> **Check: The page is not set to noindex.** A noindex directive tells Google to drop this page from search results. Unless that is intended, every visit from search is lost.
>
> 1. Remove noindex from the robots meta tag and from the X-Robots-Tag header, unless the page should stay out of search.
> 2. In WordPress, untick Settings → Reading → "Discourage search engines" and check the page's SEO plugin settings.

A page that is blocked in robots.txt and carries `noindex` shows both findings, which is the signal to fix them in the order above.

## Common mistakes

- **Adding a `Disallow` to remove a page from Google.** It stops crawling, not indexing; the page often stays as a bare URL.
- **Blocking and noindexing at the same time,** and waiting for the page to drop out. It never will.
- **Fixing the file on the server while a CDN or plugin serves another one.** Always check the live `/robots.txt`.
- **Treating every "Blocked by robots.txt" URL as a problem.** Carts, internal search and filters belong there.
- **Unblocking thousands of junk URLs at once** without `noindex` or a canonical. Google then crawls and may index all of them.
- **Validating the fix before Google has the new file.** Request a recrawl of robots.txt first.

## Questions people ask

### How do I fix "Indexed, though blocked by robots.txt"?

Decide whether the page should be in Google. If yes, remove or narrow the `Disallow` rule and request indexing. If no, add a `noindex` meta tag or `X-Robots-Tag` header, then remove the robots.txt block so Google can see it, and wait for the URL to drop out. Once it is gone, you can block it again.

### Is "Blocked by robots.txt" in Search Console an error?

No. It is a status, not an error: Google is telling you it respected a rule in your robots.txt. For carts, account pages, internal search and filter URLs, that is exactly what you want. It only needs a fix when the listed URLs include pages you want in Google, such as products or articles caught by a rule that is too broad.

### Why does Google index pages I blocked in robots.txt?

Because robots.txt blocks crawling, not indexing. When other pages link to a blocked URL, Google can add the address to its index from those links without ever fetching the page. It then shows it without a description. To keep a page out of results, allow crawling and use `noindex`, or protect it with a password.

### How long does it take Google to see a robots.txt change?

Google usually caches robots.txt for up to 24 hours, so a change takes effect within about a day. You can ask for a faster refetch under Settings → robots.txt in Search Console. After that, each URL is recrawled on Google's normal schedule: key pages within days, rarely visited ones over weeks.

### Should I use the Removals tool for pages blocked by robots.txt?

Only as a stopgap. The Removals tool hides a URL from Google's results for about six months but does not remove it from the index. For a permanent fix, let Google crawl the page and add `noindex`, or delete it so it answers 404 or 410, or protect it with a password.