# Noindex in robots.txt: why it does not work and what to use instead

> A Noindex line in robots.txt does nothing in Google since September 2019. Why it never really worked, how robots.txt and noindex differ, and how to move each old rule to a meta tag or an X-Robots-Tag header.

Updated 2026-09-26 · Technical SEO · HTML version: https://getreport.app/guides/robots-txt-noindex

You cannot noindex a page in robots.txt. A line such as `Noindex: /tag/` was never part of the robots.txt standard, and Google stopped honouring it on 1 September 2019; today it is ignored like any other unknown line. robots.txt controls crawling, while `noindex` controls indexing, and the two only work in a specific combination. This guide explains the difference, why a page that is both disallowed and noindexed stays in Google, and how to move each old robots.txt rule to a robots meta tag or an `X-Robots-Tag` header, with server snippets for folders and file types. For the file itself, start with the [robots.txt guide](https://getreport.app/guides/robots-txt).

## Quick answer

- **`Noindex:` in robots.txt does nothing** in Google since 1 September 2019. Delete those lines.
- **robots.txt says "do not fetch".** `noindex` says "you may fetch, but do not list". They answer different questions.
- **Use `noindex` to keep pages out of results:** `<meta name="robots" content="noindex">` in the HTML, or an `X-Robots-Tag: noindex` header for files and whole folders.
- **Never disallow a page you want noindexed.** Google cannot read a tag on a page it is not allowed to fetch.
- **Use robots.txt only to save crawling** of URLs with no search value, and only after they are out of the index.
- **For private content,** use a password. Neither robots.txt nor `noindex` protects anything.

## Why noindex in robots.txt does not work

For years, some sites put indexing rules in robots.txt:

```text
User-agent: *
Disallow: /cart/
Noindex: /tag/
Noindex: /search/
```

Google's crawler used to act on `Noindex:` lines, although Google never documented them. In July 2019, when Google proposed robots.txt as an internet standard and open-sourced its parser, it announced that it would retire code handling unsupported and unpublished rules, including `noindex`, `nofollow` and `crawl-delay`, from 1 September 2019. The standard that followed, RFC 9309, defines only `user-agent`, `allow` and `disallow`, and Google additionally reads `sitemap`.

Since then, Google skips a `Noindex:` line without any warning. Search engines that never supported it ignore it too. So a site that still relies on such lines has no indexing control at all over those URLs: they are crawlable, because nothing disallows them, and indexable, because nothing says `noindex`.

If you find these lines, you have two jobs: remove them, and put the intent they expressed somewhere that works.

## robots.txt vs noindex: what each one does

| | robots.txt `Disallow` | `noindex` (meta tag or header) |
| --- | --- | --- |
| Question it answers | May a crawler fetch this URL? | May a search engine list this page? |
| Where it lives | One file at `/robots.txt` | In each page's HTML, or in the HTTP response |
| Effect on crawling | The URL is not fetched | The page is fetched normally |
| Effect on indexing | None directly; the URL can still be indexed from links | The page is dropped from results after the next crawl |
| Works on PDFs and images | Yes | Yes, with `X-Robots-Tag` |
| Keeps a page out of Google | No | Yes |

The table explains the most common mistake. People add a `Disallow` to remove a page from Google. Google stops fetching it, but links from other pages still tell Google the URL exists, so it can stay in results as a bare address with no description, which Search Console reports as "Indexed, though blocked by robots.txt".

### Why disallow and noindex together fail

If a page carries `noindex` and is also disallowed in robots.txt, Google never fetches the page, so it never sees the `noindex`. The page stays indexed for as long as links point to it. This combination is common after clean-ups where someone "made sure" by doing both.

The order that works:

1. **Allow crawling** of the URLs.
2. **Add `noindex`** to them.
3. **Wait** until they drop out: the Page indexing report in Search Console moves them to "Excluded by 'noindex' tag".
4. **Only then**, if crawling them wastes resources, add a `Disallow`, knowing that new links can bring them back as bare URLs.

For most sites, step 4 is unnecessary. Google crawls `noindex` pages less often over time on its own. The guide to [fixing "Blocked by robots.txt" in Search Console](https://getreport.app/guides/blocked-by-robots-txt) covers the case where pages are already stuck in that status.

## How to move robots.txt Noindex rules to the right place

Take each old `Noindex:` line and decide what it was for. Most fall into a few patterns.

### Single pages and page types in a CMS

Tag archives, author pages, internal search, thank-you pages and thin archives are best handled in the CMS, which adds the robots meta tag to the HTML:

```html
<meta name="robots" content="noindex">
```

- **WordPress:** SEO plugins set this per page type and per page. In Yoast SEO, "Show in search results" set to No; in Rank Math, the Robots Meta settings under Titles & Meta. WordPress core already adds `noindex` to internal search results.
- **Shopify:** edit the theme's `<head>` in `theme.liquid` with a condition for the template, or use an SEO app.
- **Other platforms:** most have a "hide from search engines" switch per page.

### Whole folders

A line such as `Noindex: /internal/` becomes an HTTP header on everything under that path.

Apache, in the virtual host or the main `.htaccess` (requires `mod_headers`):

```apache
# .htaccess in the /internal/ folder
Header set X-Robots-Tag "noindex"
```

nginx, in the `server` block:

```nginx
location /internal/ {
    add_header X-Robots-Tag "noindex" always;
    # ... the rest of this location's configuration
}
```

In nginx, an `add_header` inside a `location` replaces any `add_header` lines inherited from the server level, such as security headers, so repeat those inside the block or use a shared include.

### File types

PDFs, Word files and images have no HTML `<head>`, so they need the header:

```apache
<FilesMatch "\.(pdf|docx?)$">
    Header set X-Robots-Tag "noindex"
</FilesMatch>
```

```nginx
location ~* \.(pdf|docx?)$ {
    add_header X-Robots-Tag "noindex" always;
}
```

Keep in mind that PDFs are often useful search results. Noindex them only if they duplicate a web page or should not be found.

### Staging and test hosts

`Noindex: /` on a staging host never worked, and `Disallow: /` does not keep staging URLs out of Google either if they get linked. Put staging behind a password, and add a host-wide `X-Robots-Tag: noindex` header as a second layer. [Staging sites: testing changes before they go live](https://getreport.app/guides/staging-with-wordpress-plugins-and-host-tools) covers the setup, including how to keep that header from reaching production.

### Parameter and filter URLs

`Noindex: /*?sort=` and similar lines were usually about duplicate URLs. Those are better handled with a canonical tag to the main version, plus robots.txt only for combinations that should not be crawled at all. The [faceted navigation guide](https://getreport.app/guides/faceted-navigation-and-parameter-urls) walks through the decision.

The [noindex, nofollow and robots directives reference](https://getreport.app/guides/noindex-nofollow-complete-reference) lists every directive the meta tag and header accept, such as `nosnippet` and `max-image-preview`, and which page types should get which.

## What else people put in robots.txt that Google ignores

`Noindex:` is not the only rule that looks meaningful but is skipped. Google supports only `user-agent`, `allow`, `disallow` and `sitemap`. Also ignored by Google:

- **`Nofollow:`** Use `rel="nofollow"` on links, or the `nofollow` robots directive on a page.
- **`Crawl-delay:`** Bing honours it; Google does not. Googlebot slows down on its own when a server answers slowly or with 429 or 503.
- **`Host:`** Used by one search engine in the past; ignored by Google. Use redirects to your main host.
- **`Clean-param:`** Also specific to one search engine; Google ignores it.

The [robots.txt patterns guide](https://getreport.app/guides/robots-txt-patterns-wildcards-allow-crawl-delay) covers the rules that do work, including wildcards and `Allow`.

## How to check it

> **Free tool:** [robots.txt tester: check and validate robots.txt](https://getreport.app/tools/robots-txt-tester): Free robots.txt tester: see whether robots.txt blocks a page for Googlebot, whether your sitemap is declared and reachable, and if the page can be indexed.

Enter a URL you want kept out of Google. The tester fetches the live robots.txt and says whether that URL is allowed for Googlebot, then checks the page's robots meta tag and `X-Robots-Tag` header for `noindex`. The final verdict says whether the page can be indexed. What you want for a page that should stay out of results: allowed in robots.txt, with `noindex` found. If it says blocked and `noindex`, you have the combination that never works.

The full getReport report runs the same checks on every page you test:

> **Check: The page is not set to noindex.** A noindex directive tells Google to drop this page from search results. Unless that is intended, every visit from search is lost.
>
> 1. Remove noindex from the robots meta tag and from the X-Robots-Tag header, unless the page should stay out of search.
> 2. In WordPress, untick Settings → Reading → "Discourage search engines" and check the page's SEO plugin settings.

> **Check: robots.txt allows this page.** A Disallow rule stops search engines from crawling this page, so they cannot read its content or see any updates. The page can still appear in results as a bare URL with no description.
>
> 1. Remove or narrow the Disallow rule in robots.txt so this URL is allowed for all crawlers.
> 2. If the page should stay out of search, allow crawling and use a noindex tag instead; a blocked page cannot be de-indexed.

> **Check: The page can be indexed.** To appear in search, a page must return HTTP 200, carry no noindex directive and be allowed in robots.txt. If any of the three fails, the page is invisible to searchers.
>
> 1. Open the technical detail to see which condition fails, then fix that one; the individual findings above have the steps.
> 2. After the fix, request indexing in Google Search Console to speed things up.

In Search Console, URL Inspection shows for one URL whether crawling is allowed and whether indexing is allowed, and which of the two stopped it.

## Common mistakes

- **Keeping `Noindex:` lines** because they "might still help". They do nothing in Google.
- **Disallowing pages to remove them from results.** They often stay as bare URLs.
- **Disallow plus `noindex` on the same URL.** The tag is never seen.
- **Noindexing with robots.txt on staging.** Use a password.
- **An `X-Robots-Tag: noindex` header left on the production server** after copying the staging configuration. Check the headers of the live home page after every deployment.
- **Noindexing pages that other pages canonicalise to.** The canonical target should always be indexable.

## Questions people ask

### Does noindex work in robots.txt?

No. Google stopped honouring `Noindex:` lines in robots.txt on 1 September 2019, and the robots.txt standard, RFC 9309, never included them. The line is ignored like any unknown rule. To keep a page out of search results, add a robots meta tag with `noindex` to the page, or send an `X-Robots-Tag: noindex` HTTP header.

### Why did Google stop supporting noindex in robots.txt?

Because it was never an official rule. When Google proposed robots.txt as an internet standard in July 2019 and published its parser, it retired code for undocumented rules such as `noindex`, `nofollow` and `crawl-delay`, effective 1 September 2019. Google pointed site owners to robots meta tags, 404 or 410 status codes, password protection and the Removals tool instead.

### How do I noindex a whole folder without robots.txt?

Send an `X-Robots-Tag: noindex` HTTP header for every URL in the folder. On Apache, put `Header set X-Robots-Tag "noindex"` in an `.htaccess` file in that folder; on nginx, add `add_header X-Robots-Tag "noindex" always;` inside a `location` block for the path. Make sure the folder is not disallowed in robots.txt, or Google never sees the header.

### Should a noindexed page also be disallowed in robots.txt?

Not while you want it out of the index. Google must crawl the page to see `noindex`, so a disallowed page with the tag can stay indexed indefinitely. Leave it crawlable until Search Console shows it as excluded by the noindex tag. Adding a `Disallow` afterwards is optional and only saves crawling.