Noindex in robots.txt: why it does not work and what to use instead

You cannot noindex a page in robots.txt. A line such as Noindex: /tag/ was never part of the robots.txt standard, and Google stopped honouring it on 1 September 2019; today it is ignored like any other unknown line. robots.txt controls crawling, while noindex controls indexing, and the two only work in a specific combination. This guide explains the difference, why a page that is both disallowed and noindexed stays in Google, and how to move each old robots.txt rule to a robots meta tag or an X-Robots-Tag header, with server snippets for folders and file types. For the file itself, start with the robots.txt guide.

Quick answer

• Noindex: in robots.txt does nothing in Google since 1 September 2019. Delete those lines.
• robots.txt says "do not fetch". noindex says "you may fetch, but do not list". They answer different questions.
• Use noindex to keep pages out of results: <meta name="robots" content="noindex"> in the HTML, or an X-Robots-Tag: noindex header for files and whole folders.
• Never disallow a page you want noindexed. Google cannot read a tag on a page it is not allowed to fetch.
• Use robots.txt only to save crawling of URLs with no search value, and only after they are out of the index.
• For private content, use a password. Neither robots.txt nor noindex protects anything.

Why noindex in robots.txt does not work

For years, some sites put indexing rules in robots.txt:

User-agent: *
Disallow: /cart/
Noindex: /tag/
Noindex: /search/

Google's crawler used to act on Noindex: lines, although Google never documented them. In July 2019, when Google proposed robots.txt as an internet standard and open-sourced its parser, it announced that it would retire code handling unsupported and unpublished rules, including noindex, nofollow and crawl-delay, from 1 September 2019. The standard that followed, RFC 9309, defines only user-agent, allow and disallow, and Google additionally reads sitemap.

Since then, Google skips a Noindex: line without any warning. Search engines that never supported it ignore it too. So a site that still relies on such lines has no indexing control at all over those URLs: they are crawlable, because nothing disallows them, and indexable, because nothing says noindex.

If you find these lines, you have two jobs: remove them, and put the intent they expressed somewhere that works.

robots.txt vs noindex: what each one does

The table explains the most common mistake. People add a Disallow to remove a page from Google. Google stops fetching it, but links from other pages still tell Google the URL exists, so it can stay in results as a bare address with no description, which Search Console reports as "Indexed, though blocked by robots.txt".

Why disallow and noindex together fail

If a page carries noindex and is also disallowed in robots.txt, Google never fetches the page, so it never sees the noindex. The page stays indexed for as long as links point to it. This combination is common after clean-ups where someone "made sure" by doing both.

The order that works:

1. Allow crawling of the URLs.
2. Add noindex to them.
3. Wait until they drop out: the Page indexing report in Search Console moves them to "Excluded by 'noindex' tag".
4. Only then, if crawling them wastes resources, add a Disallow, knowing that new links can bring them back as bare URLs.

For most sites, step 4 is unnecessary. Google crawls noindex pages less often over time on its own. The guide to fixing "Blocked by robots.txt" in Search Console covers the case where pages are already stuck in that status.

How to move robots.txt Noindex rules to the right place

Take each old Noindex: line and decide what it was for. Most fall into a few patterns.

Single pages and page types in a CMS

Tag archives, author pages, internal search, thank-you pages and thin archives are best handled in the CMS, which adds the robots meta tag to the HTML:

<meta name="robots" content="noindex">

• WordPress: SEO plugins set this per page type and per page. In Yoast SEO, "Show in search results" set to No; in Rank Math, the Robots Meta settings under Titles & Meta. WordPress core already adds noindex to internal search results.
• Shopify: edit the theme's <head> in theme.liquid with a condition for the template, or use an SEO app.
• Other platforms: most have a "hide from search engines" switch per page.

Whole folders

A line such as Noindex: /internal/ becomes an HTTP header on everything under that path.

Apache, in the virtual host or the main .htaccess (requires mod_headers):

# .htaccess in the /internal/ folder
Header set X-Robots-Tag "noindex"

nginx, in the server block:

location /internal/ {
    add_header X-Robots-Tag "noindex" always;
    # ... the rest of this location's configuration
}

In nginx, an add_header inside a location replaces any add_header lines inherited from the server level, such as security headers, so repeat those inside the block or use a shared include.

File types

PDFs, Word files and images have no HTML <head>, so they need the header:

<FilesMatch "\.(pdf|docx?)$">
    Header set X-Robots-Tag "noindex"
</FilesMatch>

location ~* \.(pdf|docx?)$ {
    add_header X-Robots-Tag "noindex" always;
}

Keep in mind that PDFs are often useful search results. Noindex them only if they duplicate a web page or should not be found.

Staging and test hosts

Noindex: / on a staging host never worked, and Disallow: / does not keep staging URLs out of Google either if they get linked. Put staging behind a password, and add a host-wide X-Robots-Tag: noindex header as a second layer. Staging sites: testing changes before they go live covers the setup, including how to keep that header from reaching production.

Parameter and filter URLs

Noindex: /*?sort= and similar lines were usually about duplicate URLs. Those are better handled with a canonical tag to the main version, plus robots.txt only for combinations that should not be crawled at all. The faceted navigation guide walks through the decision.

The noindex, nofollow and robots directives reference lists every directive the meta tag and header accept, such as nosnippet and max-image-preview, and which page types should get which.

What else people put in robots.txt that Google ignores

Noindex: is not the only rule that looks meaningful but is skipped. Google supports only user-agent, allow, disallow and sitemap. Also ignored by Google:

• Nofollow: Use rel="nofollow" on links, or the nofollow robots directive on a page.
• Crawl-delay: Bing honours it; Google does not. Googlebot slows down on its own when a server answers slowly or with 429 or 503.
• Host: Used by one search engine in the past; ignored by Google. Use redirects to your main host.
• Clean-param: Also specific to one search engine; Google ignores it.

The robots.txt patterns guide covers the rules that do work, including wildcards and Allow.

How to check it

Enter a URL you want kept out of Google. The tester fetches the live robots.txt and says whether that URL is allowed for Googlebot, then checks the page's robots meta tag and X-Robots-Tag header for noindex. The final verdict says whether the page can be indexed. What you want for a page that should stay out of results: allowed in robots.txt, with noindex found. If it says blocked and noindex, you have the combination that never works.

The full getReport report runs the same checks on every page you test:

In Search Console, URL Inspection shows for one URL whether crawling is allowed and whether indexing is allowed, and which of the two stopped it.

Common mistakes

• Keeping Noindex: lines because they "might still help". They do nothing in Google.
• Disallowing pages to remove them from results. They often stay as bare URLs.
• Disallow plus noindex on the same URL. The tag is never seen.
• Noindexing with robots.txt on staging. Use a password.
• An X-Robots-Tag: noindex header left on the production server after copying the staging configuration. Check the headers of the live home page after every deployment.
• Noindexing pages that other pages canonicalise to. The canonical target should always be indexable.

Questions people ask

Does noindex work in robots.txt?

No. Google stopped honouring Noindex: lines in robots.txt on 1 September 2019, and the robots.txt standard, RFC 9309, never included them. The line is ignored like any unknown rule. To keep a page out of search results, add a robots meta tag with noindex to the page, or send an X-Robots-Tag: noindex HTTP header.

Why did Google stop supporting noindex in robots.txt?

Because it was never an official rule. When Google proposed robots.txt as an internet standard in July 2019 and published its parser, it retired code for undocumented rules such as noindex, nofollow and crawl-delay, effective 1 September 2019. Google pointed site owners to robots meta tags, 404 or 410 status codes, password protection and the Removals tool instead.

How do I noindex a whole folder without robots.txt?

Send an X-Robots-Tag: noindex HTTP header for every URL in the folder. On Apache, put Header set X-Robots-Tag "noindex" in an .htaccess file in that folder; on nginx, add add_header X-Robots-Tag "noindex" always; inside a location block for the path. Make sure the folder is not disallowed in robots.txt, or Google never sees the header.

Should a noindexed page also be disallowed in robots.txt?

Not while you want it out of the index. Google must crawl the page to see noindex, so a disallowed page with the tag can stay indexed indefinitely. Leave it crawlable until Search Console shows it as excluded by the noindex tag. Adding a Disallow afterwards is optional and only saves crawling.