Noindex: the robots meta tag, nofollow and every other directive

A noindex directive tells search engines not to show a page in their results: add <meta name="robots" content="noindex"> to the page's head, or send an X-Robots-Tag: noindex HTTP header, and leave the page crawlable so Google can read it. The robots meta tag carries the other directives too, such as nofollow, nosnippet and max-image-preview, and it is the one tag in our map of meta tags for SEO that can remove a page from Google on its own. This reference lists every directive, how noindex differs from robots.txt and from rel="nofollow" on a link, and which control to use for staging, thank-you, search and filter pages, with the WordPress and Shopify settings that produce them.

Quick answer

Check a page with the robots.txt tester: it reports noindex from the meta tag and the header, whether robots.txt blocks the URL, and the final verdict.

Why the noindex tag matters

A noindex is the most powerful line in SEO: one tag, and the page leaves Google's results after the next crawl, taking every visit from search with it. That is right for a thank-you page and a disaster on a category page. The classic accident is a staging site built with "Discourage search engines" switched on and copied to production with the switch still on. Nothing looks wrong in the browser; traffic drains over the following weeks. The WordPress setting that hides your site covers that case.

The opposite mistake is quieter. Pages you never meant to be found (internal search results, filter combinations, order confirmations) get indexed, compete with your real pages and show thin or private content to searchers.

How getReport checks it

The tester reads the HTML exactly as the server sends it, plus the response headers and your /robots.txt:

• Meta tags named robots and googlebot, looking for noindex or none.
• The X-Robots-Tag header, including values aimed at one crawler such as googlebot: noindex.
• robots.txt, tested for Googlebot and for all crawlers (*).

indexable-status is the summary: HTTP 200, no noindex, and allowed in robots.txt. Its evidence line shows all three, so one glance tells you which one fails. The meta tags checker runs the same noindex check alongside the title and description.

The SEO module of the full report also counts link attributes:

It is information only and costs no points. It counts links whose rel contains nofollow; a link marked only sponsored or ugc is not counted.

Because the tester reads the served HTML, a noindex that JavaScript removes after load still shows up. That matches Google: its JavaScript SEO documentation says that when Google sees noindex in the initial HTML it may skip rendering, so removing the tag with JavaScript is not reliable.

How to write the robots meta tag

The meta robots tag, written <meta name="robots" content="...">, goes in the <head> of the page:

<meta name="robots" content="noindex, follow">

name="robots" applies to all crawlers; name="googlebot" to Google only (bingbot to Bing). Directives are comma-separated and not case-sensitive. When several tags, or a tag and a header, disagree, Google applies the most restrictive combination. With no tag at all the default is index, follow, so you never need to write index or all.

noindex, nofollow: what each half does

• noindex keeps this page out of results. Google still crawls it, reads it and, over time, crawls it less often.
• nofollow on the page asks crawlers not to follow any link on it, internal ones included.
• noindex, follow keeps the page out of results but lets crawlers follow its links. See the next section.
• none is shorthand for noindex, nofollow.

noindex, follow: what it means and how long it lasts

<meta name="robots" content="noindex, follow"> says: do not show this page, but do follow the links on it. Because follow is the default, noindex, follow and plain noindex behave the same; writing follow only makes the intent visible to the next person who reads the template. It is the usual choice for pages you hide but that link to pages you want found: tag archives, internal search results, paginated filters.

The catch is time. Google's John Mueller said in a December 2017 office-hours hangout that when a page stays noindex for a long time, Google eventually drops it completely and stops following its links, so in the long run noindex, follow works like noindex, nofollow. Google has never said how long "long" is, and the statement is not in its written documentation, which only says a noindexed page may take months to be recrawled. Treat it as likely: a page that stays noindex is crawled less and less, so its links lose weight. Every page you want indexed needs a link from an indexable page or a place in the XML sitemap, not only a path through hidden ones.

Every directive Google supports

These are the directives in Google's robots meta tag specification on Search Central; the robots directives fix page has the short version.

Two older values do nothing in Google now. noarchive (and its alias nocache) controlled the cached copy, which Google removed in 2024; it moved the rule to a historical section of its documentation, though Bing still reads it. nositelinkssearchbox controlled the sitelinks search box, which Google retired in November 2024.

For part of a page, the data-nosnippet attribute keeps an element's text out of snippets:

<p data-nosnippet>Prices shown are for members only.</p>

The snippet directives are the only direct control you have over the text under your result; the meta description guide covers the indirect one.

X-Robots-Tag: noindex for files and whole hosts

The header accepts the same directives and works for anything the server sends, including PDFs, images and whole sites.

nginx, for all PDFs:

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex" always;
}

Apache, in .htaccess or the virtual host:

<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "noindex"
</FilesMatch>

For a staging host, the whole server block:

add_header X-Robots-Tag "noindex, nofollow" always;

A header can target one crawler: X-Robots-Tag: googlebot: noindex. Check it from the command line, since it never shows in the page source: curl -sI https://example.com/file.pdf | grep -i x-robots-tag.

Noindex vs robots.txt

They answer different questions. robots.txt says "do not fetch this URL"; noindex says "you may fetch it, but do not list it".

• A page blocked in robots.txt cannot be read, so its noindex is never seen. If other sites link to it, it can still appear in results as a bare URL.
• Google stopped honouring Noindex: lines inside robots.txt on 1 September 2019. Such lines do nothing.
• To remove an already indexed page, allow crawling, add noindex, wait until it drops out (URL Inspection shows it), and only then add a Disallow if you also want to save the crawling.

The robots.txt guide covers the file itself, and canonical vs redirect vs noindex covers the choice for duplicate and retired pages.

Nofollow on the page vs rel on a link

Page-level nofollow applies to every link on the page. Usually you want to mark individual links:

<a href="https://partner.example/offer" rel="sponsored">Our partner's offer</a>
<a href="https://someone.example/" rel="ugc nofollow">Commenter's site</a>

• sponsored: paid placements, affiliate links, anything you were compensated for.
• ugc: links in comments, forum posts, reviews.
• nofollow: any other link you do not want to vouch for.

Google announced in 2019 that it treats all three as hints rather than orders, for crawling and indexing too since March 2020, and accepts combinations. Leave internal links and links to sources you trust without rel values.

Which one for which page

• Staging and development sites: a password (HTTP authentication) first, X-Robots-Tag: noindex, nofollow as a second layer. Never only robots.txt. See the staging sites fix page.
• Thank-you and order confirmation pages: noindex, crawlable.
• Internal search results: noindex. If they are not indexed yet, a Disallow: /search saves the crawling too.
• Filter combinations (?colour=red&size=m): see faceted navigation and parameter URLs; on very large shops, Google's guidance favours blocking crawling in robots.txt.
• Paginated category pages (/shoes/page/2/): keep them indexable with their own canonical, so products linked only from later pages are still found.
• Offers that expire: unavailable_after with the end date, or a redirect after it ends.

Platform notes

WordPress

• Settings → Reading → "Discourage search engines from indexing this site" makes every page send <meta name='robots' content='noindex, nofollow' />. getReport's WordPress checks flag it separately, because it is so often left on after launch.
• With the box unticked, WordPress 5.7 and later add max-image-preview:large to every page by default.
• Yoast SEO, per page: the Advanced section of the Yoast box, "Allow search engines to show this content in search results?". Per content type: Yoast SEO → Settings → Content types.
• Rank Math, per page: the Advanced tab, Robots Meta. Site-wide defaults: Rank Math SEO → Titles & Meta.

A setting in the SEO plugin and the Reading setting can both produce a tag. The tester lists every source it found.

Shopify

Shopify's built-in way is the seo.hidden metafield: under Settings → Metafields and metaobjects, add an integer metafield with namespace seo and key hidden to products, pages or blog posts, and set it to 1 on the item to hide. Shopify then adds noindex and drops the item from the sitemap and the store's own search. Shopify's help centre also describes adding a condition to theme.liquid inside <head>:

{% if handle contains 'thank-you-offer' %}
  <meta name="robots" content="noindex">
{% endif %}

Static sites and custom code

Put the meta tag in the template for the page type, or set X-Robots-Tag in the host's headers file (_headers on Netlify and Cloudflare Pages).

How to check it worked

• Run the robots.txt tester. For pages you want found, robots-meta-noindex passes, robots-txt-blocks-page reads "robots.txt allows this page" and indexable-status reads "The page can be indexed". For pages you want hidden, the noindex finding is the confirmation.
• In Search Console, URL Inspection → Test live URL shows "Indexing allowed?" for the live version. After removing a noindex, use Request indexing.

Common mistakes

• noindex left on after launch. Search traffic declines over weeks with nothing visibly wrong.
• Blocking CSS and JavaScript in robots.txt. Google cannot render the page properly.
• nofollow on internal links. It only stops Google following links to your own pages. Remove it.
• Canonical and noindex on the same page. They contradict each other: "this is a copy of page X" and "drop this page". Use the canonical alone for duplicates.
• Disallow and noindex together on a page you want removed. Google cannot fetch the page, so it never sees the noindex.

Questions people ask

What does noindex, nofollow mean?

It tells search engines two things: do not show this page in results (noindex) and do not follow any of the links on it (nofollow). It is written as <meta name="robots" content="noindex, nofollow"> or sent as an X-Robots-Tag header, and none means the same. For most hidden pages, noindex alone is better, because the links on the page can still help crawlers find your other pages.

Is noindex, follow the same as noindex?

Yes, in effect. follow is the default, so <meta name="robots" content="noindex"> already lets crawlers follow the links. Writing noindex, follow only states it. Over time both work like noindex, nofollow: Google's John Mueller said in 2017 that a page left on noindex for long is eventually dropped and its links no longer followed. Make sure the pages it links to are reachable from indexable pages too.

What is the difference between noindex and nofollow?

noindex is about the page itself: it keeps the page out of search results. nofollow is about the links on it: it asks crawlers not to follow them. You can use either alone or both together. On a single link, rel="nofollow" marks just that link, which is usually what you want for paid, affiliate or user-submitted links, rather than a page-wide nofollow.

How long does noindex take to remove a page from Google?

Usually days to a few weeks. Google has to recrawl the page and see the tag, which happens sooner on pages it visits often. To speed it up, inspect the URL in Search Console and request indexing. For an urgent removal, the Removals tool hides the URL for about six months while the noindex takes effect. The page must stay crawlable throughout.

Does noindex stop Google from crawling a page?

No. Google still has to crawl a noindex page to see the directive, and keeps crawling it from time to time, less often as time passes. That is why noindex does not save crawl budget on large sites. To stop crawling, use a robots.txt Disallow, but only after the page has dropped out of the index, because a blocked page's noindex can no longer be read.

What is the X-Robots-Tag header?

It is an HTTP response header that carries the same directives as the robots meta tag, such as X-Robots-Tag: noindex. Because it is sent by the server rather than written in the HTML, it works for PDFs, images and other files that have no <head>, and for whole hosts such as a staging site. Set it in the nginx, Apache or host configuration, and check it with curl -I.