Settings → Reading has one checkbox that can take a whole site out of Google: "Discourage search engines from indexing this site". It is meant for the weeks a site is being built, and it is the most common reason a WordPress site that looks finished gets no visits from search. This guide explains exactly what the box does, how it survives a launch, how to switch it off and how to get back into the index quickly. The fix takes two minutes; recovery takes days to weeks, and the steps below shorten it.
Quick answer
- Check: view the page source and search for
robots.<meta name='robots' content='noindex, nofollow' />on every page means the box is ticked. - Fix: Settings → Reading → untick "Discourage search engines from indexing this site" → Save Changes. From a terminal:
wp option update blog_public 1. - Purge the page cache and the CDN, or cached pages keep the tag for hours.
- Tell Google: resubmit the sitemap and request indexing for the home page and your top pages in Search Console.
- Expect important pages back within days and the rest over the following weeks.
- Staging needs a password (HTTP authentication or your host's staging feature), not this checkbox.
Why the checkbox matters
What it does, exactly
The box stores a single option, blog_public: 1 when the site is public, 0 when search engines are discouraged. Since WordPress 5.3, 0 makes WordPress print this tag in the <head> of every front-end page:
<meta name='robots' content='noindex, nofollow' />noindex asks search engines to drop the page from results; nofollow asks them not to follow its links. Google obeys both. Two side effects make it worse: WordPress's own sitemap at /wp-sitemap.xml switches off while the option is 0, so Google also loses the list of your URLs; and before version 5.3 the box worked differently, printing noindex,follow and adding Disallow: / to the virtual robots.txt WordPress serves when there is no robots.txt file on disk. Sites that were set up on an old version, or whose robots.txt was saved to a file back then, can still carry that line.
What it does not do
It hides nothing from people. There is no password and no login; anyone with the address can read every page. The settings screen itself says it is up to search engines to honour the request. Well-behaved crawlers do, scrapers and bad bots do not, and nothing stops a visitor from sharing a link. If the goal is privacy, this is the wrong tool; the staging sites learn page covers the right ones.
Why it stays on after launch
Three routes, all common:
- The installer offers it. The install screen has the same "Search engine visibility" checkbox. Ticked during the build, it is easy to forget on launch day because nothing on the site looks different.
- Migrations copy it.
blog_publicis an ordinary row in thewp_optionstable. It travels with every database export, every migration plugin and every "push to live", so a staging copy that was correctly hidden hands the setting to the live site. - Host staging tools set it. Several managed hosts tick the box on the staging copy they create. Pushing the staging database back to production later brings the tick along.
The symptom is silence. Pages load, forms work, and search traffic either never starts or drifts down to zero over a few weeks as Google recrawls and drops page after page. The only hint inside wp-admin is a small "Search engines discouraged" line in the Dashboard's At a Glance box.
How getReport checks it
The WordPress health check runs a normal report. When the page reveals WordPress (/wp-content/ or /wp-includes/ paths, the generator tag, the wp-json link or the emoji script), the WordPress Doctor findings switch on. For this setting, nothing extra is requested: the check reads the robots meta tags of the page it already fetched. A tag containing both noindex and nofollow on a WordPress page raises the finding, and the evidence line quotes the tag exactly as the page sent it.
The report looks at indexability from three more sides that do not depend on WordPress at all:
Read together, they tell you where the block comes from:
| What the report shows | What it usually means |
|---|---|
| Discourage finding and noindex finding both fail | The Reading box, or an SEO plugin set noindex, nofollow on this page |
| Noindex fails, Discourage passes | noindex without nofollow: an SEO plugin setting, a header, or WordPress older than 5.3 |
| robots.txt blocks the page | A Disallow rule in a robots.txt file or plugin, not the Reading box on current WordPress |
| Only "cannot be indexed" fails | The page answers something other than HTTP 200 |
The check sees one page. An SEO plugin can set noindex, nofollow on a single page, and that page raises the same finding. Run the report on the home page and on one ordinary post: if both are flagged, the cause is almost certainly site-wide.
Step by step
1. Confirm what is adding the tag
Three quick tests, from least to most technical:
- Dashboard → At a Glance. "Search engines discouraged" means the box is ticked.
- Settings → Reading → Search engine visibility. The checkbox shows the same thing.
- From a terminal on the server:
wp option get blog_public
# 0 = discouraged, 1 = publicIf the option is 1 and the tag is still in the page source, the Reading box is not the cause. Go to step 4.
2. Untick the box
Settings → Reading → untick "Discourage search engines from indexing this site" → Save Changes. Or:
wp option update blog_public 1On a multisite network, each site has its own option. List them all and fix the ones that print 0:
for url in $(wp site list --field=url); do
echo "$url $(wp option get blog_public --url="$url")"
done
wp option update blog_public 1 --url=https://shop.example.com/3. Purge every cache
A page cache stores the HTML with the old tag in it. Purge the cache plugin (WP Rocket, LiteSpeed Cache, W3 Total Cache and the rest all have a "clear all" button), the host's server cache if it has one, and the CDN (on Cloudflare: Caching → Configuration → Purge Everything). Then open a post in a private window, view the source and search for robots. With the box unticked, the tag either disappears or reads max-image-preview:large (current WordPress) or index, follow (most SEO plugins), all of which allow indexing.
4. Check the SEO plugin layer
SEO plugins set robots tags per page and per content type, on top of the Reading setting. A leftover shows as noindex on some pages rather than all.
- Yoast SEO: in the editor, the Yoast panel → Advanced → "Allow search engines to show this Page in search results?" should be Yes (or the default). Site-wide, Yoast SEO → Settings → Content types: "Show pages in search results" and the same switch for posts and products should be on.
- Rank Math: in the editor, the Rank Math panel → Advanced → Robots Meta should have Index, not No Index. Site-wide, Rank Math → Titles & Meta → Global Meta → Robots Meta, then the tab for each post type.
Yoast also shows a warning in its dashboard while the Reading box is ticked, which makes a useful second check.
5. Look at robots.txt
Open https://yoursite.com/robots.txt. A current WordPress install with no robots.txt file on disk serves something like:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/wp-sitemap.xmlIf you see Disallow: / under User-agent: *, a robots.txt file in the site root or a plugin's robots.txt editor is blocking everything. Remove that line. Blocked pages cannot be de-indexed properly either, because Google never reads their noindex; robots.txt: test it before it hides your site explains the difference.
6. Ask Google to come back
In Google Search Console:
- Sitemaps: submit
/wp-sitemap.xml, or the index your SEO plugin makes (/sitemap_index.xmlfor Yoast and Rank Math). Core's sitemap only exists again now that the box is unticked. - URL Inspection: enter the home page, click "Test live URL" and confirm it reports indexing as allowed, then click "Request indexing". Repeat for your five to ten most important pages. There is a daily limit, so start with the pages that earn money.
- Pages report: the "Excluded by 'noindex' tag" count should start falling within a week.
How long it takes depends on how long the tag was on and how often Google visits. A home page that was noindexed for a week is usually back within days, with rankings close to where they were. A site that was hidden for months has to be discovered almost from scratch; expect weeks for the long tail. Bing reads the same tag, and Bing Webmaster Tools has an equivalent URL submission.
7. Protect staging without the checkbox
A password is the real protection for a staging copy, and your host's staging feature or HTTP authentication gives you one; the WordPress rookie mistakes guide has the Apache and nginx snippets. If you also want the staging copy noindexed, set it from a file rather than the database, so no database push can ever carry it to production. On the staging copy only, in wp-config.php:
define('WP_ENVIRONMENT_TYPE', 'staging');Then, in a must-use plugin on every copy, wp-content/mu-plugins/staging-noindex.php:
<?php
// Forces "Discourage search engines" on any copy that is not production.
// wp-config.php is not part of the database, so a DB push cannot move this.
if (wp_get_environment_type() !== 'production') {
add_filter('pre_option_blog_public', '__return_zero');
}WordPress treats a site with no WP_ENVIRONMENT_TYPE as production, so the live site is unaffected, and on staging the Reading box shows as ticked whatever the database says.
Platform notes
Managed WordPress hosts
Host staging tools differ in how they keep the copy out of search: some tick the box on the copy, others put it behind a password or on a separate host domain. Find out what yours does before the first push to live, and check Settings → Reading on production straight after every push. The Doctor also requests the staging., dev. and test. subdomains of your domain; a copy there that answers without any noindex is reported as a public staging copy.
Sites installed before WordPress 5.3
Updating WordPress changes how the setting is expressed, but a robots.txt file written to disk in the old days stays. If the report flags robots-txt-blocks-page with Disallow: /, delete or edit that file even after the box is unticked.
Verify
- Re-run the WordPress health check on the home page and one post. "Search engines are not discouraged in Settings → Reading", "The page is not set to noindex" and "The page can be indexed" should all pass.
- In Search Console, URL Inspection's live test reports indexing as allowed for the home page.
- Over the next two to four weeks, Search Console → Performance shows impressions returning and the noindex exclusions in the Pages report shrinking.
Common mistakes
- Unticking the box but not purging the cache. Visitors and Googlebot keep getting cached pages with the tag. Purge the plugin cache, the server cache and the CDN, then check the source of a post.
- Treating the checkbox as a lock for staging. The content stays public and a later database push copies the setting to live. Use a password, and set any noindex from a file as in step 7.
- Fixing the box and missing the plugin settings. A content type set to "not in search results" in Yoast or Rank Math keeps every post of that type out. Check both layers.
- Adding
Disallow: /to hide a site. Blocked pages can still appear in results as bare addresses, and Google cannot see a noindex it is not allowed to fetch. Use noindex or a password, never both a block and a noindex. - Changing URLs or content while waiting. Rankings return fastest when the pages Google remembers come back unchanged. Wait two to four weeks after requesting indexing before judging anything else.
Google's own documentation on blocking indexing with noindex covers the rules behind the tag in more depth.