WordPress robots.txt: where it is, how to edit it and a safe default

The WordPress robots.txt is usually not a file at all: when no robots.txt exists in the site root, WordPress generates one on every request, with a rule for the admin area and a link to your sitemap. You can change it in your SEO plugin, or replace it by uploading a real file, which then wins. This guide shows where the WordPress robots.txt comes from, how to edit it in Yoast SEO, Rank Math or over FTP, a safe default for blogs and WooCommerce shops, and how to check the result. For how robots.txt works on any site, start with the robots.txt guide.

Quick answer

• Open it first: go to https://yourdomain.com/robots.txt. That is what Google reads, whatever the dashboard says.
• No file on the server is normal. WordPress builds a virtual robots.txt when none exists.
• Edit in one place: Yoast SEO under Yoast SEO → Tools → File editor, Rank Math under Rank Math SEO → General Settings → Edit robots.txt, or a real file uploaded to the folder that holds wp-config.php.
• A real file always wins over the virtual one, including over the Rank Math editor.
• A safe default blocks /wp-admin/ (with admin-ajax.php allowed), nothing else that Google needs, and lists your sitemap.
• Do not block /wp-content/, /wp-includes/, themes, plugins or uploads. Google needs the CSS, JavaScript and images to render your pages.
• "Discourage search engines" does not touch robots.txt in current WordPress. It adds noindex to every page instead.

Where is robots.txt in WordPress?

WordPress answers /robots.txt from PHP when the web server finds no real file with that name. The function behind it, do_robots(), writes this in current WordPress:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

The first three lines come from WordPress core: every crawler may fetch everything except the admin area, and one admin file that themes and plugins call from the front end stays open. The Sitemap: line is added by the built-in sitemap feature that arrived in WordPress 5.5, and only when the site is public. SEO plugins that replace the core sitemap also replace this line with their own, for example sitemap_index.xml for Yoast SEO and Rank Math.

That is why you will not find robots.txt over FTP or in your host's file manager on a fresh install, even though the URL works. The file only exists as the answer to a request.

What the "Discourage search engines" box does to it

Settings → Reading has a box called "Discourage search engines from indexing this site". Before WordPress 5.3 it changed the virtual file to Disallow: /. Since 5.3, the WordPress source says it no longer does that; it adds a noindex robots meta tag to every page instead, and drops the Sitemap: line. So a site can disappear from Google while its robots.txt looks perfectly normal. The guide on the WordPress box that hides your site covers finding and fixing it, and getReport reports it as its own finding:

How to edit robots.txt in WordPress

There are three ways. Pick one and stick to it, because two editors fighting over one file is how old rules come back.

With Yoast SEO

1. Go to Yoast SEO → Tools → File editor.

2. If no file exists yet, click Create robots.txt file. Yoast writes a real file in the site root, starting with its own block:

   # START YOAST BLOCK
   # ---------------------------
   User-agent: *
   Disallow:
   
   Sitemap: https://example.com/sitemap_index.xml
   # ---------------------------
   # END YOAST BLOCK

3. Edit the text and save.

The editor only appears when WordPress may write files. On hosts that set DISALLOW_FILE_EDIT in wp-config.php, or where the root folder is not writable, it is missing or read-only, and you edit the file over FTP instead.

Yoast's crawl optimisation settings (Yoast SEO → Settings → Advanced → Crawl optimization) can also add lines, such as rules that stop crawling of internal site search URLs, and its bot-blocking toggles add groups for specific crawlers. If lines appear that you did not type, look there first.

With Rank Math

1. Go to Rank Math SEO → General Settings → Edit robots.txt.
2. Type your rules and save.

Rank Math edits the virtual file and stores your text in the database. If a real robots.txt exists in the root, changes in Rank Math's editor have no effect, because the web server answers with the real file before WordPress runs. Rank Math's own documentation says to delete the real file, or edit it over FTP instead.

With a real file over FTP or the file manager

1. Create a plain text file named robots.txt, all lowercase, saved as UTF-8.
2. Upload it to the folder that contains wp-config.php and wp-admin, usually public_html or www.
3. Open https://yourdomain.com/robots.txt in a private window to confirm it loads.

A real file is the most predictable option: it works without any plugin, survives a plugin switch and is easy to keep in version control. The catch is that nobody sees it from the dashboard, so an old uploaded file can quietly override everything you later set in a plugin. If your plugin's editor seems to do nothing, check for a real file first.

A safe default robots.txt for WordPress

Most WordPress sites need very little. This is a safe starting point for a blog or company site:

# https://example.com/robots.txt
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

Use the sitemap address your site really serves: wp-sitemap.xml for WordPress core, sitemap_index.xml for Yoast SEO and Rank Math, or the one your sitemap plugin shows. The line must be a full URL.

For WooCommerce shops

Shops generate endless URL variations, so a few more lines help:

# https://example.com/robots.txt
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*?add-to-cart=
Disallow: /*&add-to-cart=

Sitemap: https://example.com/sitemap_index.xml

add-to-cart links are ordinary links in product lists, so crawlers that follow them fill carts and fetch pages that have no search value. Blocking the parameter stops that.

You do not need to block the cart, checkout and My account pages. WooCommerce already sends noindex on those three pages, and Google can only see that tag on a page it is allowed to fetch. WordPress core also adds noindex to internal search results, so a Disallow: /?s= line is optional: add it only if crawlers hammer your search pages, which a server log review will show.

For filtered and sorted product lists, such as ?orderby= or layered-navigation filters, the choice depends on how your theme builds them. The guide to faceted navigation and parameter URLs helps you decide, and robots.txt patterns has the wildcard syntax.

Lines to remove if you find them

Older tutorials recommended rules that now do harm. Delete these if your file has them:

• Disallow: /wp-content/ or Disallow: /wp-content/plugins/ and /themes/: they block the CSS and JavaScript Google needs to render the page.
• Disallow: /wp-includes/: same problem; core scripts such as jQuery live there.
• Disallow: /wp-content/uploads/: blocks your images from Google Images and from the page render.
• Disallow: /feed/, /tag/, /category/ or /author/: if you do not want those archives in Google, set them to noindex in your SEO plugin instead, so Google can see the tag.
• Noindex: or Crawl-delay: lines: Google ignores both; see why noindex in robots.txt does not work.

AI crawlers

If you want to keep AI training crawlers out, add their groups above the * group, as described in AI crawlers and robots.txt. Check what else already decides this for you: some SEO plugins have their own AI-bot toggles, and if the site sits behind Cloudflare, its managed robots.txt can add AI rules to your file; the Cloudflare robots.txt guide explains how.

How to check your WordPress robots.txt

Enter any page URL. The tester fetches the live robots.txt from that host, so it sees exactly what Google sees, whether the answer came from a real file, a plugin or a CDN. It tells you whether that page is allowed for Googlebot and for all other agents, names the rule when one blocks it, fetches every sitemap the file declares and reads its URL count, and checks the page's robots meta tag and X-Robots-Tag header. The final verdict says whether the page can be indexed, which also catches the "Discourage search engines" box.

Test at least the home page, a post, a page, a category and, on a shop, a product and a product category. A rule that blocks one section shows as allowed everywhere else.

In Google Search Console, Settings → robots.txt shows when Google last fetched your file and lets you ask for a fresh fetch after a change. Google usually caches robots.txt for up to 24 hours.

Common mistakes

• Editing in the plugin while a real file exists. The real file wins. Delete it or edit it directly.
• Two SEO plugins, two opinions. Yoast SEO and Rank Math both write robots.txt and sitemap lines; run one. See running two SEO plugins together.
• A staging Disallow: / that went live. Staging plugins and host tools often block crawlers on the copy; check the file on launch day. Test robots.txt before it hides your site covers the staging trap.
• Blocking a page to remove it from Google. It stays indexed as a bare URL. Use noindex and let Google crawl.
• A sitemap line that points at a sitemap you no longer serve, after switching SEO plugins. Open the URL in the file and make sure it loads.
• Caching plugins and CDNs serving an old copy. After a change, purge the cache and check /robots.txt in a private window.

Questions people ask

Why can't I find robots.txt in my WordPress files?

Because WordPress generates it on request. When there is no real robots.txt in the site root, WordPress answers the URL itself with a virtual file, so the address works but nothing shows up over FTP or in the file manager. To change it, use your SEO plugin's editor or upload your own file, which replaces the virtual one.

How do I edit robots.txt in WordPress without a plugin?

Create a plain text file named robots.txt, write your rules, and upload it over FTP or your host's file manager to the folder that contains wp-config.php. The web server then serves your file instead of the one WordPress generates. Open /robots.txt in a private window afterwards to confirm the new rules are live.

What is the default WordPress robots.txt?

Current WordPress serves User-agent: *, Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php, plus a Sitemap: line for wp-sitemap.xml when the site is public. SEO plugins usually swap the sitemap line for their own. It allows everything else, which suits most sites; shops may add a rule for add-to-cart links.

Should I block wp-content in robots.txt?

No. /wp-content/ holds your theme, plugin files and uploads, including the CSS, JavaScript and images Google needs to render your pages and show your images. Blocking it can make pages look broken to Google and drops your images from image search. Block only the admin area and URL patterns with no search value.

Why does my WordPress robots.txt not change after I edit it?

Usually because something else answers first. A real robots.txt in the site root overrides both the virtual file and the Rank Math editor; a caching plugin, host cache or CDN may serve an old copy; and Google itself caches the file for up to 24 hours. Delete or edit the real file, purge caches, then check the live URL.