# WordPress robots.txt: where it is, how to edit it and a safe default > WordPress serves a virtual robots.txt until you add a real file. Where it comes from, how to edit it with Yoast SEO, Rank Math or a file, and a safe default for blogs and WooCommerce shops. Updated 2026-09-26 · WordPress & WooCommerce · HTML version: https://getreport.app/guides/wordpress-robots-txt The WordPress robots.txt is usually not a file at all: when no `robots.txt` exists in the site root, WordPress generates one on every request, with a rule for the admin area and a link to your sitemap. You can change it in your SEO plugin, or replace it by uploading a real file, which then wins. This guide shows where the WordPress robots.txt comes from, how to edit it in Yoast SEO, Rank Math or over FTP, a safe default for blogs and WooCommerce shops, and how to check the result. For how robots.txt works on any site, start with the [robots.txt guide](https://getreport.app/guides/robots-txt). ## Quick answer - **Open it first:** go to `https://yourdomain.com/robots.txt`. That is what Google reads, whatever the dashboard says. - **No file on the server is normal.** WordPress builds a virtual robots.txt when none exists. - **Edit in one place:** Yoast SEO under Yoast SEO → Tools → File editor, Rank Math under Rank Math SEO → General Settings → Edit robots.txt, or a real file uploaded to the folder that holds `wp-config.php`. - **A real file always wins** over the virtual one, including over the Rank Math editor. - **A safe default** blocks `/wp-admin/` (with `admin-ajax.php` allowed), nothing else that Google needs, and lists your sitemap. - **Do not block** `/wp-content/`, `/wp-includes/`, themes, plugins or uploads. Google needs the CSS, JavaScript and images to render your pages. - **"Discourage search engines"** does not touch robots.txt in current WordPress. It adds `noindex` to every page instead. ## Where is robots.txt in WordPress? WordPress answers `/robots.txt` from PHP when the web server finds no real file with that name. The function behind it, `do_robots()`, writes this in current WordPress: ```text User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/wp-sitemap.xml ``` The first three lines come from WordPress core: every crawler may fetch everything except the admin area, and one admin file that themes and plugins call from the front end stays open. The `Sitemap:` line is added by the built-in sitemap feature that arrived in WordPress 5.5, and only when the site is public. SEO plugins that replace the core sitemap also replace this line with their own, for example `sitemap_index.xml` for Yoast SEO and Rank Math. That is why you will not find `robots.txt` over FTP or in your host's file manager on a fresh install, even though the URL works. The file only exists as the answer to a request. ### What the "Discourage search engines" box does to it Settings → Reading has a box called "Discourage search engines from indexing this site". Before WordPress 5.3 it changed the virtual file to `Disallow: /`. Since 5.3, the WordPress source says it no longer does that; it adds a `noindex` robots meta tag to every page instead, and drops the `Sitemap:` line. So a site can disappear from Google while its robots.txt looks perfectly normal. The guide on [the WordPress box that hides your site](https://getreport.app/guides/wordpress-discourage-search-engines) covers finding and fixing it, and getReport reports it as its own finding: > **Check: Search engines are not discouraged in Settings → Reading.** The WordPress setting adds a site-wide noindex, nofollow tag. It is meant for sites under construction and is often left on after launch, which removes the whole site from Google. > > 1. In wp-admin go to Settings → Reading and untick "Discourage search engines from indexing this site". > 2. Then request indexing in Google Search Console; pages usually return within days to weeks. ## How to edit robots.txt in WordPress There are three ways. Pick one and stick to it, because two editors fighting over one file is how old rules come back. ### With Yoast SEO 1. Go to **Yoast SEO → Tools → File editor**. 2. If no file exists yet, click **Create robots.txt file**. Yoast writes a real file in the site root, starting with its own block: ```text # START YOAST BLOCK # --------------------------- User-agent: * Disallow: Sitemap: https://example.com/sitemap_index.xml # --------------------------- # END YOAST BLOCK ``` 3. Edit the text and save. The editor only appears when WordPress may write files. On hosts that set `DISALLOW_FILE_EDIT` in `wp-config.php`, or where the root folder is not writable, it is missing or read-only, and you edit the file over FTP instead. Yoast's crawl optimisation settings (Yoast SEO → Settings → Advanced → Crawl optimization) can also add lines, such as rules that stop crawling of internal site search URLs, and its bot-blocking toggles add groups for specific crawlers. If lines appear that you did not type, look there first. ### With Rank Math 1. Go to **Rank Math SEO → General Settings → Edit robots.txt**. 2. Type your rules and save. Rank Math edits the virtual file and stores your text in the database. If a real `robots.txt` exists in the root, changes in Rank Math's editor have no effect, because the web server answers with the real file before WordPress runs. Rank Math's own documentation says to delete the real file, or edit it over FTP instead. ### With a real file over FTP or the file manager 1. Create a plain text file named `robots.txt`, all lowercase, saved as UTF-8. 2. Upload it to the folder that contains `wp-config.php` and `wp-admin`, usually `public_html` or `www`. 3. Open `https://yourdomain.com/robots.txt` in a private window to confirm it loads. A real file is the most predictable option: it works without any plugin, survives a plugin switch and is easy to keep in version control. The catch is that nobody sees it from the dashboard, so an old uploaded file can quietly override everything you later set in a plugin. If your plugin's editor seems to do nothing, check for a real file first. ## A safe default robots.txt for WordPress Most WordPress sites need very little. This is a safe starting point for a blog or company site: ```text # https://example.com/robots.txt User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap_index.xml ``` Use the sitemap address your site really serves: `wp-sitemap.xml` for WordPress core, `sitemap_index.xml` for Yoast SEO and Rank Math, or the one your sitemap plugin shows. The line must be a full URL. ### For WooCommerce shops Shops generate endless URL variations, so a few more lines help: ```text # https://example.com/robots.txt User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /*?add-to-cart= Disallow: /*&add-to-cart= Sitemap: https://example.com/sitemap_index.xml ``` `add-to-cart` links are ordinary links in product lists, so crawlers that follow them fill carts and fetch pages that have no search value. Blocking the parameter stops that. You do not need to block the cart, checkout and My account pages. WooCommerce already sends `noindex` on those three pages, and Google can only see that tag on a page it is allowed to fetch. WordPress core also adds `noindex` to internal search results, so a `Disallow: /?s=` line is optional: add it only if crawlers hammer your search pages, which a [server log review](https://getreport.app/guides/server-log-analysis-googlebot-and-ai-bots) will show. For filtered and sorted product lists, such as `?orderby=` or layered-navigation filters, the choice depends on how your theme builds them. The guide to [faceted navigation and parameter URLs](https://getreport.app/guides/faceted-navigation-and-parameter-urls) helps you decide, and [robots.txt patterns](https://getreport.app/guides/robots-txt-patterns-wildcards-allow-crawl-delay) has the wildcard syntax. ### Lines to remove if you find them Older tutorials recommended rules that now do harm. Delete these if your file has them: - `Disallow: /wp-content/` or `Disallow: /wp-content/plugins/` and `/themes/`: they block the CSS and JavaScript Google needs to render the page. - `Disallow: /wp-includes/`: same problem; core scripts such as jQuery live there. - `Disallow: /wp-content/uploads/`: blocks your images from Google Images and from the page render. - `Disallow: /feed/`, `/tag/`, `/category/` or `/author/`: if you do not want those archives in Google, set them to `noindex` in your SEO plugin instead, so Google can see the tag. - `Noindex:` or `Crawl-delay:` lines: Google ignores both; see [why noindex in robots.txt does not work](https://getreport.app/guides/robots-txt-noindex). ### AI crawlers If you want to keep AI training crawlers out, add their groups above the `*` group, as described in [AI crawlers and robots.txt](https://getreport.app/guides/ai-crawlers-and-robots-txt). Check what else already decides this for you: some SEO plugins have their own AI-bot toggles, and if the site sits behind Cloudflare, its managed robots.txt can add AI rules to your file; the [Cloudflare robots.txt guide](https://getreport.app/guides/cloudflare-robots-txt) explains how. ## How to check your WordPress robots.txt > **Free tool:** [robots.txt tester: check and validate robots.txt](https://getreport.app/tools/robots-txt-tester): Free robots.txt tester: see whether robots.txt blocks a page for Googlebot, whether your sitemap is declared and reachable, and if the page can be indexed. Enter any page URL. The tester fetches the live robots.txt from that host, so it sees exactly what Google sees, whether the answer came from a real file, a plugin or a CDN. It tells you whether that page is allowed for Googlebot and for all other agents, names the rule when one blocks it, fetches every sitemap the file declares and reads its URL count, and checks the page's robots meta tag and `X-Robots-Tag` header. The final verdict says whether the page can be indexed, which also catches the "Discourage search engines" box. Test at least the home page, a post, a page, a category and, on a shop, a product and a product category. A rule that blocks one section shows as allowed everywhere else. > **Check: robots.txt allows this page.** A Disallow rule stops search engines from crawling this page, so they cannot read its content or see any updates. The page can still appear in results as a bare URL with no description. > > 1. Remove or narrow the Disallow rule in robots.txt so this URL is allowed for all crawlers. > 2. If the page should stay out of search, allow crawling and use a noindex tag instead; a blocked page cannot be de-indexed. > **Check: robots.txt declares a sitemap.** A "Sitemap:" line in robots.txt is how crawlers find your XML sitemap without you registering it anywhere. Without it, new pages are discovered later. > > 1. Add a line to robots.txt: Sitemap: https://your-site.com/sitemap.xml (use the full URL). In Google Search Console, **Settings → robots.txt** shows when Google last fetched your file and lets you ask for a fresh fetch after a change. Google usually caches robots.txt for up to 24 hours. ## Common mistakes - **Editing in the plugin while a real file exists.** The real file wins. Delete it or edit it directly. - **Two SEO plugins, two opinions.** Yoast SEO and Rank Math both write robots.txt and sitemap lines; run one. See [running two SEO plugins together](https://getreport.app/guides/duplicate-seo-plugins-yoast-and-rank-math-together). - **A staging `Disallow: /` that went live.** Staging plugins and host tools often block crawlers on the copy; check the file on launch day. [Test robots.txt before it hides your site](https://getreport.app/guides/robots-txt-before-it-hides-your-site) covers the staging trap. - **Blocking a page to remove it from Google.** It stays indexed as a bare URL. Use `noindex` and let Google crawl. - **A sitemap line that points at a sitemap you no longer serve,** after switching SEO plugins. Open the URL in the file and make sure it loads. - **Caching plugins and CDNs serving an old copy.** After a change, purge the cache and check `/robots.txt` in a private window. ## Questions people ask ### Why can't I find robots.txt in my WordPress files? Because WordPress generates it on request. When there is no real `robots.txt` in the site root, WordPress answers the URL itself with a virtual file, so the address works but nothing shows up over FTP or in the file manager. To change it, use your SEO plugin's editor or upload your own file, which replaces the virtual one. ### How do I edit robots.txt in WordPress without a plugin? Create a plain text file named `robots.txt`, write your rules, and upload it over FTP or your host's file manager to the folder that contains `wp-config.php`. The web server then serves your file instead of the one WordPress generates. Open `/robots.txt` in a private window afterwards to confirm the new rules are live. ### What is the default WordPress robots.txt? Current WordPress serves `User-agent: *`, `Disallow: /wp-admin/` and `Allow: /wp-admin/admin-ajax.php`, plus a `Sitemap:` line for `wp-sitemap.xml` when the site is public. SEO plugins usually swap the sitemap line for their own. It allows everything else, which suits most sites; shops may add a rule for `add-to-cart` links. ### Should I block wp-content in robots.txt? No. `/wp-content/` holds your theme, plugin files and uploads, including the CSS, JavaScript and images Google needs to render your pages and show your images. Blocking it can make pages look broken to Google and drops your images from image search. Block only the admin area and URL patterns with no search value. ### Why does my WordPress robots.txt not change after I edit it? Usually because something else answers first. A real `robots.txt` in the site root overrides both the virtual file and the Rank Math editor; a caching plugin, host cache or CDN may serve an old copy; and Google itself caches the file for up to 24 hours. Delete or edit the real file, purge caches, then check the live URL.