How to write and fix robots.txt
robots.txt tells crawlers which paths they may fetch. A missing file is harmless; a wrong rule can hide the whole site from Google.
Step by step, with screenshots: Fake Googlebot: how to verify Googlebot IP addresses →
Check your own site
Runs these 2 checks and the other 191, free, in about 45 seconds.
Or run only this check with the free Log file analyzer, the free robots.txt tester or the free robots.txt AI rules generator.
What it is
robots.txt is a plain-text file at the root of your site, such as https://example.com/robots.txt, that tells crawlers which paths they may and may not fetch. It follows the Robots Exclusion Protocol (RFC 9309) and is the first file most crawlers request.
It controls crawling, not indexing: a blocked URL can still appear in Google as a bare link if others point to it. One wrong line, such as Disallow: / left over from a staging site, stops search engines from reading your whole site. Google reads up to 500 KiB of the file and caches it for about a day.
OK looks like this: the file answers 200 as text, it blocks only sections that do not belong in search, it never blocks the page you test, and it lists your sitemap with a Sitemap: line. A missing file is harmless; getReport still suggests one for the sitemap line.
What a passing site looks like
- robots.txt allows this pagefail · −10 ptseffort S
- robots.txt is presentwarning · −1 ptseffort S
1. robots.txt rules for this page
Why it matters. A Disallow rule stops search engines from crawling this page, so they cannot read its content or see any updates. The page can still appear in results as a bare URL with no description.
- Remove or narrow the Disallow rule in robots.txt so this URL is allowed for all crawlers.
- If the page should stay out of search, allow crawling and use a noindex tag instead; a blocked page cannot be de-indexed.
2. robots.txt is missing
Why it matters. robots.txt tells crawlers which parts of the site to skip and where the sitemap is. Without one, crawlers use their own defaults and may waste time on search results or admin pages.
- Publish a plain-text robots.txt at the site root with at least "User-agent: *", "Allow: /" and a "Sitemap:" line.
- Most CMSs generate one; in WordPress it is under the SEO plugin's Tools section.
Questions people ask
What is robots.txt?
robots.txt is a text file at the root of a website that gives crawlers rules about which URLs they may fetch. Each group starts with a User-agent line naming a crawler, or * for all, followed by Allow and Disallow rules for paths. It is public, so never use it to hide private pages.
What does Disallow: / do in robots.txt?
Disallow: / under User-agent: * tells every crawler not to fetch any page on the site. It is common on staging sites and sometimes survives a launch, which makes the site slowly vanish from search. Change it to Allow: / or remove the line, then check the result in Google Search Console.
Why is robots.txt important?
Because one line in it decides whether search engines can read your site at all. A good file keeps crawlers focused on pages that matter and points them to your sitemap; a bad one blocks your content without any visible error. Check it after every launch, migration or theme change.
What does indexed, though blocked by robots.txt mean?
It is a Search Console warning that Google indexed a URL it is not allowed to crawl, usually because other pages link to it. The result shows no description because Google never read the page. If the page should rank, remove the Disallow rule; if it should vanish, allow crawling and add noindex.