How to write and fix robots.txt
SEO1 min readFixes 2 checks robots-txt-blocks-page, robots-txt-present
robots.txt tells crawlers which paths they may fetch. A missing file is harmless; a wrong rule can hide the whole site from Google.
Step by step, with screenshots: "Discourage search engines": the WordPress box that hides your site →
Check your own site
Runs these 2 checks and the other 185, free, in about 45 seconds.
What a passing site looks like
- robots.txt allows this pagefail · −10 ptseffort S
- robots.txt is presentwarning · −1 ptseffort S
1. robots.txt rules for this page
Why it matters. A Disallow rule stops search engines from crawling this page, so they cannot read its content or see any updates. The page can still appear in results as a bare URL with no description.
How to fix it.
- Remove or narrow the Disallow rule in robots.txt so this URL is allowed for all crawlers.
- If the page should stay out of search, allow crawling and use a noindex tag instead; a blocked page cannot be de-indexed.
2. robots.txt is missing
Why it matters. robots.txt tells crawlers which parts of the site to skip and where the sitemap is. Without one, crawlers use their own defaults and may waste time on search results or admin pages.
How to fix it.
- Publish a plain-text robots.txt at the site root with at least "User-agent: *", "Allow: /" and a "Sitemap:" line.
- Most CMSs generate one; in WordPress it is under the SEO plugin's Tools section.