# How to write and fix robots.txt

> robots.txt tells crawlers which paths they may fetch. A missing file is harmless; a wrong rule can hide the whole site from Google.

Technical SEO · HTML version: https://getreport.app/learn/robots-txt

## robots.txt rules for this page

Passing looks like: robots.txt allows this page.

**Why it matters.** A Disallow rule stops search engines from crawling this page, so they cannot read its content or see any updates. The page can still appear in results as a bare URL with no description.

**How to fix it.**

1. Remove or narrow the Disallow rule in robots.txt so this URL is allowed for all crawlers.
2. If the page should stay out of search, allow crawling and use a noindex tag instead; a blocked page cannot be de-indexed.

## robots.txt file

Passing looks like: robots.txt is present.

**Why it matters.** robots.txt tells crawlers which parts of the site to skip and where the sitemap is. Without one, crawlers use their own defaults and may waste time on search results or admin pages.

**How to fix it.**

1. Publish a plain-text robots.txt at the site root with at least "User-agent: *", "Allow: /" and a "Sitemap:" line.
2. Most CMSs generate one; in WordPress it is under the SEO plugin's Tools section.

Check your own page: https://getreport.app/