# robots.txt: what it is, how it works and how to write one that does not hide your site

> robots.txt is the plain text file that tells search engines and other crawlers which URLs on your site they may fetch. Learn how it works, how Google reads it, a safe file to start from and the mistakes that hide sites.

Updated 2026-09-26 · Technical SEO · HTML version: https://getreport.app/guides/robots-txt

robots.txt is a plain text file at the root of your site, such as `https://example.com/robots.txt`, that tells search engines and other crawlers which URLs they may fetch and where your sitemap is. It controls crawling, not indexing, and it is a request that well-behaved bots follow, not a lock. This guide is for anyone who owns or edits a site: it explains what the file does, how Google reads it line by line, what a safe robots.txt looks like for a typical site, and how to test it, so that the one file that can hide your whole site from Google never does.

## Quick answer

- **Location:** one file per host, at `/robots.txt` on that exact protocol and host. `https://example.com/robots.txt` does not cover `https://www.example.com/`.
- **Format:** groups that start with `User-agent:`, followed by `Allow:` and `Disallow:` rules for URL paths, plus optional `Sitemap:` lines with full URLs.
- **No file is fine.** A missing robots.txt (404) means everything may be crawled.
- **It controls crawling, not indexing.** A blocked page can still appear in Google as a bare URL. To keep a page out of results, allow crawling and use `noindex`.
- **Never ship `Disallow: /` to a live site**, and never block CSS or JavaScript; Google needs them to render your pages.
- **Google ignores `Crawl-delay` and `noindex` lines** in robots.txt.
- Test any URL against your live file with the free [robots.txt tester](https://getreport.app/tools/robots-txt-tester).

## What is robots.txt?

robots.txt is the file behind the Robots Exclusion Protocol, a convention from 1994 that became an internet standard as RFC 9309 in September 2022. Before a compliant crawler fetches pages from a host, it requests `/robots.txt` from that host, reads the rules that apply to it, and skips the URLs they disallow.

A typical file looks like this:

```text
# https://example.com/robots.txt
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?s=

Sitemap: https://example.com/sitemap.xml
```

In plain words: every crawler (`*`) may fetch everything except the admin area, the cart, the checkout and internal search results; one admin file that the front end needs stays open; and the sitemap lives at the address on the last line.

People search for it as "robots.txt", "robot.txt" and "robots txt", but only one spelling works: the file must be named `robots.txt`, all lowercase, and sit at the root of the host. A file at `/robot.txt`, `/Robots.txt` or `/seo/robots.txt` is never read.

### Who reads it

Googlebot, Bingbot, the other major search engine crawlers, most SEO tools and the declared AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) all read and follow it. Scrapers, spam bots and attackers do not have to, and many do not. That is why robots.txt is a traffic-management tool for honest crawlers and never a security measure.

## Where is robots.txt?

Always at the root of the host: type the domain followed by `/robots.txt` into your browser. That works for any public site, yours or a competitor's, because the file is public by design.

The rule is per protocol, host and port. Google's robots.txt documentation gives the examples: a file at `https://example.com/robots.txt` applies to `https://example.com/` only, not to `https://www.example.com/`, `http://example.com/` or `https://shop.example.com/`. Each subdomain needs its own file. Redirect the variants of your domain to one canonical host and you have one file to maintain.

On many sites there is no physical file on the server. WordPress, Shopify, Wix, Squarespace and most frameworks generate robots.txt on request, which is why you may not find it over FTP even though the URL works. The platform sections below say where each one lets you edit it.

## How crawlers read robots.txt

The file is short, but four rules decide what it actually blocks. Google follows RFC 9309 and has published its own parser as open source, so these rules are not guesswork.

### 1. A crawler uses one group

Each group starts with one or more `User-agent:` lines. A crawler picks the single group with the most specific match for its name, and falls back to `User-agent: *` only when no group names it. Groups do not add up: if you add a `User-agent: Googlebot` group with one rule, Googlebot ignores everything in the `*` group.

### 2. The longest matching rule wins

`Allow` and `Disallow` rules match the start of the URL path. When several rules match, the one with the longest path wins; when an `Allow` and a `Disallow` are the same length, Google applies the less restrictive one, the `Allow`. Order in the file does not matter.

```text
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
```

For `/wp-admin/admin-ajax.php`, both rules match, the `Allow` is longer, and the file may be fetched.

### 3. Rules are prefixes, with two wildcards

`Disallow: /p` blocks `/pricing/`, `/products/` and `/press/`, because a rule matches every path that starts with it. `*` matches any run of characters and `$` marks the end of the URL: `Disallow: /*.pdf$` blocks URLs that end in `.pdf` and nothing else. Paths are case-sensitive, so `/Private/` and `/private/` are different rules.

### 4. Unknown lines are ignored

Google supports four fields: `user-agent`, `allow`, `disallow` and `sitemap`. Anything else is skipped without an error, which includes a typo such as `Dissallow:`, and two lines people still add on purpose:

- **`Crawl-delay`** is not supported by Google. Bing does honour it; Googlebot slows down on its own when your server answers slowly or with 429 or 503 errors.
- **`Noindex:`** in robots.txt was never an official rule, and Google stopped honouring it on 1 September 2019. Use a robots meta tag or an `X-Robots-Tag` header.

For copy-paste patterns for parameters, file types, one-file exceptions and crawler-specific groups, read [robots.txt examples and syntax](https://getreport.app/guides/robots-txt-patterns-wildcards-allow-crawl-delay), which has complete files for a blog, WordPress, a shop, Next.js and AI crawlers.

### How Google fetches and caches the file

These details come from Google's Search Central page on how Google interprets robots.txt, and they matter when something breaks:

| What happens | What Google does |
| --- | --- |
| File answers 200 | Rules are applied |
| File redirects | Google follows up to 5 hops, then treats it as missing |
| 404, 410 or another 4xx (except 429) | No robots.txt: everything may be crawled |
| 5xx or 429 | Crawling stops for the first 12 hours; after that, the last cached copy is used for up to 30 days |
| File over 500 KiB | Rules after the first 500 KiB are ignored |
| After a change | Google usually caches the file for up to 24 hours |

The row people miss is the server error. A robots.txt that returns 500 does not mean "no rules"; it pauses crawling of the whole site. Make sure the file answers 200 or, if you have none, a clean 404.

## What robots.txt does not do

**It does not remove pages from Google.** A disallowed URL is not fetched, but Google can still index the URL if other pages link to it, and show it without a description. To keep a page out of results, let Google crawl it and add `<meta name="robots" content="noindex">` or an `X-Robots-Tag: noindex` header. A page that is both disallowed and noindexed stays indexed, because Google never sees the tag.

**It does not protect anything.** Every line in the file is public. Listing `/admin-backup/` tells everyone where to look. Put private areas behind a login or HTTP authentication.

**It does not bind every bot.** RFC 9309 itself says the rules are not a form of access authorization. Honest crawlers follow them; others do not have to. Blocking abusive bots is a job for your firewall, CDN or server rules.

The full list of indexing controls, and which one to use for staging, thank-you, search and filter pages, is in [noindex, nofollow and the other robots directives](https://getreport.app/guides/noindex-nofollow-complete-reference).

### robots.txt vs noindex

The two are often confused, so here is the short version. robots.txt says "do not fetch this URL". `noindex` says "you may fetch this page, but do not show it in results". Use robots.txt to keep crawlers away from sections with no search value, such as carts, internal search results and endless filter combinations. Use `noindex` for pages that must not appear in search, and never disallow a page you want noindexed, because Google cannot read a tag on a page it may not fetch.

Older sites sometimes still carry `Noindex:` lines in robots.txt. Google has ignored them since 1 September 2019, so those URLs have had no indexing control since. [Noindex in robots.txt: why it does not work](https://getreport.app/guides/robots-txt-noindex) shows how to move each old rule to a robots meta tag or an `X-Robots-Tag` header, with Apache and nginx snippets for folders and file types.

## Does robots.txt matter for SEO?

Yes, in two ways, one small and one large.

The small one is focus. On a shop or a large site, blocking infinite URL spaces such as sorted, filtered and search-result pages keeps Googlebot on the pages you want indexed, and the `Sitemap:` line helps crawlers find new pages sooner. For a site of a few hundred pages, Google says crawl budget is not something to worry about, and the file changes little; see [crawl budget for sites under 10,000 pages](https://getreport.app/guides/crawl-budget-for-sites-under-10000-pages).

The large one is damage. One wrong line can stop Google from reading your pages at all, and nothing on the site looks broken. The common cases:

- `Disallow: /` copied from a staging site to production.
- A rule broader than intended, such as `Disallow: /p` or `Disallow: /*?`.
- Blocked CSS and JavaScript folders, so Google renders a broken page.
- A robots.txt that answers with a server error, which pauses crawling.

That is why the file deserves a test after every launch, migration, theme change or CDN change, even though it is rarely edited.

## How to create a robots.txt file

Most sites should start from their platform's generated file and edit it there. If you write one yourself:

1. **Create a plain text file named `robots.txt`**, saved as UTF-8. Use a code editor or Notepad, not a word processor.
2. **Start with the group for everyone:** `User-agent: *`.
3. **Add `Disallow` lines only for sections with no search value:** admin areas, cart, checkout, account pages, internal search, sorted and filtered duplicates. When in doubt, leave it out; an allowed page costs nothing.
4. **Add the sitemap** as a full URL: `Sitemap: https://example.com/sitemap.xml`. You can list several.
5. **Upload it to the root** of every host that serves pages, so it opens at `https://yourdomain/robots.txt`.
6. **Test a URL from each section** you care about with a tester before you rely on it.

The smallest useful file for a site that wants everything crawled:

```text
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
```

An empty `Disallow:` line means the same as `Allow: /`. Both are valid.

## Disallow rules and the accidental site-wide block

`Disallow` is the line that does all the work and all the damage. `User-agent: *` followed by `Disallow: /` asks every crawler to fetch nothing, which is right for a staging copy and a disaster on a live site: pages drop out of search over the following days and weeks, and already indexed URLs linger as results without descriptions.

The protection is process, not syntax: keep robots.txt out of the deployment (generate it per environment, or serve it from a rule that checks the hostname), protect staging with a password rather than robots.txt alone, and run the tester after every launch. The guide to [robots.txt Disallow rules](https://getreport.app/guides/robots-txt-before-it-hides-your-site) covers blocking a folder, a file type or one crawler, disallow all and allow all, the staging trap and how to recover when it happened.

## robots.txt on WordPress

WordPress serves a virtual robots.txt when no physical file exists. Since WordPress 5.5 it looks like this, with your own domain:

```text
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml
```

SEO plugins replace the sitemap line with their own and let you edit the file: in Yoast SEO under Tools → File editor, in Rank Math under General Settings → Edit robots.txt. Yoast's editor writes a physical file; Rank Math edits the virtual one. A physical `robots.txt` in the site root always wins over the virtual file, so an old uploaded file can silently override the rules you set in a plugin.

The "Discourage search engines from indexing this site" box under Settings → Reading is a separate switch. In current WordPress it adds a `noindex` robots meta tag to every page rather than a `Disallow` line, which is why a site can vanish from Google with a perfectly normal robots.txt; see [the WordPress box that hides your site](https://getreport.app/guides/wordpress-discourage-search-engines).

The [WordPress robots.txt guide](https://getreport.app/guides/wordpress-robots-txt) walks through editing the file in Yoast SEO, Rank Math or over FTP, a safe default for blogs and WooCommerce shops, and the old rules, such as blocking `/wp-content/`, that now do harm.

## robots.txt on Shopify and behind Cloudflare

**Shopify** generates robots.txt for every store with sensible defaults that block the cart, checkout, account pages and sorted or filtered collection URLs. You can change it through a `robots.txt.liquid` template in the theme code editor; Shopify's own help centre recommends keeping the default rules and adding to them rather than replacing the file, and calls the template an unsupported customization. The [Shopify robots.txt guide](https://getreport.app/guides/shopify-robots-txt) shows the default template, how to add or remove a rule inside its Liquid loop, how to add groups for AI crawlers, and how to reset the file when something breaks.

**Cloudflare** offers a managed robots.txt for AI crawlers on all plans, under Security → Settings → Bot traffic. When it is on, Cloudflare prepends its own rules to your file (or serves them as the whole file if you have none), including `Disallow` groups for AI training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot, and a `Content-Signal` line stating your preferences for search and AI use. The file you see at `/robots.txt` is then longer than the one on your server, which surprises people who edit only the origin copy. Your own rules keep working, because crawlers merge the two `User-agent: *` groups and your longer `Disallow` lines still win. The [Cloudflare robots.txt guide](https://getreport.app/guides/cloudflare-robots-txt) shows the exact block it adds, what the Content-Signal line means, how to switch it off, and how it differs from Cloudflare's AI bot blocking, which is enforced rather than requested.

## "Blocked by robots.txt" in Search Console

Search Console's Page indexing report shows two robots.txt statuses:

- **Blocked by robots.txt:** Google found the URL but did not crawl it because a rule disallows it. If the page should be in Google, find the rule and remove or narrow it.
- **Indexed, though blocked by robots.txt:** Google indexed the URL anyway, from links, without reading the page. If it should rank, unblock it; if it should disappear, unblock it and add `noindex`, then block it again later if you want.

Neither is an error in itself: carts, internal search and filter URLs belong in the first list. They need work when a page you want in Google is caught by a rule that is too broad, or when a page you want out of Google is indexed from links. The second case has a trap: a page that is both blocked and `noindex` stays indexed, because Google never fetches it and never sees the tag.

The guide to [fixing "Blocked by robots.txt" and "Indexed, though blocked by robots.txt"](https://getreport.app/guides/blocked-by-robots-txt) shows how to find the rule behind each URL and the order of steps for each case, including when leaving blocked parameter URLs alone is the right call.

## AI crawlers and robots.txt

AI companies run their own crawlers and publish their user-agent tokens: some collect training data (such as GPTBot, ClaudeBot, CCBot, and the Google-Extended token that controls Gemini's use of your content), others fetch pages live to answer a question or cite a source (such as OAI-SearchBot and PerplexityBot). The declared ones follow robots.txt, so it is where you decide who may read your site.

Blocking a training crawler keeps your pages out of future model training; blocking an answer crawler keeps you out of the AI answers that could link to you. `Google-Extended` does not affect Googlebot or your Google Search rankings. The trade-offs, the full list of tokens and a file to copy are in [AI crawlers and robots.txt](https://getreport.app/guides/ai-crawlers-and-robots-txt). If you also want to give AI tools a curated map of your site, that is a different file: [llms.txt, what it is and how to write one](https://getreport.app/guides/llms-txt-what-it-is-and-how-to-write-one).

## How to test robots.txt

> **Free tool:** [robots.txt tester: check and validate robots.txt](https://getreport.app/tools/robots-txt-tester): Free robots.txt tester: see whether robots.txt blocks a page for Googlebot, whether your sitemap is declared and reachable, and if the page can be indexed.

The robots.txt tester fetches the live file from the host of the URL you enter, parses it with an RFC 9309 parser (group selection, longest match, `Allow` on ties, `*` and `$`), and tells you whether that URL is allowed for Googlebot and for all other agents. When a rule blocks the page, the finding names the line. It also fetches every sitemap the file declares and reads its URL count, and it checks the page's robots meta tag and `X-Robots-Tag` header, so the final verdict says whether the page can be indexed, not only crawled.

The verdict is for one URL. A rule that blocks `/account/` shows as allowed on the home page, so test one URL from each section a rule could touch.

> **Check: robots.txt allows this page.** A Disallow rule stops search engines from crawling this page, so they cannot read its content or see any updates. The page can still appear in results as a bare URL with no description.
>
> 1. Remove or narrow the Disallow rule in robots.txt so this URL is allowed for all crawlers.
> 2. If the page should stay out of search, allow crawling and use a noindex tag instead; a blocked page cannot be de-indexed.

> **Check: robots.txt is present.** robots.txt tells crawlers which parts of the site to skip and where the sitemap is. Without one, crawlers use their own defaults and may waste time on search results or admin pages.
>
> 1. Publish a plain-text robots.txt at the site root with at least "User-agent: *", "Allow: /" and a "Sitemap:" line.
> 2. Most CMSs generate one; in WordPress it is under the SEO plugin's Tools section.

> **Check: robots.txt declares a sitemap.** A "Sitemap:" line in robots.txt is how crawlers find your XML sitemap without you registering it anywhere. Without it, new pages are discovered later.
>
> 1. Add a line to robots.txt: Sitemap: https://your-site.com/sitemap.xml (use the full URL).

> **Check: The page can be indexed.** To appear in search, a page must return HTTP 200, carry no noindex directive and be allowed in robots.txt. If any of the three fails, the page is invisible to searchers.
>
> 1. Open the technical detail to see which condition fails, then fix that one; the individual findings above have the steps.
> 2. After the fix, request indexing in Google Search Console to speed things up.

In Google Search Console, **Settings → robots.txt** shows the robots.txt files Google found for your property, when it last fetched each one and any parsing warnings, and lets you ask Google to fetch a changed file sooner. It replaced the old robots.txt Tester, which Google retired in December 2023, and it does not test individual URLs. The URL Inspection tool tells you whether one URL is blocked for Googlebot.

To see which crawlers actually request your pages, and whether they obey the file, read your server's access log; the [server log guide](https://getreport.app/guides/server-log-analysis-googlebot-and-ai-bots) shows how, including how to spot a fake Googlebot.

## Common mistakes

- **`Disallow: /` on a live site.** The most common cause of a site vanishing from Google, and the easiest to prevent with a test after each launch.
- **Blocking a page to remove it from Google.** It stays indexed as a URL without a description. Use `noindex` and let Google crawl.
- **A `Googlebot` group with fewer rules than `*`.** Googlebot then follows only its own group and ignores the rest.
- **Blocking `/wp-content/`, `/assets/` or `.js` and `.css` files.** Google renders pages and needs these to see your layout.
- **Relative sitemap URLs.** `Sitemap: /sitemap.xml` is not valid; use the full URL.
- **Listing secret paths.** The file is public; anything in it is an invitation.
- **A robots.txt that returns 5xx or a login page.** A server error pauses crawling; an HTML page at `/robots.txt` is parsed as rules and usually matches nothing.
- **Editing the origin file while a CDN or plugin serves a different one.** Check what `/robots.txt` really returns in the browser after every change.

## Questions people ask

### What is a robots.txt file used for?

A robots.txt file is used to tell crawlers which parts of a site not to fetch and where the sitemap is. Site owners use it to keep search engines out of admin pages, carts, internal search and endless filter URLs, and to state which AI crawlers may read the content. It does not hide pages from search results or protect private content.

### Where can I find a website's robots.txt?

Add `/robots.txt` to the site's domain, for example `https://example.com/robots.txt`, and open it in a browser. The file is always at the root of the host and always public. Each subdomain has its own file, so `shop.example.com/robots.txt` can differ from the main one. A 404 means the site has no file and allows all crawling.

### Do I need a robots.txt file?

No, a site works without one: a missing robots.txt means crawlers may fetch everything, which suits many small sites. It is still worth having a short file with a `Sitemap:` line, so every crawler finds your sitemap, and a few `Disallow` rules if you have carts, internal search or filter URLs that should not be crawled.

### Does robots.txt affect SEO?

Yes. A good file keeps crawlers on the pages that matter and points them to your sitemap, which helps most on large sites and shops. A bad one does real damage: a single `Disallow: /` or a rule broader than intended stops Google from reading your pages, and blocked CSS or JavaScript makes pages render badly. Test it after every launch.

### What happens if a site has no robots.txt?

Nothing bad. When `/robots.txt` answers 404 or 410, Google and other crawlers assume there are no restrictions and crawl the whole site. The risk is a file that answers with a server error instead: Google then pauses crawling for up to 12 hours and relies on its cached copy. A missing file should return a clean 404, not a 500.

### How big can a robots.txt file be?

Google reads the first 500 KiB of a robots.txt file and ignores any rules after that point. Almost every site stays far below this; a file that approaches the limit usually lists individual URLs that one wildcard rule could cover. Keep the file short and readable, because every line is one someone must understand when it goes wrong.

### Do all bots obey robots.txt?

No. Search engine crawlers such as Googlebot and Bingbot and the declared AI crawlers follow it, but the file is a voluntary standard and RFC 9309 says it is not access authorization. Scrapers, spam bots and attackers can ignore it. To stop unwanted bots, block them at your firewall, CDN or server, and use a login for anything private.