A robots.txt Disallow rule tells compliant crawlers not to fetch any URL whose path starts with the value you give it: Disallow: /cart/ keeps them out of the cart, and Disallow: / keeps them out of the whole site. It stops crawling, not indexing, and it is the line most likely to wipe a site from Google when a staging file reaches production. This guide shows how to write the Disallow rule you actually need, how to allow everything, and how to find and undo an accidental block. For what the file is and how crawlers read it end to end, start with the complete robots.txt guide.
Quick answer
Disallow: /path/blocks every URL that starts with/path/for the crawlers named in the group above it. Paths are case-sensitive.- Disallow all:
User-agent: *thenDisallow: /. Right for staging, a disaster on a live site. - Allow all:
User-agent: *then an emptyDisallow:, or no robots.txt at all (a 404 means "no restrictions"). Disallowdoes not remove pages from Google. A blocked URL can still be indexed from links, shown without a description. Usenoindexon a crawlable page instead.- Never disallow CSS, JavaScript or image folders; Google needs them to render the page.
- Test the exact page with the robots.txt tester: it names the rule and line that blocks it.
How a Disallow rule works
A rule belongs to the group of User-agent lines directly above it. A crawler picks the one group that names it most specifically (Googlebot beats *), and reads only that group. Inside the group, each Disallow value is compared with the start of the URL path, including the query string:
User-agent: *
Disallow: /checkout/This blocks /checkout/, /checkout/step-2 and /checkout/?coupon=x. It does not block /Checkout/ (case differs) or /shop/checkout/ (does not start with it).
Three details decide whether a rule does what you think:
- Prefixes, not folders.
Disallow: /pblocks/pricing/,/products/and/press/. Always end a folder rule with/. - Longest match wins. When an
Allowand aDisallowboth match, the one with the longer path applies, andAllowwins a tie. Order in the file does not matter. - Wildcards.
*matches any run of characters and$anchors the end of the URL. Google, Bing and RFC 9309 support both. The full syntax, with copy-paste patterns, is in robots.txt examples and syntax.
Disallow rules for common jobs
Each block below is a complete group; put your shared rules in the * group and add a Sitemap: line at the end of the file.
Block one folder:
User-agent: *
Disallow: /private/Block one page but not its children:
User-agent: *
Disallow: /thank-you$
Disallow: /thank-you/$Without $, /thank-you would also block /thank-you-for-applying/ and anything below it.
Block a file type anywhere on the site:
User-agent: *
Disallow: /*.pdf$Block URLs with a parameter (internal search, sort orders):
User-agent: *
Disallow: /*?s=
Disallow: /*&s=
Disallow: /*?orderby=
Disallow: /*&orderby=A ? is literal in robots.txt, so the & line catches the parameter when it is not first.
Block a folder but keep one file in it:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpThe Allow path is longer, so it wins for that one file.
Block one crawler only:
User-agent: GPTBot
Disallow: /Every other crawler is unaffected, because it does not match this group. Deciding which AI crawlers to block is covered in AI crawlers and robots.txt.
robots.txt disallow all, and allow all
Block everything
User-agent: *
Disallow: /Every compliant crawler stops fetching every URL on that host. This is what staging and development sites use. On a live site it removes pages from Google's results over the following days and weeks: some drop out, others linger as bare URLs with "No information is available for this page", because Google can index a URL it may not fetch.
A robots.txt block is also not privacy. The file is public, and bots that ignore the standard fetch the pages anyway. For a site that must stay out of search, use HTTP authentication, and treat Disallow: / as a second layer.
Block everything except Google
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /Googlebot reads only its own group, where an empty Disallow: allows everything; every other crawler falls into the * group and is blocked. Google's other crawlers (Googlebot-Image, AdsBot-Google) have their own tokens; Googlebot-Image falls back to the Googlebot group when it has none of its own.
Allow everything
Any of these means "no restrictions":
User-agent: *
Disallow:User-agent: *
Allow: /Or no file at all. Google treats a 404 (and any other 4xx except 429) on /robots.txt as "everything allowed". An empty Disallow: is the most widely understood form; Allow is part of RFC 9309 but some older crawlers never supported it.
The accidental site-wide block
This is how most sites lose their search traffic to robots.txt. A developer sets Disallow: / on staging, the file is committed or copied, and the next deployment carries it to production. The site works for every visitor, so nobody notices until rankings and traffic fall a week or two later.
Variants that do the same damage:
- A
User-agent: Googlebotgroup withDisallow: /left from a test. Disallow: /*orDisallow: /*?meant for one parameter, which blocks every URL or every URL with a query string.- A server that answers
/robots.txtwith a5xxerror or429. Google treats that as "the whole site is disallowed". According to Search Console Help, it stops crawling for the first 12 hours while it retries, then uses the last good copy of the file for up to 30 days; after that, it acts as if there is no file if the site is otherwise reachable. - A CDN or firewall rule that serves bots a challenge page instead of the file.
WordPress sites have one more cause that does not touch robots.txt at all: the "Discourage search engines" box, which since WordPress 5.3 adds noindex instead. See the WordPress box that hides your site.
How to prevent it
- Generate robots.txt per environment. Next.js
app/robots.ts, a template filled from an environment variable, or a web server rule that checks the hostname. The staging version then cannot be deployed to production, because it is never a file in the repository. - Protect staging with a password, so the robots.txt there is a backup, not the only guard.
- Test after every deploy. One request to the tester on your home page, or a check in your deploy pipeline that
curl -s https://example.com/robots.txtdoes not containDisallow: /on its own line.
How to undo it
- Remove or narrow the rule, deploy, and confirm with
curl -s https://example.com/robots.txtthat the live file changed (clear the CDN cache if it did not). - In Search Console → Settings → robots.txt, open the file's menu and choose Request a recrawl. Otherwise Google picks up the change within about 24 hours, the longest it normally caches the file.
- Request indexing for your most important URLs with URL Inspection.
- Watch Pages → "Blocked by robots.txt" shrink over the next weeks. Recovery usually takes days to a few weeks; rankings built on the old pages generally return once they are recrawled.
How getReport checks it
The tester fetches /robots.txt, parses it with an RFC 9309 parser (group selection, longest match, Allow on ties, * and $), and tests the URL you entered for Googlebot and for the generic * agent. When a rule blocks it, the finding names the agent and the line number of the rule. It also fetches the declared sitemaps and reads the robots meta tag and the X-Robots-Tag header, so the indexability verdict says which signal actually keeps the page out.
The verdict is for one URL. A rule that blocks /members/ shows as "allowed" on the home page, so test one page from each section a rule could touch.
Disallow or noindex?
They answer different questions. Disallow says "do not fetch this". noindex says "fetch it, but do not show it in results". Pick by what you want:
| You want | Use |
|---|---|
| A page out of Google's results | noindex meta tag or X-Robots-Tag header, and no Disallow |
| Crawlers to stop spending requests on endless URLs (filters, internal search) | Disallow |
| A PDF or image out of results | X-Robots-Tag: noindex on the file |
| Something private | Authentication; neither of the above |
Combining both on one page is the classic mistake: the Disallow stops Google from fetching the page, so it never sees the noindex, and the URL stays indexed. robots.txt noindex explains why Noindex: inside robots.txt stopped working in 2019, and Blocked by robots.txt covers the two Search Console statuses a Disallow produces.
Common mistakes
Disallow: /in production. The number one cause of "our site disappeared from Google".- A folder rule without the trailing slash.
Disallow: /adminalso blocks/administration-services/. - Blocking CSS and JavaScript.
Disallow: /wp-content/or/static/makes Google render a broken page. Allow them. - A specific group with fewer rules than
*. AUser-agent: Googlebotgroup with one line lifts every*rule for Googlebot. Repeat the shared rules. - Disallowing a page to remove it from search. It stays as a title-only result. Unblock it and use
noindex. - Listing secret paths.
Disallow: /backup-2026/tells everyone where to look. Protect it instead.
Questions people ask
How do I disallow everything in robots.txt?
Two lines do it: User-agent: * followed by Disallow: /. That asks every compliant crawler to fetch nothing on the host, which is what staging sites use. Be careful: on a live site it removes you from search within days, and already indexed URLs can linger as title-only results. To keep a site out of Google reliably, use password protection or noindex instead, and never let this file reach production.
How do I allow everything in robots.txt?
Write User-agent: * followed by an empty Disallow: line, or Allow: /. Both tell every crawler it may fetch any URL. Having no robots.txt at all has the same effect, because Google treats a 404 on the file as "no restrictions". Keep a Sitemap: line in the file if you have one, since that is the part that still helps crawlers.
How do I block a folder in robots.txt?
Add Disallow: /folder/ under User-agent: *, with the trailing slash. Rules match the start of the path, so without the slash Disallow: /folder also blocks /folder-archive/ and /folders.html. Paths are case-sensitive, so /Folder/ needs its own rule if both exist. Test a URL inside and one outside the folder to confirm the rule matches only what you meant.
Can I block all bots except Googlebot?
Yes. Give Googlebot its own group with an empty Disallow:, then add User-agent: * with Disallow: /. Googlebot reads only the group that names it, so it may fetch everything, while every other compliant crawler falls into the * group and is blocked. Bing, AI crawlers and SEO tools then stay out, but badly behaved bots ignore robots.txt anyway.
Should I block CSS and JavaScript files in robots.txt?
No. Google renders pages like a browser and needs your CSS and JavaScript to see the layout and content; blocking them can make pages look broken or not mobile-friendly to Google. Older WordPress setups often disallowed /wp-includes/ or /wp-content/, so remove those lines. Block only areas with no search value, such as internal search results or the cart.