A good robots.txt example for most sites is short: one User-agent: * group that disallows the cart, account and internal search, allows everything else including CSS and JavaScript, and ends with a Sitemap: line. This guide gives complete files you can copy for a blog, a WordPress site, a shop, a Next.js app and an AI crawler policy, then explains the syntax behind them (groups, longest match, * and $) as Google implements it under RFC 9309. For what robots.txt is and why it matters, see the robots.txt guide; to test a URL against your finished file, use the tester.
Quick answer
- A robots.txt file is groups: one or more
User-agent:lines, thenAllow:andDisallow:rules. A crawler reads only the one group that names it most specifically, or*if none does. - Rules match the start of the URL path. The longest matching rule wins;
Allowwins a tie. *matches any characters,$anchors the end. Paths are case-sensitive; field names are not.Sitemap:takes an absolute URL, can repeat, and applies to every group.- Google ignores
Crawl-delay. Bing honours it. - One file per scheme and host at
/robots.txt, UTF-8 plain text; Google reads the first 500 KiB. - Check the result with the robots.txt tester, which names the line that matches.
A robots.txt example for a typical site
# https://example.com/robots.txt
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search/
Disallow: /*?s=
Disallow: /*&s=
Sitemap: https://example.com/sitemap.xmlWhat each part does:
User-agent: *opens a group that applies to every crawler without a group of its own.- The four folder rules keep crawlers out of per-visitor pages with no search value. Each ends in
/, so/cart/does not also block/cartography/. /*?s=and/*&s=block internal search results wherever thesparameter appears.- Everything not matched, including
/assets/, images, CSS and JavaScript, is allowed. Sitemap:points every crawler at the sitemap.
What should be in robots.txt is mostly what is not in it: no CSS or JavaScript folders, no pages you want removed from search (that is noindex), and no secret paths.
robots.txt format and syntax
The file is plain text, UTF-8, one directive per line in the form field: value. Google recognises four fields and ignores the rest:
| Field | Example | What it does |
|---|---|---|
User-agent | User-agent: Googlebot | Starts a group for the named crawler. * means every crawler without its own group. |
Disallow | Disallow: /private/ | Paths the group may not fetch. Empty value = nothing blocked. |
Allow | Allow: /private/press/ | Paths the group may fetch, used to make exceptions to a Disallow. |
Sitemap | Sitemap: https://example.com/sitemap.xml | Absolute URL of a sitemap or sitemap index. Independent of groups. |
Crawl-delay | Crawl-delay: 10 | Seconds between requests. Ignored by Google; honoured by Bing. |
Comments start with #. Unknown lines are skipped silently, which is why a typo such as Dissallow: allows everything it was meant to block. The file lives at the root of each host: https://example.com/robots.txt does not cover https://www.example.com/ or http://example.com/. Google follows at least five redirects when fetching it and caches the result for up to 24 hours.
User-agent groups
User-agent: *
Disallow: /search/
User-agent: Googlebot
Disallow: /search/
Disallow: /drafts/Googlebot reads only the second group. If that group held only Disallow: /drafts/, Googlebot would be allowed into /search/, because the * group is not consulted once a specific group matches. Groups do not add up: repeat the shared rules in every named group.
Several User-agent lines directly above one set of rules form one group. The value is the crawler's product token (Googlebot, Bingbot, GPTBot), matched case-insensitively, not its full user-agent string.
Longest match wins, Allow wins ties
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpFor /wp-admin/admin-ajax.php, both rules match. The Allow path is longer (24 characters against 10), so it wins; everything else under /wp-admin/ stays blocked. Order in the file does not matter.
Wildcards and the end anchor
* matches any sequence of characters, including none and including /. $ matches only at the end of the URL. Without $, every rule is a prefix:
User-agent: *
# Sorted and filtered listings, keep the base category pages
Disallow: /*?orderby=
Disallow: /*&orderby=
# PDFs anywhere, but not /pdf-guide/ pages
Disallow: /*.pdf$
# One folder closed, one file in it open
Disallow: /private/
Allow: /private/press-kit.zip? is literal, so /*?orderby= matches only when orderby is the first parameter; the & line catches it anywhere else. There is no other pattern syntax: robots.txt has no regular expressions, character classes or +.
Empty rules and case
An empty Disallow: allows everything for that group. Disallow: /Admin/ does not block /admin/, because paths are compared case-sensitively, while user-agent and USER-AGENT are the same field. The Disallow guide has the block-all, allow-all and one-bot-only files.
robots.txt examples by site type
WordPress
WordPress serves this file when none exists on disk:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlA common extension blocks internal search: add Disallow: /?s= and Disallow: /search/. Do not add /wp-content/ or /wp-includes/; they hold the CSS and JavaScript Google needs. SEO plugins replace the sitemap line with their own index. Where the file lives and how to edit it is in WordPress robots.txt.
Online shop
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?add-to-cart=
Disallow: /*?orderby=
Disallow: /*&orderby=
Disallow: /*?filter_
Disallow: /*&filter_
Sitemap: https://example.com/sitemap_index.xmlThe parameter rules stop crawlers spending requests on endless sort and filter combinations. Only block filters whose pages you do not want found; a filtered page that people search for ("red running shoes") should stay crawlable and indexable. On Shopify, rules are added through the robots.txt.liquid template; see Shopify robots.txt.
Next.js and other JavaScript frameworks
Generate the file so preview deployments are blocked and production is not:
// app/robots.ts
import type { MetadataRoute } from 'next';
export default function robots(): MetadataRoute.Robots {
if (process.env.VERCEL_ENV !== 'production') {
return { rules: { userAgent: '*', disallow: '/' } };
}
return {
rules: { userAgent: '*', disallow: ['/api/private/', '/account/'] },
sitemap: 'https://example.com/sitemap.xml',
};
}Do not disallow /_next/ or /api/ routes the page needs to render; Google fetches them while rendering.
Staging
User-agent: *
Disallow: /Pair it with password protection and keep it out of the repository, so it cannot be deployed to production.
AI crawlers
Named groups do not inherit *, so each group states its own rules:
# Model-training crawlers: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /
# AI search and user-requested fetches: yes, same limits as everyone
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/Google-Extended and Applebot-Extended are control tokens, not crawlers: blocking them keeps content out of Gemini and Apple model training without touching Googlebot or Applebot. Which bots to allow is a business decision covered in AI crawlers and robots.txt.
The Sitemap line
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/news-sitemap.xmlThe value must be an absolute URL; Sitemap: /sitemap.xml is invalid. The line can go anywhere in the file, applies to every crawler regardless of groups, and can repeat. It may point to a sitemap on another host, which is how a CDN-hosted sitemap is declared. It is the one line that helps crawlers other than Google find your sitemap without a Search Console submission.
Crawl-delay
Crawl-delay: 10 asks a crawler to wait 10 seconds between requests. Bingbot honours it. Googlebot never has, and it is not part of RFC 9309. Google also retired Search Console's crawl-rate setting in January 2024. If Googlebot overloads your server, return 503 or 429 for a short period and it slows down on its own; for a lasting problem, fix capacity or use the crawl-rate report form Google links from its documentation.
Be careful with large values for Bing: Crawl-delay: 60 limits Bingbot to 1,440 requests a day, which is too few for a shop with tens of thousands of URLs.
How getReport checks it
The tester fetches /robots.txt (up to 512 KB), parses it with an RFC 9309 parser (wildcards, $, Allow precedence, group selection) and asks, for the exact URL you entered, whether Googlebot and the generic * agent may fetch it. When either is blocked, the finding names the agent and the 1-based line number of the rule that matched.
The same parser drives the AI crawler check, which asks the question for 18 AI user-agent tokens and shows whether each was decided by its own group or inherited the * group:
robots.txt best practices
- Keep the
*group short and block only what has no search value: carts, accounts, internal search, endless parameters. - End folder rules with
/, and use$for exact files. - Repeat shared rules in every named group.
- Never block CSS, JavaScript, images or the API endpoints pages render from.
- Use
noindex, not Disallow, to keep a page out of results. - Declare the sitemap with an absolute URL.
- Generate the file per environment and test a page from each section after every change.
- Check Search Console → Settings → robots.txt for the file Google last fetched on each host.
Common mistakes
- A named group with fewer rules than
*.User-agent: Googlebotwith one line lifts every*rule for Googlebot. - Prefix rules that are too short.
Disallow: /pblocks/pricing/,/products/and/press/. Disallow: /*?to block one parameter. It blocks every URL with a query string, including/?p=123permalinks.- A relative Sitemap URL.
Sitemap: /sitemap.xmlis ignored. Crawl-delayas the fix for Googlebot. It is ignored; fix the server or return 503 briefly.- Editing the wrong host's file. Rules on
example.comwhile Google crawlswww.example.com. Redirect the variants and keep one file.
Questions people ask
What should a robots.txt file look like?
A plain text file at /robots.txt with a User-agent: * line, a few Disallow: lines for areas with no search value such as the cart, account and internal search, and a Sitemap: line with the sitemap's full URL. Most sites need fewer than ten lines. Anything not disallowed is allowed, so there is no need to list the pages you want crawled.
How do I block a file type such as PDF in robots.txt?
Combine the wildcard with the end anchor: Disallow: /*.pdf$ under User-agent: * blocks every URL that ends in .pdf, in any folder. The $ means /report.pdf?v=2 stays crawlable; leave it out to block those too. Blocking only stops crawling, so PDFs already in Google stay listed without a snippet. To remove them, allow crawling and send an X-Robots-Tag: noindex header with the files instead.
How do I create a robots.txt file?
Create a plain text file named robots.txt in UTF-8, add User-agent: *, your Disallow lines (for example /wp-admin/ or internal search), and a Sitemap: line with your sitemap's full URL, then upload it to the root of the host so it loads at /robots.txt. On WordPress, edit it in your SEO plugin's tools; on Shopify, in the robots.txt.liquid template. Test the result before you rely on it.
How do I read a robots.txt file?
Find the group whose User-agent matches the crawler; a crawler uses only its most specific group, and * only when no named group matches. Within the group, the longest matching rule wins, and Allow wins a tie with Disallow. * matches any characters and $ marks the end of the URL. Rules match URL paths by prefix, so Disallow: /p also blocks /pricing/. Anything not matched is allowed.
How does robots.txt work?
Before crawling a site, a compliant crawler requests /robots.txt from that exact host and scheme, caches it (Google for up to 24 hours) and checks every URL against the rules for its user agent before fetching it. A missing file (404) means everything is allowed. The file controls crawling only: a blocked page can still be indexed from links, and bots that ignore the standard are not stopped at all.
Is robots.txt case sensitive?
Partly. The paths in Allow and Disallow rules are case sensitive, so Disallow: /Private/ does not block /private/. Directive names and user-agent names are not: user-agent: googlebot matches Googlebot. The file itself must be named robots.txt in lowercase and sit at the root of the host, and each subdomain and protocol needs its own file.
Does Google support crawl-delay in robots.txt?
No. Googlebot ignores Crawl-delay, and Search Console's crawl-rate setting was retired in January 2024. Google adjusts its crawl rate to how fast and reliably your server answers. To slow it down briefly, return 503 or 429 for a day or two. Bing does honour Crawl-delay, so a value there still limits Bingbot.
Can I use regular expressions in robots.txt?
No. robots.txt supports only two special characters: *, which matches any run of characters, and $, which anchors the end of the URL. Everything else, including ?, . and +, is matched literally. To cover several variants, write several rules, for example one for ?sort= and one for &sort=.