Robots.txt tester: see if your robots.txt keeps Google out
This robots.txt tester shows whether your robots.txt blocks a page for Googlebot, whether your sitemap is declared and reachable, and whether the page can be indexed. A free alternative to Google's retired tester.
- Free, no account
- Results in under a minute
- Any public site
Example result:example-shop.hr
WordPress 6.8Sample shop · homepage · mobile Chrome · the same check you get for your own site
What we found
Everything this tool checks passed.
Show 6 passed checks
What the robots.txt tester checks
The robots.txt rules, the sitemaps it declares and the noindex signals, ending in one verdict on whether the page can be indexed.
robots.txt file
Checks for the file that tells crawlers which parts of the site to skip and where your sitemap is.
robots.txt rules for this page
Checks robots.txt lets search engines crawl this page, so they can read its content and updates.
Sitemap in robots.txt
Checks robots.txt points to your XML sitemap, so crawlers find new pages sooner.
XML sitemap
Checks for a working sitemap, the list of pages you want indexed, so new and deep pages are found faster.
Noindex directive
Checks whether the page tells Google to leave it out of search results, which loses all search visits.
Search indexability
Checks the page returns 200, has no noindex and is allowed in robots.txt, so it can appear in search.
Show 2 more checks
Noindex directive
Checks whether the page tells Google to leave it out of search results, which loses all search visits.
Search indexability
Checks the page returns 200, has no noindex and is allowed in robots.txt, so it can appear in search.
How the robots.txt tester works
Parse robots.txt
We fetch the robots.txt file and parse it with the same rules Google uses, then test the page URL for Googlebot and for all agents.
Fetch the sitemaps
Every sitemap the file declares is fetched and its URL count is read.
Check noindex
Robots meta tags and X-Robots-Tag headers are checked for noindex, and the final verdict says whether the page can be indexed.
robots.txt controls crawling, not indexing, so a page it blocks can still appear in Google from links, just without its content.
Robots.txt tester questions
Does the tester check Bingbot and AI crawlers?
Not separately. It tests the page for Googlebot and for the * group, which applies to every crawler without a group of its own, so Bingbot follows those rules unless your file names it. For AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, run the AI crawler check, which reads the same robots.txt for 18 AI user agents and shows which of them may fetch your pages.
How do I test if a URL is blocked by robots.txt?
Enter the URL above. The tester fetches the site's robots.txt, parses it with the same rules Google uses and says whether that URL is allowed or blocked for Googlebot and for all other agents. It also checks noindex tags and headers, so the verdict tells you whether the page can be indexed, not only crawled.
Does the tester check the sitemap in robots.txt?
Yes. Every Sitemap line in the file is fetched, and the tester reports whether each sitemap is reachable and how many URLs it lists. A Sitemap line is optional but helps crawlers other than Google find your sitemap. Use the full URL, such as Sitemap: https://example.com/sitemap.xml; a relative path is not valid there.
Why is Googlebot still crawling a page I disallowed?
Usually because Google has not fetched your new robots.txt yet; it caches the file for up to 24 hours. Other causes: the rule does not match the path exactly, since paths are case sensitive, a more specific Allow rule wins, or the requests come from a fake Googlebot. Test the URL above to see which rule applies, and verify the addresses in your logs.
Is there a replacement for Google's robots.txt tester?
Yes. Google retired the robots.txt Tester in Search Console in late 2023 and replaced it with the robots.txt report, which shows the files Google fetched but does not test a URL against them. This free tester fills that gap: it fetches your live file, tests the page for Googlebot and for all agents, and checks the sitemap and noindex signals too.
Does robots.txt hide a page from search results?
Not reliably. robots.txt stops crawling, not indexing, so a blocked page can still appear in results from links, just without a description. For pages that must stay out, use a noindex meta tag or X-Robots-Tag header and let Google crawl the page so it sees the tag. This tester checks both, and its verdict says whether the page can be indexed.
Is this robots.txt tester free?
Yes, with no sign-up. getReport is funded by donations, not plans, and the complete report with all seven modules is one click away, also free. It stays at its shareable link for 12 months, is not listed publicly and carries a noindex tag, and nothing about your visitors is collected.
Related free tools
All 59 tools →Free, funded by the people who use it
No account, no paid plan. Donations keep it running.