robots.txt tester:Is your robots.txt keeping Google out?
See whether your robots.txt blocks the page for Googlebot, whether your sitemap is declared and reachable, and whether the page can be indexed at all.
- Free, no account
- Results in under a minute
- Any public site
example-shop.hr
WordPress 6.8Sample shop · homepage · mobile Chrome · the same check you get for your own site
6 passed · 0 failed · 0 warnings. Nothing to fix here.
What we found
Everything this tool checks passed.
Show 6 passed checks
What this tool checks
The robots.txt rules, the sitemaps it declares and the noindex signals, ending in one verdict on whether the page can be indexed.
robots.txt file
Checks for the file that tells crawlers which parts of the site to skip and where your sitemap is.
robots.txt rules for this page
Checks robots.txt lets search engines crawl this page, so they can read its content and updates.
Sitemap in robots.txt
Checks robots.txt points to your XML sitemap, so crawlers find new pages sooner.
XML sitemap
Checks for a working sitemap, the list of pages you want indexed, so new and deep pages are found faster.
Noindex directive
Checks whether the page tells Google to leave it out of search results, which loses all search visits.
Search indexability
Checks the page returns 200, has no noindex and is allowed in robots.txt, so it can appear in search.
Show 2 more checks
Noindex directive
Checks whether the page tells Google to leave it out of search results, which loses all search visits.
Search indexability
Checks the page returns 200, has no noindex and is allowed in robots.txt, so it can appear in search.
How it works
Parse robots.txt
We fetch the robots.txt file and parse it with the same rules Google uses, then test the page URL for Googlebot and for all agents.
Fetch the sitemaps
Every sitemap the file declares is fetched and its URL count is read.
Check noindex
Robots meta tags and X-Robots-Tag headers are checked for noindex, and the final verdict says whether the page can be indexed.
robots.txt controls crawling, not indexing, so a page it blocks can still appear in Google from links, just without its content.
Questions
Is this really free?
Yes. getReport is funded by donations, not plans. This tool runs the full report and shows you the part it is about; the complete report with all seven modules is one click away, also free.
My site has no robots.txt. Is that bad?
No. Without the file everything is allowed. Add one if you want to keep crawlers out of search pages or admin paths, and to declare the sitemap.
Does robots.txt hide a page from search results?
Not reliably. A blocked page can still be indexed from links, just without content. Use a noindex meta tag for pages that must stay out, and let Google crawl it so it sees the tag.
Do you store my results?
The report is kept for 12 months at its shareable link so you can come back to it. Reports are not listed publicly and carry a noindex tag; nothing about your visitors is collected.
Why does robots.txt matter?
A single Disallow line can remove a whole site from Google, and it happens; staging rules copied to production are a classic. The file is public, so checking it takes seconds. Declaring the sitemap in robots.txt lets every crawler find it without a manual submission.
Related free tools
All 41 tools →Free, funded by the people who use it
€0 of €75 this month. At €75, site crawl up to 500 pages + weekly re-check switches on for everyone.