Our crawler: GetReportBot

User agent: GetReportBot/1.0 (+https://getreport.app/bot)

Requests come from our servers in the Frankfurt region, Germany (OVHcloud, Limburg). When the page is rendered in Chromium, requests come from the same addresses with a browser user agent that ends in the token above. Google’s own PageSpeed crawler fetches the page separately for the speed measurement.

One exception: to detect hacked sites that show spam only to Google, the page is also fetched once with Googlebot’s user agent and once as a visitor arriving from Google, from the same addresses. Those two requests carry no other disguise and happen only as part of a report a person asked for.

• The page a person asked to check, once, following up to 10 redirects.
• The standard files around it: robots.txt, XML sitemaps (with a small sample of their URLs), favicon, security.txt, llms.txt and a random missing path to see your 404 page.
• HEAD requests to up to 100 links on the page and 50 external links to find broken ones, up to 50 hreflang alternates, and up to 100 images to read their size.
• A fixed list of at most 15 well-known paths, such as /.env and /.git/HEAD for exposed secrets and WordPress defaults such as /?author=1. It never guesses paths, never logs in and never submits forms.
• In Chromium: the page’s own scripts, styles, fonts and images, capped at 15 MB and 60 seconds. Downloads, WebSockets and private addresses are blocked.

• At most 6 requests to a site at the same time during one report.
• A site can be checked at most 5 times an hour; the same page is served from a 12-hour cache instead of being fetched again.
• When a site answers 429 or 503, it gets no further requests from us for 10 minutes.
• It never fetches anything from private networks or localhost.

Add this to your robots.txt and the bot will refuse to check the site. Reports are requested by a person for one page, like Google’s PageSpeed Insights, so the general User-agent: * group does not apply; name GetReportBot to opt out.

User-agent: GetReportBot
Disallow: /

If a report of your site should be removed, or nobody should check it here at all, use the removal form or write to [email protected] from an address at that domain.