Googlebot user agent strings: every Google crawler and what it does

The Googlebot user agent is the string Google's main crawler sends with every request, and it always contains Googlebot/2.1 and a link to http://www.google.com/bot.html. Googlebot Smartphone, which does most of the crawling, looks like Chrome on an Android phone; Googlebot Desktop looks like desktop Chrome. Around them sit a dozen other Google crawlers with their own strings, from Googlebot-Image to AdsBot and GoogleOther. This guide lists them, explains what each one does and which robots.txt rules it follows, and shows how to find each in your server log. For the bigger picture of how Google crawls, see the guide to Googlebot and crawling.

Quick answer

• Googlebot Smartphone: Chrome on Android, ending in (compatible; Googlebot/2.1; +http://www.google.com/bot.html). It makes most of Google's requests.
• Googlebot Desktop: desktop Chrome with the same Googlebot/2.1 token, or the short legacy string Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html).
• W.X.Y.Z is a placeholder for the current Chrome version, which changes as Google updates its renderer. Match on the token, never on the full string.
• One robots.txt token, Googlebot, covers both. You cannot give Smartphone and Desktop different rules.
• Other Google crawlers carry their own names: Googlebot-Image, Googlebot-Video, Storebot-Google, GoogleOther, Google-InspectionTool, AdsBot-Google and more.
• A user agent is a claim, not proof. Scrapers copy it; verify the IP address before you trust or block a visit.

The two Googlebot user agent strings

Google's documentation publishes the strings with W.X.Y.Z where the Chrome version goes. In a real log you will see a version number such as Chrome/129.0.6668.100 in its place.

Googlebot Smartphone

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Googlebot Desktop

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Google also lists two older desktop forms you may still see occasionally:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)

The Nexus 5X and Android 6.0.1 in the smartphone string are fixed; they do not mean Google renders your page on an old phone. Rendering happens in an up-to-date Chromium, and the Chrome version in the string tracks it.

Why Smartphone dominates

Google indexes the mobile version of pages (mobile-first indexing), so Googlebot Smartphone fetches most HTML pages and its view is the one that gets indexed. Googlebot Desktop still visits, but as a minority. If your log shows mostly Desktop hits, look again: the smartphone string also contains "Chrome" and "Safari", and a filter that matches "Mobile" or "Android" is the easiest way to split them.

Every Google crawler you will meet in a log

Google groups its crawlers into three kinds, and they behave differently towards robots.txt.

Common crawlers

These always obey robots.txt.

Google-Extended is also a robots.txt token Google documents, but it never appears in a log: it has no user agent. It tells Google whether content its crawlers already fetch may be used for Gemini training and grounding.

If you write no group for Googlebot-Image or Googlebot-News, they follow your Googlebot group. A Google-InspectionTool hit in your log is you, or someone with access to your Search Console, testing a URL.

Special-case crawlers

These do specific jobs for Google products, and some ignore the User-agent: * group, so a general Disallow does not stop them.

User-triggered fetchers

These fetch a page because a person asked for it, so they generally ignore robots.txt. Examples include FeedFetcher-Google (RSS and Atom feeds for Google News and others), Google-Site-Verification (checking your Search Console verification file or tag), Google-Read-Aloud, Google-NotebookLM and Google-Agent, which Google added in March 2026 for AI agents running on its infrastructure that browse and act on a user's behalf.

Google updates this list several times a year. The authoritative, current version is on Google's crawler documentation pages for common crawlers, special-case crawlers and user-triggered fetchers.

How to find Googlebot in your log

Your server's access log records the user agent in the last quoted field of each line in the Apache and nginx combined format. If you have not opened one before, reading Apache and nginx access logs explains where it is and what each field means.

# All requests claiming to be Googlebot (Search), excluding the Image/Video variants
grep 'Googlebot/2.1' access.log | wc -l

# Split smartphone and desktop
grep 'Googlebot/2.1' access.log | grep -c 'Android'      # smartphone
grep 'Googlebot/2.1' access.log | grep -vc 'Android'     # desktop

# Googlebot's most requested URLs
grep 'Googlebot/2.1' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# Every Google crawler by name
grep -oE 'Googlebot[-A-Za-z]*|Storebot-Google|GoogleOther[-A-Za-z]*|Google-[A-Za-z]+|AdsBot-Google[-A-Za-z]*|Mediapartners-Google|FeedFetcher-Google' access.log | sort | uniq -c | sort -rn

Or skip the shell:

The log file analyser groups every hit by crawler, including Googlebot, Googlebot-Image, Googlebot-Video, Google-InspectionTool, GoogleOther, AdsBot, Mediapartners, Storebot and FeedFetcher, with hits per day, status codes and top URLs. It runs in your browser, and an optional check verifies Googlebot's IP addresses by reverse DNS.

Writing robots.txt rules for Google's crawlers

Rules are grouped by token, and each crawler follows the most specific group that names it. A few patterns cover most needs:

# Keep images in one folder out of Google Images, allow everything else
User-agent: Googlebot-Image
Disallow: /private-images/

# Keep Google Shopping's crawler out of the cart
User-agent: Storebot-Google
Disallow: /cart/

# Opt out of Gemini training and grounding (no effect on Search)
User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /search/

Note that a crawler which finds a group naming it ignores the * group completely, so the Storebot-Google group above lets it crawl /search/. Repeat shared rules in each named group. Test the result with the robots.txt tester, which shows the matching rule for each URL and crawler. The report checks the basics too:

Bingbot user agents for comparison

Bing's crawler sends bingbot/2.0 in a desktop and a mobile variant, with a Chrome version that changes over time:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The older Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) still appears too. The robots.txt token is bingbot, and unlike Googlebot, Bingbot honours Crawl-delay. Bing's other agents include adidxbot for Bing Ads and BingPreview / MicrosoftPreview for page previews.

Common mistakes

• Matching the full string. The Chrome version changes every few weeks; a firewall rule or analytics filter on the exact string silently stops matching. Match Googlebot instead.
• Treating every Googlebot line as Google. Scrapers send the same string. Verify addresses before drawing conclusions from volumes.
• Blocking AdsBot with User-agent: *. It ignores that group. If you run Google Ads, you usually want AdsBot to crawl anyway.
• Expecting a Google-Extended line in the log. It is only a token; the fetching is done by Googlebot.
• Serving different content by user agent. Showing Googlebot something visitors do not see is cloaking and breaks Google's spam policies.

Questions people ask

What does Googlebot Smartphone's user agent look like?

It looks like Chrome on an Android phone: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). W.X.Y.Z is replaced by the current Chrome version. The phone model is fixed and does not describe how your page is rendered.

Why does the Chrome version in Googlebot's user agent keep changing?

Because Googlebot renders pages with an evergreen Chromium, and the user agent reports the version it currently uses. Google updates it regularly, so a log from last month and one from today show different numbers. Any filter, firewall rule or bot allow-list should match the Googlebot token rather than the full string, or it will break at the next update.

Can I set different robots.txt rules for Googlebot Smartphone and Desktop?

No. Both crawlers use the same robots.txt token, Googlebot, so one group governs both. Google chose this deliberately, because with mobile-first indexing the smartphone crawler's view is the one indexed. If you need to keep content from desktop users or crawlers, that is a page design question, not a robots.txt one.

What is Google-InspectionTool in my logs?

It is the crawler behind Search Console's URL Inspection tool and the Rich Results Test. A hit from it means someone, usually you or a colleague with Search Console access, tested that URL. It follows your Googlebot rules in robots.txt. Like any user agent it can be faked, so verify the address if the volume looks odd.

Does Googlebot-Image follow my Googlebot rules?

Yes, when you have not written a group for it. Googlebot-Image uses the most specific group in robots.txt that names it; if there is no User-agent: Googlebot-Image group, it falls back to the Googlebot group, and then to *. Write a dedicated group only if you want images crawled differently from pages.