Googlebot: what it is and how to block or allow it

What Googlebot does

Googlebot fetches pages to build Google's Search index, in a smartphone and a desktop version that share one robots.txt name. Its crawl feeds Google Search, Discover, Google Images, Video and Google News. Google says AI is built into Search, so the same crawl decides whether a page can appear as a link in AI Overviews and AI Mode.

• Googlebot-Image, Googlebot-Video and Googlebot-News are separate crawlers that also follow rules written for Googlebot. Source
• nosnippet, data-nosnippet, max-snippet and noindex limit what Google shows from a page, in AI Overviews and AI Mode as in classic results. Source
• Googlebot renders pages in an up-to-date Chromium, so it runs JavaScript, but not from files that robots.txt blocks. Source
• Blocking Google-Extended doesn't affect Google Search. It only limits Gemini training and grounding. Source

Does Googlebot follow robots.txt?

Yes. Google classes Googlebot as a common crawler, and says common crawlers always obey robots.txt rules when they crawl automatically.

It doesn't honor Crawl-delay.

How to block or allow Googlebot

Block it from the whole site:

User-agent: Googlebot
Disallow: /

Keep it out of one folder only:

User-agent: Googlebot
Disallow: /private/

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the Googlebot group. robots.txt works per host, so add the rules to each subdomain you want covered.

Google doesn't support Crawl-delay. To slow Google down for a short while, answer with 500, 503 or 429 status codes; errors that last several days can drop pages from the index.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking Googlebot changes

Google stops crawling your pages for Search, Discover, Images, Video and News, and they drop out of AI Overviews and AI Mode too, because those link only to indexed pages that can show a snippet. The bare URL can still appear in results, because robots.txt stops crawling, not indexing; use noindex to keep a page out of Search.

Where to see Googlebot visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes Googlebot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.

Other bots from Google

• Google-Extended: Not a crawler: the robots.txt setting for Gemini training and grounding. It doesn't affect Google Search.
• Google-InspectionTool: Google-InspectionTool fetches your page when someone runs one of Google's Search testing tools, such as the Rich Results Test or URL Inspection in Search Console.
• Storebot-Google: Storebot-Google is Google's shopping crawler. It fetches pages for Google Shopping surfaces such as the Shopping tab in Search.
• GoogleOther: GoogleOther is a general-purpose Google crawler that product teams use for jobs outside Search, such as research crawls. Blocking it doesn't affect Google Search.
• Google-Agent: Google-Agent is the name AI agents running on Google's infrastructure use when they browse sites and take actions because a user asked them to.
• Google-GeminiNotebook: Google-GeminiNotebook fetches a page when a Gemini Notebook user adds it as a source. It replaced the NotebookLM fetcher in July 2026.
• Google-Read-Aloud: Google-Read-Aloud fetches a page when someone asks Google to read it out loud. robots.txt doesn't stop it; a meta tag does.
• AdsBot-Google: AdsBot-Google checks the quality of Google Ads landing pages. It ignores the User-agent: * group, so it has to be named to be blocked, and blocking it can get your ads disapproved.
• Mediapartners-Google: Mediapartners-Google is the AdSense crawler. It reads pages that carry Google ad tags to pick relevant ads, and it ignores the User-agent: * group.