Google-Agent: what it is and how to block or allow it

What Google-Agent does

Google-Agent identifies AI agents on Google's infrastructure that browse websites and take actions at a user's request. Google added it to its crawler documentation in March 2026. Its page doesn't name the agent products that use it.

• Google is testing the Web Bot Auth protocol for Google-Agent, so sites can check its requests cryptographically. Source
• Google documents no robots.txt or meta tag opt-out for Google-Agent. Source
• Google lists Google-Agent with its other user-triggered fetchers, such as Google-Read-Aloud and Google-GeminiNotebook. Source

Does Google-Agent follow robots.txt?

Generally not. Google classes Google-Agent as a user-triggered fetcher, and says that because a user requested the fetch, these fetchers generally ignore robots.txt rules.

It doesn't honor Crawl-delay.

How to stop Google-Agent

robots.txt won't reliably stop it. A firewall or CDN rule that matches its user agent does, and also stops the feature for the people who asked for it.

Where to see Google-Agent visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence AI visibility shows which pages AI crawlers read and fetch for answers, next to the visitors AI assistants send you and the ones that convert.

Other bots from Google

• Google-Extended: Not a crawler: the robots.txt setting for Gemini training and grounding. It doesn't affect Google Search.
• Googlebot: Googlebot is Google Search's crawler. Its crawl also decides whether your pages can appear in AI Overviews and AI Mode, so blocking it takes you out of both.
• Google-InspectionTool: Google-InspectionTool fetches your page when someone runs one of Google's Search testing tools, such as the Rich Results Test or URL Inspection in Search Console.
• Storebot-Google: Storebot-Google is Google's shopping crawler. It fetches pages for Google Shopping surfaces such as the Shopping tab in Search.
• GoogleOther: GoogleOther is a general-purpose Google crawler that product teams use for jobs outside Search, such as research crawls. Blocking it doesn't affect Google Search.
• Google-GeminiNotebook: Google-GeminiNotebook fetches a page when a Gemini Notebook user adds it as a source. It replaced the NotebookLM fetcher in July 2026.
• Google-Read-Aloud: Google-Read-Aloud fetches a page when someone asks Google to read it out loud. robots.txt doesn't stop it; a meta tag does.
• AdsBot-Google: AdsBot-Google checks the quality of Google Ads landing pages. It ignores the User-agent: * group, so it has to be named to be blocked, and blocking it can get your ads disapproved.
• Mediapartners-Google: Mediapartners-Google is the AdSense crawler. It reads pages that carry Google ad tags to pick relevant ads, and it ignores the User-agent: * group.