Storebot-Google: what it is and how to block or allow it

What Storebot-Google does

Storebot-Google crawls pages for Google Shopping. Google says its crawl affects every Shopping surface, such as the Shopping tab in Search and shopping.google.com.

• Google's robots.txt rules use Storebot-Google as the example of a crawler that falls back to the * group when it has no group of its own. The two are never merged. Source

Does Storebot-Google follow robots.txt?

Yes. Google lists Storebot-Google among its common crawlers, which always obey robots.txt. It doesn't follow rules written for Googlebot: it needs its own group, or it uses the User-agent: * group.

It doesn't honor Crawl-delay.

How to block or allow Storebot-Google

Block it from the whole site:

User-agent: Storebot-Google
Disallow: /

Keep it out of one folder only:

User-agent: Storebot-Google
Disallow: /private/

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the Storebot-Google group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking Storebot-Google changes

Google can no longer crawl those pages for its Shopping surfaces, which can affect how your products appear in Google Shopping.

Where to see Storebot-Google visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes Storebot-Google and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.

Other bots from Google

• Google-Extended: Not a crawler: the robots.txt setting for Gemini training and grounding. It doesn't affect Google Search.
• Googlebot: Googlebot is Google Search's crawler. Its crawl also decides whether your pages can appear in AI Overviews and AI Mode, so blocking it takes you out of both.
• Google-InspectionTool: Google-InspectionTool fetches your page when someone runs one of Google's Search testing tools, such as the Rich Results Test or URL Inspection in Search Console.
• GoogleOther: GoogleOther is a general-purpose Google crawler that product teams use for jobs outside Search, such as research crawls. Blocking it doesn't affect Google Search.
• Google-Agent: Google-Agent is the name AI agents running on Google's infrastructure use when they browse sites and take actions because a user asked them to.
• Google-GeminiNotebook: Google-GeminiNotebook fetches a page when a Gemini Notebook user adds it as a source. It replaced the NotebookLM fetcher in July 2026.
• Google-Read-Aloud: Google-Read-Aloud fetches a page when someone asks Google to read it out loud. robots.txt doesn't stop it; a meta tag does.
• AdsBot-Google: AdsBot-Google checks the quality of Google Ads landing pages. It ignores the User-agent: * group, so it has to be named to be blocked, and blocking it can get your ads disapproved.
• Mediapartners-Google: Mediapartners-Google is the AdSense crawler. It reads pages that carry Google ad tags to pick relevant ads, and it ignores the User-agent: * group.