Mediapartners-Google: what it is and how to block or allow it

What Mediapartners-Google does

Mediapartners-Google visits sites in Google's ad programs to work out what each page is about, so AdSense and Google Ad Manager can show relevant ads. It only tries to fetch URLs that carry Google ad tags, on Google's own schedule.

• It is separate from Googlebot but shares a cache with it. Source
• Google says it can't take requests for more frequent AdSense crawling. Source

Does Mediapartners-Google follow robots.txt?

Partly. Google classes it as a special-case crawler that ignores the User-agent: * group. It obeys rules that name Mediapartners-Google.

It doesn't honor Crawl-delay.

How to block or allow Mediapartners-Google

Block it from the whole site:

User-agent: Mediapartners-Google
Disallow: /

Keep it out of one folder only:

User-agent: Mediapartners-Google
Disallow: /private/

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the Mediapartners-Google group. robots.txt works per host, so add the rules to each subdomain you want covered.

Because Mediapartners-Google doesn't always obey robots.txt, a firewall or CDN rule that matches its user agent is the dependable way to stop it.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking Mediapartners-Google changes

Google's ad systems can't read those pages to choose relevant ads. Google says AdSense crawling has no effect on Google Search rankings.

Where to see Mediapartners-Google visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes Mediapartners-Google and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.

Other bots from Google

• Google-Extended: Not a crawler: the robots.txt setting for Gemini training and grounding. It doesn't affect Google Search.
• Googlebot: Googlebot is Google Search's crawler. Its crawl also decides whether your pages can appear in AI Overviews and AI Mode, so blocking it takes you out of both.
• Google-InspectionTool: Google-InspectionTool fetches your page when someone runs one of Google's Search testing tools, such as the Rich Results Test or URL Inspection in Search Console.
• Storebot-Google: Storebot-Google is Google's shopping crawler. It fetches pages for Google Shopping surfaces such as the Shopping tab in Search.
• GoogleOther: GoogleOther is a general-purpose Google crawler that product teams use for jobs outside Search, such as research crawls. Blocking it doesn't affect Google Search.
• Google-Agent: Google-Agent is the name AI agents running on Google's infrastructure use when they browse sites and take actions because a user asked them to.
• Google-GeminiNotebook: Google-GeminiNotebook fetches a page when a Gemini Notebook user adds it as a source. It replaced the NotebookLM fetcher in July 2026.
• Google-Read-Aloud: Google-Read-Aloud fetches a page when someone asks Google to read it out loud. robots.txt doesn't stop it; a meta tag does.
• AdsBot-Google: AdsBot-Google checks the quality of Google Ads landing pages. It ignores the User-agent: * group, so it has to be named to be blocked, and blocking it can get your ads disapproved.