Slurp: what it is and how to block or allow it

What Slurp does

Slurp crawls and indexes pages for Yahoo Search. Yahoo says it also collects content from partner sites for Yahoo News, Finance and Sports. Yahoo's web results come from both Slurp and Bing's crawler.

• Yahoo suggests a Crawl-delay, or disallowing some folders, if Slurp puts too much load on a site. Source
• Yahoo Search results come from both Slurp and Bing's crawler. Source

Does Slurp follow robots.txt?

Yes. Yahoo says Slurp follows the robots exclusion standard. It uses the first group that names Slurp, otherwise the first * group.

It honors Crawl-delay, so you can slow it down instead of blocking it.

How to block or allow Slurp

Block it from the whole site:

User-agent: Slurp
Disallow: /

Keep it out of one folder only:

User-agent: Slurp
Disallow: /private/

Slow it down instead, with a pause in seconds between requests:

User-agent: Slurp
Crawl-delay: 10

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the Slurp group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking Slurp changes

Yahoo's own crawler stops fetching your pages, which can affect Yahoo Search and Yahoo's partner content. Because Yahoo's web results also come from Bing, blocking Slurp alone may not remove you from Yahoo.

Where to see Slurp visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes Slurp and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.