msnbot: what it is and how to block or allow it

What msnbot does

msnbot collected pages for Microsoft's search index in the MSN Search and Live Search years. Microsoft switched its main crawler to Bingbot on October 1, 2010.

• Some of Bing's crawler hosts still have names that begin with msnbot, so the name can still appear in server logs. Source

Does msnbot follow robots.txt?

Bing said when it moved to Bingbot that it would keep honoring robots.txt rules written for msnbot. When a file has groups for both, the bingbot group wins.

How to block or allow msnbot

Block it from the whole site:

User-agent: msnbot
Disallow: /

Keep it out of one folder only:

User-agent: msnbot
Disallow: /private/

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the msnbot group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking msnbot changes

An msnbot group still applies to Bing's crawler when your robots.txt has no bingbot group. With a bingbot group in the file, msnbot rules have no effect.

Where to see msnbot visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes msnbot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.

Other bots from Microsoft

• Bingbot: Bingbot builds Bing's search index, which also feeds Microsoft Copilot's citations, DuckDuckGo and much of Yahoo. It follows robots.txt and Crawl-delay.
• AdIdxBot: AdIdxBot is Microsoft Advertising's crawler. It checks ads and the landing pages they link to.
• Microsoft Copilot: Microsoft documents no separate Copilot crawler. Copilot cites pages from Bing's index, so bingbot rules and two meta tags decide whether it can use your pages.