Bingbot: what it is and how to block or allow it

What Bingbot does

Bingbot is Microsoft's main web crawler. It builds the Bing search index, and Microsoft says Copilot and Bing's AI summaries cite pages from that same index. DuckDuckGo and Yahoo take much of their results from Bing too, so Bingbot reaches further than Bing itself.

• A Crawl-delay in robots.txt overrides the crawl-rate settings in Bing Webmaster Tools. Source
• Bingbot renders JavaScript with an up-to-date version of Microsoft Edge. Source
• A NOCACHE meta tag limits Bing's AI answers to your URL, title and snippet. NOARCHIVE keeps the page out of those answers while it stays in search results. Source
• Bing Webmaster Tools has an AI Performance report that shows how often your pages are cited in Copilot and Bing's AI summaries. Source

Does Bingbot follow robots.txt?

Yes. Bing says it respects the preferences content owners express in robots.txt and its other supported controls.

It honors Crawl-delay, so you can slow it down instead of blocking it.

How to block or allow Bingbot

Block it from the whole site:

User-agent: bingbot
Disallow: /

Keep it out of one folder only:

User-agent: bingbot
Disallow: /private/

Slow it down instead, with a pause in seconds between requests:

User-agent: bingbot
Crawl-delay: 10

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the Bingbot group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking Bingbot changes

Your pages drop out of Bing's index, and with it out of the sources Copilot can cite and the results DuckDuckGo and Yahoo take from Bing.

Where to see Bingbot visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes Bingbot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.

Other bots from Microsoft

• msnbot: msnbot was the crawler for MSN Search and Live Search. Bingbot replaced it in 2010, but Bing still applies msnbot rules when robots.txt has no bingbot group.
• AdIdxBot: AdIdxBot is Microsoft Advertising's crawler. It checks ads and the landing pages they link to.
• Microsoft Copilot: Microsoft documents no separate Copilot crawler. Copilot cites pages from Bing's index, so bingbot rules and two meta tags decide whether it can use your pages.