Bot directory: who's crawling your site

Fetches pages for live requests, such as Alexa questions that need current information.

Visits a page when someone's request in ChatGPT needs it. OpenAI says robots.txt rules may not apply to it.

Fetches a page when someone asks Claude a question that needs it. Anthropic says it follows robots.txt.

DuckAssistBot fetches pages in real time for DuckDuckGo's AI-assisted answers. Blocking it takes you out of those answers without touching your search rankings.

Google-Agent is the name AI agents running on Google's infrastructure use when they browse sites and take actions because a user asked them to.

Google-GeminiNotebook fetches a page when a Gemini Notebook user adds it as a source. It replaced the NotebookLM fetcher in July 2026.

Google-Read-Aloud fetches a page when someone asks Google to read it out loud. robots.txt doesn't stop it; a meta tag does.

Kimi-User fetches pages when someone using Kimi, Moonshot AI's assistant, asks for something that needs live web content. robots.txt may not fully apply to it.

Fetches links at a user's request, including for Meta AI. Meta says it may bypass robots.txt.

Microsoft documents no separate Copilot crawler. Copilot cites pages from Bing's index, so bingbot rules and two meta tags decide whether it can use your pages.

MistralAI-User fetches a page when someone asks Mistral's assistant a question that needs it. Mistral says it doesn't crawl on its own or collect training data.

Visits a page when someone's question in Perplexity needs it. Perplexity says it generally ignores robots.txt.

Makes pages eligible for Amazon search experiences such as Alexa. Not used for AI training.

Crawls the web to improve Claude's search results.

Kimi-SearchBot builds the search index behind Kimi's search features. Blocking it keeps your site out of Kimi's search results.

Improves Meta AI's search results, so Meta AI can cite and link to your pages.

MistralAI-Index crawls the web to build the index behind Mistral's search. Mistral says what it crawls isn't used for AI training.

Finds and indexes pages for ChatGPT search. Blocking it keeps your pages out of ChatGPT's search answers.

Finds and links pages for Perplexity's search results. Perplexity says it isn't used to train AI models.

Helps improve Amazon's products and may be used to train Amazon AI models.

Not a crawler: the robots.txt setting that opts your pages out of training Apple's models.

ByteDance's crawler. Independent reports say it doesn't reliably follow robots.txt.

Builds Common Crawl's free web archive, a widely used source of AI training data.

Collects public web content that may be used to train Claude. It follows robots.txt and Crawl-delay.

cohere-ai is a name found in many robots.txt block lists. Cohere, the AI company it refers to, lists no active crawlers and says it doesn't currently crawl the web to train its models.

Not a crawler: the robots.txt setting for Gemini training and grounding. It doesn't affect Google Search.

Collects public content that may be used to train OpenAI's models. It has its own robots.txt setting.

Crawls the web for uses such as training Meta's foundation AI models.

MistralAI-Training collects public web content to train Mistral's AI models. Blocking it doesn't remove you from Mistral's search or answers.

Powers search in Siri, Spotlight and Safari. What it crawls may also train Apple's models.

Bingbot builds Bing's search index, which also feeds Microsoft Copilot's citations, DuckDuckGo and much of Yahoo. It follows robots.txt and Crawl-delay.

DuckDuckBot is DuckDuckGo's own crawler. Most standard links in DuckDuckGo come from Bing, so Bingbot matters more for your DuckDuckGo results.

Google-InspectionTool fetches your page when someone runs one of Google's Search testing tools, such as the Rich Results Test or URL Inspection in Search Console.

Googlebot is Google Search's crawler. Its crawl also decides whether your pages can appear in AI Overviews and AI Mode, so blocking it takes you out of both.

GoogleOther is a general-purpose Google crawler that product teams use for jobs outside Search, such as research crawls. Blocking it doesn't affect Google Search.

msnbot was the crawler for MSN Search and Live Search. Bingbot replaced it in 2010, but Bing still applies msnbot rules when robots.txt has no bingbot group.

PetalBot is the crawler for Huawei's Petal Search, whose index also powers Huawei Assistant and AI Search. Huawei says it follows robots.txt.

SeznamBot is the crawler for Seznam.cz, the Czech search engine. It fully follows robots.txt and supports a Request-rate rule to slow it down.

Slurp is Yahoo's crawler for Yahoo Search and its News, Finance and Sports partner content. Much of Yahoo's web results come from Bing instead.

Sogou web spider crawls pages for Sogou, a Chinese search engine. It follows robots.txt, and its name must match exactly, spaces and capitals included.

Storebot-Google is Google's shopping crawler. It fetches pages for Google Shopping surfaces such as the Shopping tab in Search.

YandexBot is the main crawler for Yandex Search. It follows robots.txt but has ignored Crawl-delay since 2018.

AhrefsBot crawls the web for Ahrefs' backlink database and its Yep search engine. Ahrefs says it strictly follows robots.txt and Crawl-delay.

Barkrowler crawls the web for Babbar's graph of links, which powers its SEO tools. It follows robots.txt and Crawl-delay.

BLEXBot collects the links on pages for a backlink database used by internet marketers. It follows robots.txt and Crawl-delay.

DataForSeoBot crawls pages for a backlink database that DataForSEO sells to SEO tools through its API. It follows robots.txt and Crawl-delay.

MJ12bot is Majestic's crawler, which maps links between sites for its backlink index. It follows robots.txt and Crawl-delay up to 20 seconds.

Screaming Frog SEO Spider is desktop software that SEOs run against sites they choose. It follows robots.txt by default, but its users can switch that off.

SemrushBot crawls the web for Semrush's backlink index. Semrush runs nine more bots for its other tools, and each needs its own robots.txt rule.

SEOkicks is a German crawler that stores links between sites in a backlink database. It reads robots.txt first and follows its blocking rules.

serpstatbot crawls the web for Serpstat's backlink index. It follows robots.txt and Crawl-delay up to 20 seconds.

When a link is shared in Messages, the device fetches the page to show a rich preview. It doesn't run JavaScript.

Discordbot fetches a link posted in Discord to build its embed preview. It doesn't run JavaScript, so preview tags must be in the HTML.

Embedly fetches the links that apps send to its API, to build previews and embeds. Its documentation doesn't mention robots.txt.

Fetches pages shared on Facebook, Instagram or Messenger to build their link previews.

Iframely builds link previews and embeds for apps that use its API. Its preview fetcher skips robots.txt by design; its other agents follow it.

Pinterestbot collects content and metadata for Pins that link to your site, and re-checks pages to keep Pins current. It follows robots.txt.

Slackbot fetches a link posted in Slack to show its preview. Slack says plainly that it doesn't honor robots.txt.

Twitterbot fetches a page when its link is posted on X, to build the link card. A page robots.txt blocks for it is posted without a card.

WhatsApp fetches a link typed or pasted into a chat to show its preview, reading the page's Open Graph tags.

AdIdxBot is Microsoft Advertising's crawler. It checks ads and the landing pages they link to.

AdsBot-Google checks the quality of Google Ads landing pages. It ignores the User-agent: * group, so it has to be named to be blocked, and blocking it can get your ads disapproved.

Mediapartners-Google is the AdSense crawler. It reads pages that carry Google ad tags to pick relevant ads, and it ignores the User-agent: * group.

Better Uptime Bot is Better Stack's uptime monitor. It checks the URLs customers add as monitors and alerts them to outages.

Jetmon is Jetpack's downtime monitor. It checks a WordPress site's homepage every five minutes once the owner turns the feature on.

Pingdom's probe servers check the URLs customers set up and alert them to outages and slow responses.

Site24x7's website monitors check URLs that customers enter, as often as every 10 seconds, and alert them when a site fails.

StatusCake runs uptime tests against the URLs its customers choose and alerts them when a site stops responding.

UptimeRobot checks the pages its customers set up as monitors and alerts them when a site goes down. If it visits you, someone is monitoring your site.

archive.org_bot captures public pages for the Wayback Machine. The Internet Archive relies on robots.txt less than it used to; exclusions go by email.

ia_archiver was Alexa Internet's crawler, long used to keep sites out of the Wayback Machine. Alexa shut down in May 2022, so the rule now likely does nothing.

Diffbot crawls the web for the Diffbot Knowledge Graph and runs crawls for its customers. It follows robots.txt by default, with exceptions.

ImagesiftBot crawls public images and the text around them for Hive's image search. Without a rule of its own, it follows your Googlebot rules.

Omgilibot is Webz.io's original crawler, built for the old Omgili search engine. Webz.io's newer Webzio bot builds on it and replaces it.

Webzio collects and structures web content for the data feeds Webz.io sells to other companies. It replaced the older Omgilibot.

Requests with WordPress in the user agent come from WordPress sites themselves: your own site's scheduled tasks, or other sites checking a pingback.