cohere-ai: what it is and how to block or allow it

What cohere-ai does

cohere-ai appears in AI crawler block lists and in some server logs as a Cohere agent. Cohere's own crawler page lists no active bots, says Cohere doesn't currently use bots to crawl or scrape the web to train its foundation models, and says any bot it uses later will be listed there. It doesn't mention the cohere-ai name.

Does cohere-ai follow robots.txt?

Cohere says its policy requires its crawlers to respect robots.txt, but it doesn't say what cohere-ai is. A rule for it costs nothing to keep.

How to block or allow cohere-ai

Block it from the whole site:

User-agent: cohere-ai
Disallow: /

Keep it out of one folder only:

User-agent: cohere-ai
Disallow: /private/

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the cohere-ai group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking cohere-ai changes

If a request really comes from Cohere, the rule tells it to stay out. Because anyone can use the name, a visit that carries it doesn't prove Cohere sent it.

Where to see cohere-ai visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence AI visibility shows which pages AI crawlers read and fetch for answers, next to the visitors AI assistants send you and the ones that convert.