Barkrowler: what it is and how to block or allow it

What Barkrowler does

Barkrowler builds and updates Babbar's graph of the web, which powers its marketing and SEO tools. Babbar says it stores mainly links and page meta information, not content.

• By default it waits 5 seconds between requests to the same host. Delays apply per host, so a server hosting many domains can see several of its crawlers at once. Source
• Babbar notes that security plugins sometimes mistake it for an attack. Source

Does Barkrowler follow robots.txt?

Yes. Babbar says it respects robots.txt and Disallow rules.

It honors Crawl-delay, so you can slow it down instead of blocking it.

How to block or allow Barkrowler

Block it from the whole site:

User-agent: barkrowler
Disallow: /

Keep it out of one folder only:

User-agent: barkrowler
Disallow: /private/

Slow it down instead, with a pause in seconds between requests:

User-agent: barkrowler
Crawl-delay: 10

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the Barkrowler group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking Barkrowler changes

Babbar stops crawling the disallowed paths, so their links are no longer added to its graph.

Where to see Barkrowler visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes Barkrowler and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.