serpstatbot: what it is and how to block or allow it

What serpstatbot does

serpstatbot crawls the web continuously to find new links and track changes for Serpstat's backlink index, which Serpstat users query in its backlink tools.

• It accepts a Crawl-delay of up to 20 seconds, including one set under User-agent: *. Source

Does serpstatbot follow robots.txt?

Yes. Serpstat says serpstatbot follows the robots.txt standard and extensions such as Crawl-delay and wildcards in Disallow. If it can't read your robots.txt, for example because of a 403, it crawls as if there were no rules.

It honors Crawl-delay, so you can slow it down instead of blocking it.

How to block or allow serpstatbot

Block it from the whole site:

User-agent: serpstatbot
Disallow: /

Keep it out of one folder only:

User-agent: serpstatbot
Disallow: /private/

Slow it down instead, with a pause in seconds between requests:

User-agent: serpstatbot
Crawl-delay: 10

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the serpstatbot group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking serpstatbot changes

Serpstat stops crawling your pages, so links on them are no longer added to or updated in its index.

Where to see serpstatbot visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes serpstatbot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.