DataForSeoBot: what it is and how to block or allow it

What DataForSeoBot does

DataForSeoBot crawls pages and adds the links it finds to DataForSEO's backlink database, which developers and SEO tools buy through its Backlinks API. DataForSEO says it collects only links.

• Its default pace is one visit every 5 seconds. Source

Does DataForSeoBot follow robots.txt?

Yes. DataForSEO says the bot supports robots.txt, including Disallow rules with wildcards and Crawl-delay.

It honors Crawl-delay, so you can slow it down instead of blocking it.

How to block or allow DataForSeoBot

Block it from the whole site:

User-agent: DataForSeoBot
Disallow: /

Keep it out of one folder only:

User-agent: DataForSeoBot
Disallow: /private/

Slow it down instead, with a pause in seconds between requests:

User-agent: DataForSeoBot
Crawl-delay: 10

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the DataForSeoBot group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking DataForSeoBot changes

DataForSEO stops collecting links from your pages for the backlink data it sells.

Where to see DataForSeoBot visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes DataForSeoBot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.