BLEXBot: what it is and how to block or allow it

What BLEXBot does

BLEXBot crawls pages to collect the links on them, so marketers can see how sites link to each other. It keeps only the links. Its documentation is now on SE Ranking's help center; it was long known as WebMeUp's crawler.

• By default it makes at most one request every 3 seconds, or fewer with a Crawl-delay. Source

Does BLEXBot follow robots.txt?

Yes. Its help page says BLEXBot respects the rules in your robots.txt. After a block, allow up to 10 minutes for crawling to stop, because pages already queued may still be fetched.

It honors Crawl-delay, so you can slow it down instead of blocking it.

How to block or allow BLEXBot

Block it from the whole site:

User-agent: BLEXBot
Disallow: /

Keep it out of one folder only:

User-agent: BLEXBot
Disallow: /private/

Slow it down instead, with a pause in seconds between requests:

User-agent: BLEXBot
Crawl-delay: 10

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the BLEXBot group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking BLEXBot changes

The crawler stops collecting links from your pages for its link database.

Where to see BLEXBot visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes BLEXBot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.