SemrushBot: what it is and how to block or allow it

What SemrushBot does

SemrushBot discovers and re-checks pages to build the link graph behind Semrush's Backlink Analytics. Semrush's other tools have their own bots: SiteAuditBot for Site Audit, SemrushBot-BA for Backlink Audit, SemrushBot-SI for the On Page SEO Checker, SemrushBot-SWA for the SEO Writing Assistant, SplitSignalBot, SemrushBot-OCOB for the Content Toolkit, SemrushBot-FT for the Plagiarism Checker, SemrushBot-ESI for Enterprise Site Intelligence, and RyteBot for Ryte's tools.

• The backlink crawler honors Crawl-delay up to 10 seconds; larger values count as 10. Source
• If robots.txt answers with a server error, SemrushBot stops crawling the whole site. A 4xx counts as no robots.txt. Source
• Semrush users start Site Audit crawls. At its default speed SiteAuditBot ignores Crawl-delay, unless the user picks the setting that respects robots.txt. Source
• A site owner who verifies the site can let SiteAuditBot crawl pages robots.txt disallows. Source

Does SemrushBot follow robots.txt?

Semrush gives robots.txt as the way to block each of its bots, says it can take up to an hour or 100 requests to pick up a change, and asks owners to email [email protected] with logs if a rule seems ignored.

It honors Crawl-delay, so you can slow it down instead of blocking it.

How to block or allow SemrushBot

Block it from the whole site:

User-agent: SemrushBot
Disallow: /

Keep it out of one folder only:

User-agent: SemrushBot
Disallow: /private/

Slow it down instead, with a pause in seconds between requests:

User-agent: SemrushBot
Crawl-delay: 10

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the SemrushBot group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking SemrushBot changes

Semrush stops crawling your site for its backlink index. Each tool bot has to be blocked separately to stop that tool.

Where to see SemrushBot visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes SemrushBot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.