- Run by
- Ahrefs
- Type
- SEO crawler
- Visits
- On its own schedule
- robots.txt
- Follows robots.txt
- robots.txt name
AhrefsBot- Documentation
- Ahrefs' page
What AhrefsBot does
AhrefsBot crawls the web and records the links it finds to build Ahrefs' backlink and SEO database. Ahrefs says the same crawl powers Yep, its search engine. It follows links it finds rather than guessing URLs, and re-crawls to check known links. Ahrefs' Site Audit tool uses a separate bot, AhrefsSiteAudit.
- Crawl-delay applies to page requests but not while it renders JavaScript, so asset requests can appear more often in logs. Source
- Temporary 4xx or 5xx status codes make it back off, which Ahrefs suggests during outages or maintenance. Source
- AhrefsSiteAudit crawls only for Ahrefs Site Audit projects, at up to 30 URLs a minute unless the site owner verifies the site. Verified owners can let it crawl pages robots.txt disallows. Source
Does AhrefsBot follow robots.txt?
Yes. Ahrefs says AhrefsBot strictly respects robots.txt Disallow and Allow rules and Crawl-delay. If your robots.txt has errors, it can't read it and keeps crawling as before.
It honors Crawl-delay, so you can slow it down instead of blocking it.
How to block or allow AhrefsBot
Block it from the whole site:
User-agent: AhrefsBot
Disallow: /Keep it out of one folder only:
User-agent: AhrefsBot
Disallow: /private/Slow it down instead, with a pause in seconds between requests:
User-agent: AhrefsBot
Crawl-delay: 10 A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the AhrefsBot group. robots.txt works per host, so add the rules to each subdomain you want covered.
Test the result with the robots.txt checker, or write a new file with the robots.txt generator.
What blocking AhrefsBot changes
Ahrefs stops crawling your pages, so links on them are no longer found or rechecked for its index, and they aren't crawled for Yep.
Where to see AhrefsBot visits
Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.
OneLence recognizes AhrefsBot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.
Frequently asked questions
What is AhrefsBot?
AhrefsBot crawls the web and records the links it finds to build Ahrefs' backlink and SEO database. Ahrefs says the same crawl powers Yep, its search engine. It follows links it finds rather than guessing URLs, and re-crawls to check known links. Ahrefs' Site Audit tool uses a separate bot, AhrefsSiteAudit.
Does AhrefsBot follow robots.txt?
Yes. Ahrefs says AhrefsBot strictly respects robots.txt Disallow and Allow rules and Crawl-delay. If your robots.txt has errors, it can't read it and keeps crawling as before.
How do I block AhrefsBot?
Add a group to robots.txt with User-agent: AhrefsBot, then Disallow: /.
What happens if I block AhrefsBot?
Ahrefs stops crawling your pages, so links on them are no longer found or rechecked for its index, and they aren't crawled for Yep.
Why doesn't AhrefsBot show up in Google Analytics?
Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.