ia_archiver: what it is and how to block or allow it

What ia_archiver does

ia_archiver gathered pages for Alexa's web rankings, and much of its crawl went to the Internet Archive's Wayback Machine. Amazon retired Alexa.com on May 1, 2022.

• Amazon shut down Alexa.com on May 1, 2022. Source

Does ia_archiver follow robots.txt?

The Internet Archive's old FAQ said Alexa respected robots.txt, even retroactively. Alexa's own documentation no longer exists.

How to block or allow ia_archiver

Block it from the whole site:

User-agent: ia_archiver
Disallow: /

Keep it out of one folder only:

User-agent: ia_archiver
Disallow: /private/

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the ia_archiver group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking ia_archiver changes

Since Alexa shut down, an ia_archiver rule is unlikely to change anything. A visit that still carries the name after 2022 is worth treating as suspect.

Where to see ia_archiver visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes ia_archiver and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.