- Run by
- Internet Archive
- Type
- Web archive crawler
- Visits
- On its own schedule
- robots.txt
- Partly follows robots.txt
- Documentation
- Internet Archive's page
What archive.org_bot does
The Internet Archive's crawler captures public pages for the Wayback Machine, finding sites mostly through links from other sites. Pages saved with Save Page Now are captured because a person asked.
- The Wayback Machine's director wrote in 2017 that robots.txt files meant for search engines don't suit archives, partly because a parked domain's new robots.txt used to hide a site's whole history. Source
Does archive.org_bot follow robots.txt?
Partly. The Internet Archive's help center says robots.txt may have stopped a site being crawled or shown, but the Archive has ignored robots.txt for US government and military sites since about 2016, and said in 2017 it would rely on it less in general.
How to stop archive.org_bot
To exclude your site, email [email protected] with the URLs and the period you want excluded. Each request is reviewed case by case.
What blocking archive.org_bot changes
New snapshots of your site may stop being added. Existing ones stay unless the Archive agrees to exclude them.
Where to see archive.org_bot visits
Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.
OneLence recognizes archive.org_bot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.
Frequently asked questions
What is archive.org_bot?
The Internet Archive's crawler captures public pages for the Wayback Machine, finding sites mostly through links from other sites. Pages saved with Save Page Now are captured because a person asked.
Does archive.org_bot follow robots.txt?
Partly. The Internet Archive's help center says robots.txt may have stopped a site being crawled or shown, but the Archive has ignored robots.txt for US government and military sites since about 2016, and said in 2017 it would rely on it less in general.
How do I block archive.org_bot?
To exclude your site, email [email protected] with the URLs and the period you want excluded. Each request is reviewed case by case.
What happens if I block archive.org_bot?
New snapshots of your site may stop being added. Existing ones stay unless the Archive agrees to exclude them.
Why doesn't archive.org_bot show up in Google Analytics?
Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.