- Run by
- Majestic
- Type
- SEO crawler
- Visits
- On its own schedule
- robots.txt
- Follows robots.txt
- robots.txt name
MJ12bot- Documentation
- Majestic's page
What MJ12bot does
MJ12bot is Majestic's distributed crawler. The links it maps feed Majestic's backlink index, used in tools such as Site Explorer. Majestic says it stores only a graph of links, not page content.
- Crawl-delay is honored up to 20 seconds; higher values count as 20. Source
- A group for MJ12bot replaces the * group, so repeat any Crawl-delay from * in its group. Source
- It runs on a community network without a fixed IP range, so Majestic asks owners to use robots.txt rather than IP blocks. Source
Does MJ12bot follow robots.txt?
Yes. Majestic says MJ12bot follows the robots.txt standard and some extensions, such as Crawl-delay and wildcards in Disallow.
It honors Crawl-delay, so you can slow it down instead of blocking it.
How to block or allow MJ12bot
Block it from the whole site:
User-agent: MJ12bot
Disallow: /Keep it out of one folder only:
User-agent: MJ12bot
Disallow: /private/Slow it down instead, with a pause in seconds between requests:
User-agent: MJ12bot
Crawl-delay: 10 A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the MJ12bot group. robots.txt works per host, so add the rules to each subdomain you want covered.
Test the result with the robots.txt checker, or write a new file with the robots.txt generator.
What blocking MJ12bot changes
Majestic stops crawling your pages, so the links on them are no longer collected for its backlink index.
Where to see MJ12bot visits
Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.
OneLence recognizes MJ12bot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.
Frequently asked questions
What is MJ12bot?
MJ12bot is Majestic's distributed crawler. The links it maps feed Majestic's backlink index, used in tools such as Site Explorer. Majestic says it stores only a graph of links, not page content.
Does MJ12bot follow robots.txt?
Yes. Majestic says MJ12bot follows the robots.txt standard and some extensions, such as Crawl-delay and wildcards in Disallow.
How do I block MJ12bot?
Add a group to robots.txt with User-agent: MJ12bot, then Disallow: /.
What happens if I block MJ12bot?
Majestic stops crawling your pages, so the links on them are no longer collected for its backlink index.
Why doesn't MJ12bot show up in Google Analytics?
Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.