- Run by
- Yandex
- Type
- Search engine crawler
- Visits
- On its own schedule
- robots.txt
- Follows robots.txt
- robots.txt name
YandexBot- Documentation
- Yandex's page
What YandexBot does
YandexBot is Yandex's main indexing robot, which crawls pages for Yandex Search. Yandex runs many other specialised robots, such as YandexImages and YandexVideo, each with its own name.
- Yandex has ignored Crawl-delay since February 22, 2018. Source
- Yandex supports a Clean-param directive that tells it to ignore URL parameters such as UTM tags. Source
Does YandexBot follow robots.txt?
Yes. Yandex marks YandexBot as following the general robots.txt rules. Rules under User-agent: Yandex apply to every Yandex robot that follows general rules, and rules under YandexBot only to the main crawler. Some other Yandex robots, such as its ad landing page checker, don't follow the general rules.
It doesn't honor Crawl-delay.
How to block or allow YandexBot
Block it from the whole site:
User-agent: YandexBot
Disallow: /Keep it out of one folder only:
User-agent: YandexBot
Disallow: /private/ A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the YandexBot group. robots.txt works per host, so add the rules to each subdomain you want covered.
To slow it down, set the crawl rate in Yandex Webmaster.
Test the result with the robots.txt checker, or write a new file with the robots.txt generator.
What blocking YandexBot changes
Your pages stop being indexed for Yandex Search. Yandex says a page YandexBot can't reach isn't shown in results, even if another Yandex robot can reach it.
Where to see YandexBot visits
Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.
OneLence recognizes YandexBot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.
Frequently asked questions
What is YandexBot?
YandexBot is Yandex's main indexing robot, which crawls pages for Yandex Search. Yandex runs many other specialised robots, such as YandexImages and YandexVideo, each with its own name.
Does YandexBot follow robots.txt?
Yes. Yandex marks YandexBot as following the general robots.txt rules. Rules under User-agent: Yandex apply to every Yandex robot that follows general rules, and rules under YandexBot only to the main crawler. Some other Yandex robots, such as its ad landing page checker, don't follow the general rules.
How do I block YandexBot?
Add a group to robots.txt with User-agent: YandexBot, then Disallow: /. To slow it down, set the crawl rate in Yandex Webmaster.
What happens if I block YandexBot?
Your pages stop being indexed for Yandex Search. Yandex says a page YandexBot can't reach isn't shown in results, even if another Yandex robot can reach it.
Why doesn't YandexBot show up in Google Analytics?
Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.