PetalBot: what it is and how to block or allow it

What PetalBot does

PetalBot crawls desktop and mobile sites for Huawei's Petal Search. Huawei says the index also powers content recommendations in Huawei Assistant and AI Search. It was called AspiegelBot until around May 2020.

• Hypernode reported a 460% jump in AspiegelBot traffic in May 2020. Source

Does PetalBot follow robots.txt?

Huawei says PetalBot follows the robots protocol and can be blocked for a whole site or for specific paths. In 2020, under its old name AspiegelBot, the hosting company Hypernode reported a sharp rise in its traffic and said it didn't respect robots.txt.

How to block or allow PetalBot

Block it from the whole site:

User-agent: PetalBot
Disallow: /

Keep it out of one folder only:

User-agent: PetalBot
Disallow: /private/

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the PetalBot group. robots.txt works per host, so add the rules to each subdomain you want covered.

Huawei gives no Crawl-delay option. It says PetalBot adjusts its rate to your server, and asks owners to report crawling that seems unreasonable.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking PetalBot changes

Your pages become unsearchable in Petal Search and the Huawei services built on it. Pages already indexed may stay searchable for several months.

Where to see PetalBot visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes PetalBot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.