- Run by
- Diffbot
- Type
- Data crawler
- Visits
- On its own schedule
- robots.txt
- Partly follows robots.txt
- robots.txt name
Diffbot- Documentation
- Diffbot's page
What Diffbot does
The Diffbot name covers Diffbot's own crawl, which builds a general search index behind the Diffbot Knowledge Graph. Diffbot-User covers requests made for a person using Diffbot's software. Customers also run their own crawls with Diffbot's Crawlbot and Extract tools. Diffbot says its crawler isn't used to train generative AI foundation models.
- Diffbot advises customers running their own crawls to set a user agent naming their organization, so that traffic may not carry the Diffbot name. Source
Does Diffbot follow robots.txt?
By default, yes, including Crawl-delay. Diffbot says robots.txt can be overridden in specific cases, usually by agreement with the site, and customers running Crawlbot can turn robots.txt off.
It honors Crawl-delay, so you can slow it down instead of blocking it.
How to block or allow Diffbot
Block it from the whole site:
User-agent: Diffbot
Disallow: /Keep it out of one folder only:
User-agent: Diffbot
Disallow: /private/Slow it down instead, with a pause in seconds between requests:
User-agent: Diffbot
Crawl-delay: 10 A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the Diffbot group. robots.txt works per host, so add the rules to each subdomain you want covered.
Because Diffbot doesn't always obey robots.txt, a firewall or CDN rule that matches its user agent is the dependable way to stop it.
Test the result with the robots.txt checker, or write a new file with the robots.txt generator.
What blocking Diffbot changes
Diffbot's own crawl skips your pages, so they are less likely to be found and cited through its Knowledge Graph.
Where to see Diffbot visits
Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.
OneLence recognizes Diffbot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.
Frequently asked questions
What is Diffbot?
The Diffbot name covers Diffbot's own crawl, which builds a general search index behind the Diffbot Knowledge Graph. Diffbot-User covers requests made for a person using Diffbot's software. Customers also run their own crawls with Diffbot's Crawlbot and Extract tools. Diffbot says its crawler isn't used to train generative AI foundation models.
Does Diffbot follow robots.txt?
By default, yes, including Crawl-delay. Diffbot says robots.txt can be overridden in specific cases, usually by agreement with the site, and customers running Crawlbot can turn robots.txt off.
How do I block Diffbot?
Add a group to robots.txt with User-agent: Diffbot, then Disallow: /.
What happens if I block Diffbot?
Diffbot's own crawl skips your pages, so they are less likely to be found and cited through its Knowledge Graph.
Why doesn't Diffbot show up in Google Analytics?
Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.