Twitterbot: what it is and how to block or allow it

What Twitterbot does

Twitterbot visits a page when someone posts its link on X. It reads the card markup (title, description and image) to build the link card shown in the post.

Does Twitterbot follow robots.txt?

Twitter's cards documentation said its crawler respects robots.txt: a blocked page gets no card, and a blocked image gets no thumbnail. X has since removed that documentation, so the current behavior isn't published.

How to block or allow Twitterbot

Block it from the whole site:

User-agent: Twitterbot
Disallow: /

Keep it out of one folder only:

User-agent: Twitterbot
Disallow: /private/

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the Twitterbot group. robots.txt works per host, so add the rules to each subdomain you want covered.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking Twitterbot changes

Links to your pages are posted on X without a card. Block only the folders you don't want previewed, or allow it explicitly if your * group is strict.

Where to see Twitterbot visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes Twitterbot and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.