Screaming Frog SEO Spider: what it is and how to block or allow it

What Screaming Frog SEO Spider does

The SEO Spider is crawling software that SEOs and site owners run on their own computers to audit links, metadata, redirects and other technical issues. Each visit comes from someone running the tool on your site, often your own team or agency. It can render JavaScript when the user turns rendering on.

• Its default speed is 5 threads, and users can raise it. Screaming Frog warns that high settings can overload a server. Source

Does Screaming Frog SEO Spider follow robots.txt?

By default, yes: it applies rules for its own name first, then Googlebot's rules, then the * group. Users can switch it to ignore robots.txt, or make it identify as Googlebot, Bingbot or a custom name.

How to block or allow Screaming Frog SEO Spider

Block it from the whole site:

User-agent: Screaming Frog SEO Spider
Disallow: /

Keep it out of one folder only:

User-agent: Screaming Frog SEO Spider
Disallow: /private/

A crawler that finds a group with its own name follows only that group and ignores User-agent: *. If your * group disallows paths such as /admin/, repeat those lines in the Screaming Frog SEO Spider group. robots.txt works per host, so add the rules to each subdomain you want covered.

Because Screaming Frog SEO Spider doesn't always obey robots.txt, a firewall or CDN rule that matches its user agent is the dependable way to stop it.

Test the result with the robots.txt checker, or write a new file with the robots.txt generator.

What blocking Screaming Frog SEO Spider changes

Crawls with the default settings skip the blocked pages, but anyone can change those settings, so a robots.txt rule isn't a reliable block.

Where to see Screaming Frog SEO Spider visits

Most bots never appear in Google Analytics. Many don't run JavaScript, so the analytics tag never loads, and Google Analytics 4 automatically excludes traffic from known bots and spiders. Your server or CDN logs record every request with its user agent, so search them for the bot's name.

OneLence recognizes Screaming Frog SEO Spider and keeps it out of your visitor counts. With Premium, bot traffic shows which bots visit your site, how often and what kind they are.