OpenAI sends different agents for different jobs, and the robots.txt setting for each is independent of the others. OpenAI's own example: allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to keep your content out of training.
OpenAI's crawlers at a glance
| Agent | What it does | What robots.txt does | IP list |
|---|---|---|---|
OAI-SearchBot | Finds websites to show in ChatGPT's search features | A Disallow keeps your pages out of ChatGPT search answers; they can still appear as navigational links | searchbot.json |
GPTBot | Crawls content that may be used to train OpenAI's generative AI foundation models | A Disallow tells OpenAI not to use your content for training | gptbot.json |
ChatGPT-User | Visits a page for certain user actions in ChatGPT and custom GPTs, such as answering a question | Rules may not apply, because a user started the action | chatgpt-user.json |
A fourth agent, OAI-AdsBot, visits only the landing pages of ads submitted to ChatGPT to check that they follow OpenAI's ad policies. OpenAI says the data it collects isn't used to train foundation models. Its IP list is adsbot.json.
ChatGPT-User: the visit behind an answer
When someone asks ChatGPT something that needs a web page, ChatGPT may visit it with the ChatGPT-User agent. OpenAI says that because a user initiated the visit, robots.txt rules may not apply. It also says ChatGPT-User isn't used to crawl the web automatically or to decide whether content appears in ChatGPT search.
So a ChatGPT-User request in your logs means a person's conversation needed that page at that moment. The request shows that ChatGPT read the page, not whether the answer quoted or linked it.
GPTBot: training data
GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it tells OpenAI your content shouldn't be used for training. It doesn't decide whether you appear in ChatGPT search; that's OAI-SearchBot's job. If you allow both, OpenAI may use the results of one crawl for both purposes to avoid crawling twice.
OpenAI applies the same GPTBot opt-out to page content it gets through people's use of its ChatGPT Atlas browser. Even when an Atlas user opts in to training, pages that disallow GPTBot aren't trained on.
OAI-SearchBot: ChatGPT search
OAI-SearchBot finds the pages ChatGPT's search features show and link to. For your content to be included in summaries and snippets in ChatGPT, OpenAI says not to block it. Any public website can appear in ChatGPT search. After you change robots.txt, it can take about 24 hours for ChatGPT search to adjust.
robots.txt rules for OpenAI's crawlers
Stay in ChatGPT search but opt out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Opt out of ChatGPT search as well:
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /
A crawler that finds a group with its own name follows only that group and ignores the User-agent: * group. If your * group disallows paths such as /admin/, repeat those lines in each named group that should still respect them.
User agent strings
OpenAI publishes these strings. It calls the GPTBot and OAI-SearchBot strings examples, so match on the agent's name rather than the whole string.
OAI-SearchBot (example):
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
GPTBot (example):
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
ChatGPT-User:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Any script can send these strings. To confirm a request came from OpenAI, check its source IP address against the list for that agent in the table above.
Where to see OpenAI's crawlers and the visitors ChatGPT sends
Google Analytics 4 automatically excludes traffic from known bots and spiders, and the exclusion can't be turned off, so OpenAI's crawlers won't appear there. Your server and CDN logs record every request with its user agent and IP address.
People who click a link in ChatGPT's search results are different: ChatGPT adds utm_source=chatgpt.com to those links, so the visits show up in your analytics under that source. To find them in GA4, see how to track ChatGPT and AI traffic.
OneLence AI visibility puts both sides in one report when your server reports page requests to OneLence: the pages GPTBot and OAI-SearchBot read, the pages ChatGPT fetched while answering people, the visitors ChatGPT sent to each page and the ones that converted.
Frequently asked questions
What is ChatGPT-User?
ChatGPT-User is the user agent ChatGPT uses when a person's request needs a web page, for example when they ask a question in ChatGPT or use a custom GPT. OpenAI says it isn't used to crawl the web automatically or to decide what appears in ChatGPT search.
Does ChatGPT-User follow robots.txt?
Not necessarily. OpenAI says that because these actions are initiated by a user, robots.txt rules may not apply. GPTBot and OAI-SearchBot, which crawl automatically, are the agents robots.txt controls.
What's the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. OAI-SearchBot finds websites to show in ChatGPT's search features. OpenAI treats the two settings independently, so you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot.
Does blocking GPTBot remove my site from ChatGPT?
No. Blocking GPTBot tells OpenAI not to use your content for training. Whether your pages appear in ChatGPT search depends on OAI-SearchBot, and ChatGPT can still visit a page as ChatGPT-User when someone's request needs it.
How do I get my site into ChatGPT search?
Don't block OAI-SearchBot. OpenAI says any public website can appear in ChatGPT search, and that sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links. Changes to robots.txt can take about 24 hours to reach ChatGPT search.
How do I track visitors from ChatGPT?
ChatGPT adds utm_source=chatgpt.com to the links people click in its search results, so those visits show up under that source in your analytics. ChatGPT-User requests aren't visitors: they are automated fetches, which belong in your server logs, not in your visitor numbers.
How can I tell whether a request really came from OpenAI?
Check the source IP address against the list OpenAI publishes for that agent: openai.com/searchbot.json for OAI-SearchBot, openai.com/gptbot.json for GPTBot and openai.com/chatgpt-user.json for ChatGPT-User. Anyone can copy a user agent string.