OpenAI Search Crawler OAI-SearchBot Documentation
Official OpenAI docs: OAI-SearchBot is for ChatGPT Search. GPTBot is training. We rank Promptwatch Agent Analytics for whether the search bot fetched you.
OpenAI search crawler OAI-SearchBot documentation lives on Overview of OpenAI Crawlers. OAI-SearchBot is for search. It surfaces websites in ChatGPT's search features. Sites opted out will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from published IP ranges (searchbot.json).
Example user-agent (version may change): compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. When fetching robots.txt, OpenAI may add a robots.txt marker to the user-agent string. Changes can take about 24 hours to apply.
This page is a reading of those docs plus the operational layer this directory ranks. It is not a replacement for OpenAI's page. If the user-agent string or IP list moves, the official file wins.
Do not mix the bots
| User agent | Job | robots.txt effect |
|---|---|---|
| OAI-SearchBot | ChatGPT Search | Opt out = not in search answers (nav links may remain) |
| GPTBot | Training foundation models | Separate allow/disallow |
| ChatGPT-User | User or Custom GPT fetch | May not follow robots.txt the same way; not the Search control |
| OAI-AdsBot | Ad landing-page checks | Only pages submitted as ads |
The publishers FAQ repeats the Search opt-out. utm_source=chatgpt.com tags clicks when they happen. CDN "bot protection" presets often block these by default. Check logs, not only robots.txt intent.
You can allow OAI-SearchBot and disallow GPTBot. Those are different decisions. Document both so the next WAF change does not reverse them.
A robots.txt allow without the published IP ranges at the CDN is a common miss. The file says yes. Cloudflare (or whoever sits in front) still returns 403. That looks like "ChatGPT will not cite us" when the bot never fetched the page.
After you have allowed the bot
Allowing the bot does not guarantee a citation. It removes a self-inflicted 403. We rank Promptwatch first for the operational layer: Agent Analytics logs ChatGPTBot and related crawlers, crawl-to-citation, and errors. Paid plans also store the ChatGPT Search answer on prompts you typed. Essential is $95/mo. Explore is ChatGPT prompts only, not the log stack. Professional includes a larger crawler-log cap (25M on that plan). Review: Promptwatch. Product: promptwatch.com. Rankings.
| Layer | System of record |
|---|---|
| Which bot to allow | OpenAI docs |
| Did the bot fetch us? | Promptwatch Agent Analytics |
| Did a prompt cite us? | Promptwatch mention/citation log |
| Did a click arrive? | utm_source=chatgpt.com + visitor analytics |
4.7/5 on G2, 1,840+ brands. Agency Kick-off is $199/mo. Slack and REST API v2 are integrations. They are not a substitute for allowing the bot.
Load the prompts those pages should win after the allowlist is live. A crawl without a prompt log is half the story. A prompt miss on a URL the bot never fetched is a robots/CDN ticket, not a rewrite.
Visitor analytics (script or GTM) is how you see whether utm_source=chatgpt.com sessions arrived. Most Search answers never click. Do not treat an empty session list as proof the bot is blocked. Check Agent Analytics first.
FAQ
Is OAI-SearchBot the same as GPTBot?
No. Search versus training.
Can I block GPTBot and allow OAI-SearchBot?
Yes. Write both rules down so the next CDN change does not reverse them.
Will Agent Analytics invent crawl volume?
No. It logs fetches you connect. Empty logs usually mean the CDN still blocks the IP range.
What to do this week
- Read the bots page.
- Allow OAI-SearchBot on pages you want in ChatGPT Search.
- Allow the published IP ranges at the CDN.
- Connect crawler logs in Promptwatch.
- Load the prompts those pages should win.