GPTBot, OAI-SearchBot, ChatGPT-User and Friends: What Each AI Crawler Does
A plain guide to the real AI bot names in your logs, which ones fetch pages to cite them, and how to read them with Promptwatch Agent Analytics.
Open a raw server log and you will find a zoo of user agents. Some belong to AI companies, some to people pretending to be AI companies, and a few have names so similar that teams block the wrong one and wonder why citations dried up. This guide sorts the real names by what they do.
One correction first. There is no bot called "ChatGPTBot." If you see that string in a blog post or a vendor deck, the author did not check. The real OpenAI names are below.
Three jobs, not one
AI bots do three different things, and policy decisions depend on which one you are looking at.
- Collecting content, often for training. Broad crawlers that gather pages at scale.
- Building a search index. Crawlers that let a product find your page when a question arrives.
- Fetching on demand. A bot that visits a page because a user, or an answer in progress, needs it right now.
The reason to separate them: blocking job 1 does not necessarily block jobs 2 and 3, and the ones tied to showing your page in an answer are usually 2 and 3.
The names Promptwatch tracks
Promptwatch's Agent Analytics lists these tracked bots:
- OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot
- Perplexity: PerplexityBot, Perplexity-User
- Google: Google-Agent, GoogleOther
- Meta: Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher
- Mistral: MistralAI-User
Read the suffixes. "User" in a name generally signals a fetch triggered by a person's request. "Search" signals an index or retrieval role. A plain name like GPTBot or ClaudeBot is the broad crawler. That pattern is a reading aid, not a guarantee, so check each vendor's own documentation before you write a rule that depends on it.
Why the distinction matters for citations
A page can be cited only if the system can reach it when it needs to. If your robots.txt or firewall blocks OAI-SearchBot and ChatGPT-User, a page may stop appearing as a source even though GPTBot is allowed or blocked on separate grounds. The reverse also holds: allowing everything does not make a thin page citable.
Promptwatch describes a crawl-to-citation path with a citation rate per page. It connects "the bot fetched this URL" to "this URL was cited," so you can see which pages convert crawls into citations and which ones are crawled and ignored.
How to read your logs, step by step
- Pick one page that matters and filter by its path. Promptwatch supports stackable path filters.
- Split by bot. You want to see search and user-fetch bots separately from the broad crawlers.
- Check status codes. Error tracking with failure explanations shows whether the bot received a 403, a 404, a timeout, or a redirect loop.
- Compare crawl counts to citations. High crawls and low citations point to content. Low crawls and no citations point to access or discovery.
- Export if needed. CSV export is available for sharing with developers.
Spoofing is real
A user agent string is just text. Anyone can claim to be GPTBot. That is why Promptwatch's crawler logs are IP-verified and filter out spoofed user agents. If you build conclusions from raw logs alone, you may count scrapers as AI bots and conclude your traffic is better than it is. Verify by IP range as published by each provider, or use a tool that does it for you.
Getting logs into one place
Crawler data has to come from somewhere upstream of your page. Promptwatch lists integrations with Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint, or the WordPress plugin. Pick the layer closest to your edge, because origin logs will miss anything a CDN served from cache.
On plan gating: crawler logs start on Professional for brand plans, and every agency plan includes them. Essential has none.
Mistakes we see most often
- Blocking by guesswork. Someone adds a wildcard rule to stop "AI scrapers" and takes out the search and user-fetch bots with it.
- Trusting robots.txt alone. A rule in robots.txt says nothing about a WAF that rate-limits unknown agents. Logs show what actually happened.
- Counting hits instead of outcomes. Thousands of crawls on a tag page do not matter if no one cites it.
- Forgetting Meta and Google. GoogleOther and the Meta bots appear in logs, and each ties to different products.
A policy template
Write a one-page bot policy with three columns: bot name, role, decision. For each row, add the reason and a review date. For a business that wants to be cited, a common stance is to allow the search and user-fetch bots, then decide on the broad crawlers based on your content licensing views. That choice is yours; we are not going to make it for you. What we will say is that writing it down beats living with accidental defaults.
Where this fits in your toolkit
A bot list is only useful when you can see the bots hit your site. Promptwatch gives you IP-verified logs, the exact bot names above, a per-page crawl-to-citation view and the errors behind failures, all in the workspace where your citation data already lives. If you are on Professional or above and have not connected a CDN yet, do that first. One week of logs will tell you more about your AI access than any robots.txt review.