AI search indexers vs user fetchers: OAI-SearchBot and ChatGPT-User explained
Why your AI crawl totals just went up while citation crawls dropped, and how to read search indexers and user fetchers as two separate buckets in your crawler data.
You open your crawler report on Wednesday and the numbers have moved. AI crawls are up. Citation crawls are down by roughly the same amount. Nothing changed on your site, and nobody touched robots.txt. So what happened?
In this case the data did not move, the labels did. On November 4, 2026, Promptwatch changed how its GEO 360 views classify AI bot traffic. Search indexers such as OAI-SearchBot, Claude-SearchBot, and Meta-WebIndexer now count as AI crawls. A citation crawl is reserved for bots that fetch a page while answering a user, like ChatGPT-User. Citation crawl totals fall by as much as AI crawl totals rise, and some page categories can shift as a result.
This post explains the two buckets, why the split is worth having, and how to read them without fooling yourself.
Two kinds of visit, two different questions
A search indexer visits your pages ahead of time so that an AI product has an index to search. Nobody is waiting on that request. It tells you whether your content is reachable and being collected.
A user fetcher visits a page because a person asked something and the assistant went to read a source in the moment. ChatGPT-User is the example the classification uses. That request is much closer to a citation, because a live answer was being assembled when it happened.
Mixing the two answers neither question well. If indexer visits sit inside "citation crawls," a page can look like it is being used in answers when it has only been collected. If fetches are buried in a general crawl total, you lose the signal that someone's question actually led to your page.
What changed in the numbers
Under the new rule, the AI crawl bucket takes in the indexers. The citation crawl bucket keeps only the user-answer fetchers. Because the same requests simply moved from one bucket to the other, the two totals shift in opposite directions by a similar amount.
Meta-WebIndexer is a good reason to pay attention. Promptwatch's Meta-WebIndexer report shows it growing from about 2% to about 38% of tracked AI crawler requests. A bot with that share of traffic will move a classification total, whichever bucket it sits in.
Page categories can also shift, and they may not all update at once. If a page changed category on your dashboard, wait for the next daily update before drawing conclusions. A one-day mismatch is not a site problem.
How to read each bucket
| Bucket | Which bots from the classification | What it tells you |
|---|---|---|
| AI crawls | OAI-SearchBot, Claude-SearchBot, Meta-WebIndexer (search indexers) | Whether your pages are reachable and collected |
| Citation crawls | ChatGPT-User (fetches while answering a user) | Whether a live answer went to read the page |
Use the first bucket for access questions. If a page you care about has no AI crawls at all, look at robots.txt, firewall rules, and errors before you rewrite anything. Agent Analytics lists error tracking with failure explanations, so a blocked fetch shows up as a reason, not just a missing row.
Use the second bucket for demand questions. A page with steady citation crawls is being read for answers. A page with plenty of indexer crawls and no citation crawls is collected but not being pulled in. That points at content or prompt fit, not access.
Do not infer more than the label says. These bots are classified by what they do in the logs. How a given engine ranks or chooses sources afterward is outside what crawler data can show, so treat the crawl-to-citation path as a diagnostic, not a guarantee.
Where to look in Promptwatch
Agent Analytics holds the IP-verified crawler logs. The verification checks each request against the operator's published IP ranges, so spoofed user agents are filtered out. That matters here, because a fake "ChatGPT-User" would otherwise inflate the bucket you care about most.
For a page-by-page view, the crawl-to-citation path in Agent Analytics shows a citation rate per page. That figure is what separates an access problem from a content problem. If crawls are healthy and the citation rate is low, the fetch happened and the page was not chosen. If crawls are missing, you start at access.
360 Insights covers the broader view. Its AI Insights tab brings crawler logs and citation clicks together with geographic and trend breakdowns, and Topic Insights shows where pages underdeliver for their topic. Search Insights lays Google Search Console and Bing Webmaster data over the same pages. If your totals jumped on November 4, this is where you check whether the shift is purely the reclassification: compare the AI crawl and citation crawl lines across the changeover and see whether the sum stays roughly flat.
A short routine for the week after the change
- Note the date. Annotate any report or slide that spans November 4, 2026, so a reader does not mistake the reclassification for a traffic event.
- Re-baseline. Start comparing AI crawls to AI crawls from the new definition onward, not to the old figures.
- Check categories after the next daily update before you act on a page that appears to have moved.
- Pick two or three key pages and compare indexer visits with user-fetch visits on each.
If your reporting is a spreadsheet of crawler hits, this is a good moment to see why the dashboard approach holds up better. Most of the tracking tools in our directory are built around prompt monitoring and do not give you crawler logs at all. Promptwatch is the one we would evaluate first for this job, because the logs, the citation data, and the visitor analytics sit in one place and the classification is maintained for you.
Next step
If you want to see how your own site splits between indexers and user fetchers, start the 7-day free trial of Promptwatch and connect your crawler logs. You can also run the free AI Visibility Report at promptwatch.com/ai-visibility-report for a quick crawler access check first. Crawler logs themselves begin on the Professional plan, at $245 a month, so check that before you plan the setup.