Which AI crawlers visit the open web? The weekly provider mix, 2026
Promptwatch's weekly crawler data shows OpenAI at 79.8% of verified AI crawler requests, down from 94.8% in June; here is how to compare that mix with your own logs.
OpenAI accounted for 79.8% of verified AI crawler requests in the week of August 31 to September 6, 2026, down from 94.8% in the week of June 8 to June 14. The rest was split among Anthropic, Google, Perplexity and Mistral. That is the headline from Promptwatch's AI crawler traffic report, first published September 9, 2026. The page now updates weekly, and the figures here are for that last week of August into early September.
OpenAI is still the biggest single provider by a wide margin. The share has also fallen by about fifteen points in under three months, which says other providers are taking a bigger slice of what the open web sees.
What the weekly mix measures
The definitions decide how safely you can use these numbers, so it is worth being exact.
Verified requests only. Promptwatch counts a request only when its IP address matches the operator's published ranges. Requests that merely claim a crawler's user agent are excluded, which matters because user agents are trivial to spoof.
Shares, not volumes. Each week's shares total 100%. A provider's slice can grow without its absolute request count growing, if the total shrinks or another provider falls. A rising Anthropic or Perplexity line does not on its own mean more requests.
Weekly, Monday to Sunday UTC. The numbers are drawn from raw server and CDN logs connected to Promptwatch's crawler analytics.
Provider totals hide the job. An OpenAI share lumps together crawlers that collect training data and fetchers that run at answer time. The report's chart can group by crawler, and that is where the useful distinction lives. A request from a training crawler and a request fetched because a user asked a question are different events for a site owner.
What it does not tell you
It describes traffic across sites connected to Promptwatch, not the whole web, and not your site. It says nothing about which crawlers lead to citations, since a visit is not a citation. It also gives no verdict on whether any provider's share should be higher or lower, and the change from June to September is two data points weeks apart, not a trend line we should extend.
There is a separate Promptwatch changelog item from September 9 about a 12-month AI crawler benchmark by industry. We have not seen figures from it and do not quote any here.
What the report suggests, and how to act on it
The report's advice is to check robots.txt, CDN and WAF rules for every provider, to compare your own logs with the published mix, and to look per crawler to work out whether OpenAI traffic is training, answer-time fetching, or both. Here is how that works in practice.
- List the providers in the report. OpenAI, Anthropic, Google, Perplexity and Mistral.
- Read your robots.txt line by line. Look for blanket disallows, old rules that name a bot you now want, and wildcard rules that catch everything. The robots.txt guide for AI search walks through the common mistakes.
- Check the layers above robots.txt. A CDN or WAF can block a crawler that your robots.txt welcomes. This is the usual reason a site is "allowed" on paper and unreachable in practice.
- Compare your mix with the report. Where a provider is large in the report and near zero in your logs, the report's reading is that you may be blocked or unreachable for it. That is a lead to investigate, not a diagnosis. It could also be that your site simply gets less attention from that provider.
- Split by crawler. If OpenAI is most of your traffic, find out which of its bots are making the requests before deciding anything.
Our post on how to check if AI bots can crawl my website covers the quick checks you can do without logs, and Meta-WebIndexer's crawler share is a case of one crawler's share moving fast in a short period.
The measurement step: crawler logs you can trust
Comparing to a report only works if your own data is verified the same way. When we assess crawler analytics in this directory, the first question is whether requests are IP verified or simply trusted by user agent. Promptwatch is where we start, because its Agent Analytics crawler logs are real-time and IP verified, with spoofed user agents filtered out, the same method the report uses.
It tracks named bots including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher and MistralAI-User. Logging by named bot, instead of by provider, is what lets you start on the report's training-versus-answer-time question. Check each vendor's own bot documentation to see which job each bot does, since the logs show who visited and not why. The crawl-to-citation path shows a citation rate per page, error tracking explains failed fetches, and path filters and CSV export let you slice by section.
Plan details matter here. Crawler logs start on Professional at $245 a month. Essential, at $95, has none, so a budget buyer who wants this data should plan for Professional. Setup runs through Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint or the WordPress plugin. Our guide to connecting Cloudflare crawler logs to Promptwatch covers the most common route. For a free first look, the Agentic Readiness Check tests your robots.txt against nine AI crawlers without a login.
The takeaway
OpenAI still dominates verified AI crawler traffic at 79.8%, but its share has dropped from 94.8% over the summer, and Anthropic, Google, Perplexity and Mistral make up the rest. The report is a benchmark, not a verdict on your site. Read your robots.txt, check the CDN layer, and then compare your own IP-verified logs with the mix, crawler by crawler.