AI crawler logs on Shopify, Webflow and WP Engine

"I can't see server logs, I'm on Shopify." Or Webflow. Or WP Engine, or Kinsta. It is the most common reason teams give for not knowing whether GPTBot, ClaudeBot or PerplexityBot ever reach their pages. On a hosted platform you do not own the web server, so there is no log file to download and grep.

That is a real limit, and it still has a workaround. The log you want does not have to come from the origin server. It can come from whatever sits in front of it.

Why the logs matter

Analytics scripts do not see bots. A crawler does not run your JavaScript, so Google Analytics shows nothing. If you want to know whether an AI crawler fetched your new pricing page, or whether it hit errors on your product pages, you need request-level data.

That data answers practical questions:

• Did the crawler reach the page I want cited, or did it stop at the homepage?
• Is it getting errors, redirects or blocked responses?
• Which bots are visiting: training crawlers, search crawlers or the agents that fetch a page for a user?
• Did a change to robots.txt or my CDN rules cut off access by accident?

Without logs, every theory about why a page is not cited is a guess. Our guide to what each AI crawler does explains the bot names so you know what you are reading.

What you can do with no tool at all

Some checks work on any platform:

1. Read your robots.txt at yourdomain.com/robots.txt. Look for blanket disallow rules and rules naming AI bots. Hosted platforms often generate this file for you.
2. Check your CDN or firewall settings. Bot protection that blocks "unknown automation" can block AI crawlers without telling you.
3. Fetch a page as a bot. Request your page with a crawler's user agent string and see what comes back. It will not prove a real bot visited, but it shows whether you serve it a block page.
4. Look at what your host exposes. Some platforms offer basic traffic reports. They rarely separate AI bots, but it is worth a look.

These tell you whether access is possible. They do not tell you whether it is happening.

The workaround for hosted platforms

The usual approach is to put a CDN or edge layer in front of the site and collect request logs there. Cloudflare is the common choice. It can forward request logs through Logpush or run a small Worker that reports each request.

That is why the setup works on platforms where you cannot touch the server. The edge sees every request before your host does. You are not modifying Shopify or Webflow, only the DNS and edge in front of them.

There is a second route for WordPress. A plugin running inside the site can report crawler visits directly, which suits sites where adding Cloudflare is not an option. Our post on crawler log integrations across Cloudflare, CloudFront, Fastly, Vercel and Netlify compares the routes, and the Cloudflare setup walkthrough goes through the steps.

Where Promptwatch fits

Promptwatch turns those logs into something readable. Its Agent Analytics shows real-time AI crawler logs with IP verification, so spoofed user agents are filtered out. That matters because anyone can claim to be GPTBot in a header.

The changelog records a few recent changes that matter for hosted sites:

• Cloudflare Logpush is now the default crawler log setup for WP Engine, Webflow, Shopify and Kinsta sites (October 6, 2026). Kinsta support, using the same Worker deploy as WP Engine, Webflow and Shopify, was added on August 21.
• An official WordPress plugin for sites that want to report from inside WordPress, installable from the WordPress Plugin Directory or as a ZIP during CMS setup.
• Verified crawler data uses historical IP ranges from each provider. Crawler visits from ranges a provider has since retired are counted again instead of being dropped. This matters if you are reading older periods.

Once connected, you get more than a hit counter. The crawl-to-citation path shows per page whether a bot fetched it and whether it then got cited, with a citation rate. Error tracking explains failures. Path filters stack, and you can export to CSV. The tracked bots include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User, among others. Our walkthrough on reading the crawl-to-citation path page by page shows how to read it.

Plan tiers, plainly

Crawler logs start on the Professional plan at $245 a month. The Essential plan at $95 a month has none. If your only goal is to see AI bots on a hosted site, Professional is the entry point. Agency plans include crawler logs on all tiers.

I would not buy a higher plan only for this. Start where the logs begin and see whether the first week of data tells you something useful.

A setup checklist for hosted sites

1. Confirm what hosts your site: Shopify, Webflow, WP Engine, Kinsta, or something else.
2. Check robots.txt and bot protection first, so you know access is open.
3. Choose the route: Cloudflare Logpush for the four hosts above, or the WordPress plugin for WordPress.
4. Connect it and give it a day or two to collect data.
5. Look at the pages you want cited. Did a verified bot reach them? Any errors?
6. Fix access problems, then re-check a week later.

The mindset shift is that the logs live at the edge, not at the server. Once you accept that, the hosted platform stops being a blocker. If you want to compare tools that read those logs, our roundup of standalone AI crawler analytics tools covers the field, and the accessibility checker guide helps for a one-off check.

I would still begin with Promptwatch, because the value is in joining the crawl data to citations and visits, not in the log alone.