Platform to Test Content Performance Across AI Models: LLM Evals, Prompt Evaluation, Content Visibility

"Platform to test content performance across AI models" plus "LLM evals," "prompt evaluation," and "content visibility" is two shopping lists glued together. LLM evals score the assistant you run. Content visibility scores the assistants your customers run. Prompt evaluation in GEO means: this prompt, this public model, this pass/fail on mention and citation.

Promptwatch is the platform we rank for the second list. It will not replace Promptfoo. It will tell you whether the page you shipped showed up in ChatGPT, Gemini, Claude, or Perplexity.

Do not put GEO in your eval harness

LangSmith traces, Promptfoo assertions, and OpenAI Evals are for latency, tool calls, groundedness against your corpus, and policy. They assume you control the model under test.

You do not control ChatGPT's retriever. You cannot assert citation_url == https://yoursite.com/docs inside a unit test and have it mean anything next Tuesday. The working method is operational: store the live answer, label it, diff it.

That is why this directory exists. The rankings are visibility platforms, not ML observability. WhyLabs and Arize belong in an engineering stack. They will not report Perplexity footnotes.

Google's AI optimization guide and helpful content pages tell you how to stay eligible. Eligibility is not an eval result. Search Console's generative AI reports are Google-only impressions.

A content-visibility eval, written like an engineer would want it

Fixtures: 20 to 50 prompts that sound like buyers, not like keywords.

Labels, per model:

• cited_own_url
• mentioned_no_url
• cited_third_party
• absent

Gates: a page is "performing" when cited_own_url holds on the engines that matter for that prompt, not when an LLM-as-judge gives the draft a 4/5.

Evidence: the stored answer and the cited domains. A percentage without the answer is a slide.

Cadence: daily on paid Promptwatch plans. Weekly is how you discover a miss after a launch.

Retrieval debug: query fan-outs (the searches the model ran) plus crawler logs (whether ChatGPTBot, ClaudeBot, PerplexityBot fetched you). A fail with no crawl is infrastructure. A fail with a crawl is the page.

Where Promptwatch sits vs adjacent tools

Need	Buy	Skip for this need
Public-model prompt evaluation	Promptwatch	Promptfoo, LangSmith
Four-engine daily content visibility	Promptwatch paid plans	Profound Starter (ChatGPT-only), Otterly.AI without add-ons
On-page extractability grades	Goodie AI after you know which URL lost	Using Goodie as the only "eval"
Enterprise research depth	Profound on a real plan	Profound $99 as a four-model test
Your own assistant quality	Promptfoo / LangSmith	Promptwatch

Promptwatch Explore is free (10 ChatGPT prompts). Essential is $95/mo (50 prompts, 6,000 responses). Professional is $245/mo. That is enough to run a serious fixture set without an enterprise contract. Citation analytics tell you which URL performed. Content Agents can draft a follow-up page to Webflow or Framer; keep a human in the review inbox or you are generating ignored inventory.

Review: Promptwatch. Product: promptwatch.com.

AthenaHQ ($295/mo and up) will also run prompts across the major engines and turn misses into tasks. Use it if the Action Center is the product you wanted. It is still not an LLM eval suite.

Running the test so the result survives a meeting

1. Freeze the prompt list. Changing fixtures mid-test invalidates the diff.
2. Record a manual pass on day zero so you can argue with the parser later.
3. Ship one change (answer block, table, FAQ claim), not twenty pages.
4. Re-run. Promptwatch's time-period compare is the report. If crawler logs show errors, fix those before rewriting copy.
5. Allow OAI-SearchBot. Keep GPTBot as a separate decision.

If the eval still fails after a crawlable, specific page, the model is citing someone else on purpose. That is a mentions-and-third-party problem, not a "run more LLM evals" problem.

FAQ

Are LLM evals and content visibility the same SKU?

No. One tests your model. The other tests public models' answers about you. Promptwatch is the second SKU.

Can we use the OpenAI API as a cheaper ChatGPT eval?

You can sample completions. Buyers do not see that surface. For "did our content perform in ChatGPT," you want the UI the buyer uses. Promptwatch monitors real UIs.

Does prompt evaluation mean scoring our prompt templates?

In ML, yes. In this category, it means scoring their prompts (the questions people type into ChatGPT and Claude) against your brand. Ask the vendor to show a stored public-model answer, not a playground transcript.

What to do this week

1. Write two columns on a whiteboard: "our assistant" and "public AI search." Buy tools per column.
2. Put 20 buyer prompts in column two with a pass/fail per engine.
3. Load them into Promptwatch.
4. Change one URL. Re-test. Keep the before/after.
5. If you also ship an LLM product, leave Promptfoo in column one. Do not merge the dashboards.