Product managers must write AI evals — not just requirements

A product requirement document describes what a feature should do. An eval set proves whether it actually does. For deterministic software, the gap between intent and verification is narrow — unit tests, integration tests, and acceptance criteria close it. For AI products, the gap is a chasm. The output is probabilistic. A prompt change can improve one dimension and degrade another. A model upgrade can shift tone, accuracy, and hallucination rates simultaneously without any code change triggering a test failure. The only mechanism that bridges this gap is a structured eval set — and the person who understands what "good" means for the user is the product manager, not the engineer.

The prompt is temporary. The eval is permanent. Prompts get rewritten, models get swapped, retrieval pipelines get restructured. The eval set survives all of these changes because it encodes what the product must do, not how it does it. A PM who writes a paragraph in a spec and never sees the scoring rubric has handed the most consequential product decision — the definition of quality — to whoever last edited a YAML file in the eval pipeline.

AI product quality cannot rely on vibes

Traditional product quality has observable signals: the button works, the page loads in under 200ms, the data saves correctly. AI product quality is invisible until measured. An AI-generated summary might omit a critical detail. A recommendation might be plausible but wrong. A routing decision might escalate to a human when the system could have resolved it — or worse, resolve it when escalation was required.

Teams that ship AI features without structured evals rely on what the field calls "vibe checks": a human reads a handful of outputs, decides they look right, and approves the change. This works when the product has ten users. It collapses at scale because vibes do not catch regressions, do not generalize to the long tail of inputs, and do not produce evidence for rollback decisions.

The operational consequence: a model upgrade ships, tone degrades on 15% of support responses, and nobody notices until NPS drops three weeks later. By then, the damage is done, the old model is deprecated, and the team has no regression set to prevent the failure from recurring. A structured eval would have caught the degradation before deploy by scoring tone as an independent dimension against a baseline dataset.

Requirements become examples in AI products

A traditional requirement reads: "The summary must capture all action items from the meeting." For deterministic software, this translates directly into acceptance criteria. For AI software, it translates into nothing actionable — because "capture all action items" has no computable definition until someone provides examples of meetings, the action items they contain, and the criteria for scoring completeness.

AI product evals are requirements expressed as graded examples:

• Input: a transcript of a 30-minute support call.
• Expected behavior: the summary includes the customer's stated problem, the resolution offered, and any follow-up commitments.
• Scoring criteria: accuracy (did it state the problem correctly?), completeness (did it include all commitments?), tone (does it read as professional and neutral?), and failure behavior (what does it do when the transcript is ambiguous?).

Each criterion becomes a separate scorer. Lumping "quality" into one dimension makes regressions invisible — a tone improvement can mask an accuracy degradation when they're averaged into a single number. The PM defines the dimensions. Engineering automates the measurement.

The practical workflow: a PM reviews five real user scenarios from the past week, annotates what "good" looks like for each, defines the scoring criteria in plain language, and hands those to engineering as the eval contract. This is not extra work on top of requirements — it is the requirement, expressed in the only form that the AI system can be evaluated against.

PMs own the definition of good

The scoring rubric is the real product spec for an AI feature. It encodes what matters, how much it matters, and where the boundaries lie between acceptable and unacceptable. The PM is the only person on the team with the cross-functional context to make these decisions: they know the user's language, the business constraints, the edge cases that generate support tickets, and the failure modes that erode trust.

What PM ownership of AI product evals looks like in practice:

• Writing the rubric criteria — not the automation code, but the plain-language definition of what a score of 5 versus a score of 1 means for each dimension.
• Providing anchor examples — real production inputs paired with outputs the PM has graded as excellent, acceptable, and failing. These anchors calibrate the automated judge.
• Reviewing judge agreement — weekly comparison of automated scores against a sample of PM-graded outputs. If the judge and the PM disagree on more than 15% of cases, the rubric is too vague or the judge has drifted.
• Maintaining the eval as a living contract — when the product direction changes, the eval changes first. New failure modes from production logs become new regression examples.

Engineering owns the infrastructure: the CI gates that block deploys when scores drop, the production sampling pipeline, the LLM-as-judge orchestration, and the observability dashboards. But engineering cannot define what "good" means for the user. That is product work.

Engineering owns automation, instrumentation, and deployment gates

The division of responsibility is clean. The PM defines what to measure. Engineering builds the system that measures it continuously, at scale, and with enough precision to block regressions before they reach production.

The engineering stack for AI product evals includes:

• Deterministic checks as a first layer — JSON schema validation, length bounds, refusal detection, forbidden-content filters. These are binary, fast, and catch failures that no LLM judge should waste inference on.
• LLM-as-judge for nuanced quality — an automated evaluator that scores each output against the PM's rubric. The judge runs on every CI build against a versioned dataset and on a sample of production traffic to catch drift.
• Human review as calibration — a weekly sample of 50 production traces scored by the PM or a domain expert, compared against the automated judge. Cohen's Kappa above 0.6 indicates acceptable alignment; below that threshold, the rubric needs revision.
• CI gating — a deploy fails if any rubric dimension drops more than 2 points from the trailing baseline on the regression set. The threshold is per-route and per-rubric, so a prompt change in the support agent does not trigger a full re-evaluation of the sales assistant.
• Production observability — the same rubrics that run in CI run on sampled live traffic to catch failures the offline dataset does not represent. Alert when a dimension sustains a drop over a configurable window.

The eval contract — the tuple of (judge_model_id, rubric_version, prompt_template_hash) — is pinned and versioned. Changing any element is a deliberate act that requires PM review, not a side effect of a vendor model upgrade.

What makes a useful AI product eval?

The difference between a useful eval and a checkbox exercise is specificity. Useful evals catch real regressions. Useless evals pass everything and lull the team into false confidence.

How should PMs create eval examples from real usage?

Start from production failures, not abstract scenarios. Pull the last 30 days of user complaints, support escalations, and negative feedback. Each recurring complaint becomes a criterion: "replies sound robotic" translates to a tone eval; "wrong dates in the summary" becomes a factual accuracy eval; "missed the customer's second question" becomes a completeness eval. Five well-chosen production examples with PM-graded outputs are worth more than fifty synthetic test cases that never represent real distribution.

How many dimensions should an eval rubric have?

Score one dimension per evaluator. Three to five orthogonal criteria cover most AI features: accuracy, completeness, tone, safety, and task completion. The temptation is to add more — resist it. Each additional dimension requires its own anchor examples, its own calibration cycle, and its own baseline tracking. Start with three dimensions where the team has strong PM-graded examples, then add dimensions only when a production failure reveals a gap the existing criteria do not catch.

When should evals block a deployment?

Gate on regression, not on absolute score. A prompt change that drops faithfulness by 3 points on the regression set — even if the absolute score is still above 0.7 — represents a degradation the team should investigate before shipping. Absolute thresholds are useful as floors (no deploy if accuracy drops below 0.6), but relative regression from the trailing baseline catches slow drift that absolute floors miss. The gating logic lives in CI and uses the PM-defined thresholds per dimension.

The opposing view: evals are an engineering responsibility

A common argument holds that evals are testing — and testing belongs to engineering. PMs define what to build; engineers define how to verify it works. This separation of concerns has served software well for decades, and extending PM responsibility into evaluation infrastructure risks role confusion, slower iteration, and rubric quality that suffers from non-technical ownership.

This argument applies to deterministic software where the spec unambiguously describes the expected behavior. It breaks for AI products because the spec is the eval. The traditional spec says "the button saves the record"; there is no ambiguity in verification. The AI spec says "the summary captures all action items" — and without graded examples showing what "captures" means at the boundary, no amount of engineering infrastructure can determine whether the system works. The PM is not testing the system. The PM is defining the system's quality contract in the only form that produces computable truth. Engineering automates that contract. The operational playbook for AI agents in production depends on this contract existing before deployment, not after the first user complaint.

Key takeaways

• For AI products, the eval set is the product requirement that does not rot — it keeps running after launch and catches regressions that prose specs cannot.
• PMs own the rubric: the plain-language definition of what a score of 5 versus 1 means for each quality dimension.
• Score one dimension per evaluator. Lumping quality into a single number hides whether a regression comes from accuracy, tone, or completeness.
• Start from production failures, not synthetic scenarios. Five real user examples with PM-graded outputs outweigh fifty abstract test cases.
• Pin the eval contract — (judge_model_id, rubric_version, prompt_template_hash) — and require PM review for any change.
• Gate deploys on regression from trailing baseline, not just absolute score floors.

Conclusion

The job of an AI product manager has absorbed a new core responsibility: defining quality in a form the system can evaluate against itself. This is not a testing task delegated to QA. It is the act of specifying what the product must do — expressed not as paragraphs that gather dust in a wiki, but as scored examples that run on every build, every deploy, and every production sample. The PM who has never opened the eval rubric is not leading the product. They are hoping the product works. And in AI, hope is the mechanism by which regressions reach users undetected.