AI Evals for Product Managers: The Complete Guide

Originally published on AllthingsPM, the AI product management course and interview practice app.

An AI eval is a repeatable test that tells you whether an AI feature does its job well enough to ship, and whether a change made it better or worse. For a product manager, the core loop is short: read about 100 real outputs, name the failures from one taxonomy, turn them into a golden dataset, grade each failure with the cheapest grader that can see it (code first, then an LLM judge you have validated, then humans), and set a pass rate that blocks launch. Hamel Husain and Shreya Shankar say they spend 60 to 80 percent of their development time on error analysis and evaluation, and in our study of 604 PM job postings, 53% of AI-native roles asked for evals and measurement.

This guide walks through each step with a worked example, then covers agent evals, cost and how the topic shows up in PM interviews.

What is an AI eval, in plain terms?

An eval has three parts: a set of inputs (test cases), a definition of good for each one (a criterion), and a grader that decides pass or fail. Anthropic's engineering team defines a task as "a single test with defined inputs and success criteria" and recommends running several trials, because the same model gives different answers on different runs.

Without evals, the only signal is what OpenAI calls "vibe-based evals": someone tries five prompts, the answers look fine, and the change ships. A prompt edit that fixes tone can quietly break citation accuracy on a tenth of inputs. An eval suite turns that hidden regression into a number that moves.

Why do product managers own evals now?

Because an eval is a product decision written as a test. Kevin Weil, then OpenAI's chief product officer, said on Lenny's Podcast in April 2025 that writing evals would become a core skill for product managers.

Hiring data backs this up. In our study of 604 PM job postings from 95 companies, 53% of the 286 AI-native roles asked for evals and measurement, and the word "eval" appeared in 32% of AI-native postings against 3% of other PM postings. Some roles are entirely about evals: Abridge's Product Lead, AI/ML (Evals) asks the PM to "define the eval gates from early build through GA."

The eval lifecycle at a glance

Step	What the PM does	Output	Lesson
1. Read traces	Review about 100 real outputs, note each failure	Open notes	Read one hundred real traces
2. Name failures	Group notes into one taxonomy, count frequency and severity	Ranked failure list	Name the failure from one taxonomy
3. Build the golden set	Turn important failures into labelled cases	Golden dataset	Turn one complaint into thirty golden examples
4. Choose graders	Cheapest grader that can catch each failure	Grader per criterion	Pay for the cheapest evaluator
5. Write criteria	Binary by default; validate any LLM judge	Rubric and judge prompt	Write the criterion
6. Gate and iterate	Failing eval before the fix, suite in CI	Launch gate	Eval-driven development
7. Watch production	Online signals feed new cases	Next month's evals	The thumbs-down that becomes an eval row

The lesson links go to the matching parts of the Designing Evals for AI Products chapter in the AllthingsPM AI PM course, if you want to practise each step on real outputs.

Step 1: Read real traces before anything else

Husain and Shankar's advice is to start with error analysis, not with writing evals. Their FAQ suggests "a working pool of roughly 100 diverse traces," with at least 30 read by hand. For each, write a short free-form note on the first thing that went wrong. This is open coding: no categories yet, just observations.

Step 2: Build a failure taxonomy

Group your notes into categories (axial coding, which the FAQ calls "the most important step"), then count:

Failure	Share of 100 traces	Severity	Fix first?
Invented a refund amount	6	Critical (money, trust)	Yes
Wrong policy cited	11	High	Yes
Ignored the actual question	9	High	Yes
Too long	17	Low	Later
Wrong tone for an angry customer	4	Medium	Later

Illustrative numbers for a support-reply assistant, not real data.

The most common failure is not the most important. "Too long" shows up most, but an invented refund amount costs money and trust, so it gets a hard floor and gets fixed first. Before fixing anything, attribute each failure to a layer (prompt, retrieval, model or tool), as covered in the failure attribution lesson.

Step 3: How big should a golden dataset be?

A golden dataset is a fixed set of inputs, each labelled with what a correct output must and must not do.

Anthropic recommends starting with "20-50 simple tasks drawn from real failures." To compare versions or set a launch bar, you need more. At a 90% pass rate, the 95% margin of error is about plus or minus 6 points with 100 cases and about 3 points with 400, so a move from 88% to 91% on 100 cases is noise.

Step 4: Code, LLM judge or human?

The rule: use the cheapest grader that can reliably see the failure.

Grader	Good for	Strengths	Weaknesses
Code assertions	Format, lookups, forbidden content, tool calls, final state	"Fast, cheap, objective, reproducible" (Anthropic)	Brittle when many answers are valid
LLM-as-judge	Tone, relevance, faithfulness, policy following	Flexible and scalable	Non-deterministic, must be calibrated
Human review	Subjective or high-stakes quality, validating judges	"Gold standard quality"	"Expensive, slow, requires expert access"

For agents, Anthropic advises: "Grade what the agent produced, not the path it took."

Step 5: Write criteria, and validate the judge

Binary by default. The FAQ argues that "binary evaluations force clearer thinking and more consistent labeling." If you need nuance, write two binary criteria rather than one five-point scale.

One criterion per judge. "Is the reply good?" fails. "Does the reply state a refund amount that is not in the order record? PASS or FAIL" works.

One decider. The FAQ recommends "a single domain expert as a 'benevolent dictator'" to settle labelling disagreements. That is often the PM.

Validate the judge like a classifier. Hand-label outputs, tune the judge on a development split, then measure it on a held-out test split: its true positive rate (human-marked fails it catches) and true negative rate (human-marked passes it passes). Zheng and colleagues (NeurIPS 2023) found strong judges reached "over 80% agreement" with human preferences, but also documented position, verbosity and self-enhancement bias.

A worked judge prompt

Here is a single-criterion judge for the support assistant above, written the way a PM would hand it to an engineer:

1. Context given to the judge: the customer message, the order record, and the assistant's reply.
2. Question: "Does the reply state any refund amount, date or policy term that does not appear in the order record or the policy excerpt? Answer PASS if every stated fact is supported, FAIL otherwise."
3. Output: one word, PASS or FAIL, followed by one sentence naming the unsupported fact.
4. Validation: 100 hand-labelled replies split 50 for tuning and 50 held out. Ship the judge only if it catches most human-marked fails on the held-out half without flagging many human-marked passes.

It is written in product language, not code, which is why the PM is usually the right owner.

Offline vs online evals

Offline evals run the golden set against a build before users see it and decide whether it ships. Online evals measure live traffic: thumbs down, edits, discards, retries, escalations, and judges run on sampled traces.

Every week, pull a sample of production traces with negative signals, read them, and promote the new failure patterns into the golden set. An eval score alone is never the success metric. Pair "90% pass on answers the question" with a business outcome such as resolution rate or fewer escalations, so you know the eval measures something users care about.

Eval-driven development and the launch gate

Write the failing eval before the fix. Reproduce the reported problem as cases, confirm they fail, change the prompt, retrieval or model, and confirm they pass without breaking the regression set. Run the suite in CI.

Then pre-commit a launch gate, written before you see results:

• Hard floors on critical failures, near zero. Abridge calls these "non-negotiable floors like critical-error rates."
• Quality targets on the main criteria, for example 90% pass on "answers the actual question."
• No regression against the current version.

When offline scores and users disagree, trust users and read traces. One real interview question describes exactly this: offline evals show strong SWE-bench-style gains, but internal dogfooders say the model feels worse.

A PM's eval checklist before launch

• At least 100 real or realistic traces read and coded.
• One failure taxonomy, ranked by frequency and severity.
• A golden set with cases for every critical failure, plus a regression set.
• A grader chosen per criterion, and every LLM judge validated on held-out labels.
• A launch gate written down before the results came in.
• An online signal (thumbs down, edits, escalations) wired to feed next month's cases.

How are agent evals different?

Check the final state: did the refund actually get issued? Anthropic defines the outcome as "the final state in the environment at the end of the trial."

Measure reliability: pass@k is "at least one correct solution in k attempts," while pass^k is "the probability that all k trials succeed." Users get one attempt, so pass^k is the product metric. The τ-bench paper found even GPT-4o succeeded on fewer than 50% of tasks, with pass^8 below 25% in its retail domain.

How much do evals cost?

Human time is the biggest cost: a few focused days to set up, then about 30 minutes a week, per Husain and Shankar on Lenny's Podcast. Judge compute is small by comparison. At Anthropic's September 2026 prices, Claude Haiku 4.5 costs $1 per million input tokens and $5 per million output tokens, with 50% off through the Batch API.

Do eval questions come up in PM interviews?

Constantly, at AI companies. In the AllthingsPM question bank of 4,122 real PM interview questions, 95 ask directly about evals, golden sets, LLM judges or hallucination, and each has its own page with an answer guide.

Two to practise:

• How would you design an evaluation framework to know whether a new Claude model is genuinely better at coding? (Anthropic)
• Design an evaluation framework for North agents (Cohere)

The company pages for Scale AI, Anthropic and OpenAI list the rest. A strong answer follows the lifecycle above: clarify what the feature is for, name the likely failures, describe the golden set and graders, set a launch gate, and explain how production signals feed back.

To rehearse out loud, paste an evals job description into a JD mock interview, or start from a live posting such as Product Manager, Safeguards (Generalist) at Anthropic. The AI interviewer asks follow-ups and scores each answer, typed or spoken. For a broader measurement primer, our How to Measure Anything summary is a quick read.

Where to learn AI evals

Option	Teaches the eval method	Real eval interview questions	Mock built from an evals JD	Price (Nov 2026)
AllthingsPM	Yes, graded course chapter	95, each with an answer guide	Yes, text or voice	$12/month or $108/year list; Chapter 1 free
Husain and Shankar's free FAQ	Yes, deep and engineering-led	Not the focus	No	Free
Husain and Shankar's AI Evals for Engineers & PMs (Maven)	Yes, 8 weeks, 33 live sessions	Not the focus	No	$4,500 per cohort; next Jan 30 to Mar 27, 2027
Generic AI chat tool	Explains on request	No curated bank	No scoring against a JD	Varies

If you are an engineer building eval infrastructure, Husain and Shankar's course goes deepest. If you are a PM who needs the method plus interview practice, the AllthingsPM AI PM course covers both: it is built from 604 real PM job postings, updated weekly, and its evals chapter ends in an integration case where you build the eval suite for your own feature.

Frequently asked questions

What are AI evals for product managers?

AI evals are repeatable tests that measure whether an AI feature produces good enough outputs, using test inputs, a definition of good for each, and a grader. For PMs, they replace "it seems to work" with a pass rate you can gate a launch on. The PM usually owns the definition of good and the launch threshold.

Do product managers write evals or do engineers?

Both. The PM typically reads traces, writes the failure taxonomy and criteria, and sets the launch gate; engineers often wire up code checks and CI.

How many examples do I need in a golden dataset?

Start with 20 to 50 cases from real failures, as Anthropic recommends. To compare versions or set a launch bar, you need more: at a 90% pass rate, 100 cases gives a margin of error of about 6 points and 400 cases about 3 points.

What is LLM-as-a-judge, and can I trust it?

It is using a language model to grade another model's output against a written criterion. Strong judges reached over 80% agreement with human preferences in a NeurIPS 2023 study, but show position, verbosity and self-enhancement biases. Trust one only after measuring its true positive and true negative rates on held-out human labels.

How do I prepare for eval questions in PM interviews?

Expect to design an eval framework out loud, including golden set, graders and ship thresholds. Practise with real questions from AI companies, then rehearse against a real evals job description so the follow-ups feel familiar.

Next step

Open the Designing Evals for AI Products chapter, read your first 100 traces on a feature you know, then try one JD mock for an evals role to hear how your answer holds up.

Sources

1. Hamel Husain, "Your AI Product Needs Evals", 29 March 2024: https://hamel.dev/blog/posts/evals/
2. Hamel Husain and Shreya Shankar, "AI Evals: Everything You Need to Know" (evals FAQ), accessed September 2026: https://hamel.dev/blog/posts/evals-faq/
3. Lenny's Podcast, "Why AI evals are the hottest new skill for product builders", Hamel Husain and Shreya Shankar, 25 September 2025: https://www.lennysnewsletter.com/p/why-ai-evals-are-the-hottest-new-skill
4. Lenny's Newsletter, "OpenAI's CPO on how AI changes must-have skills, moats, coding, startup playbooks, more" (Kevin Weil), 10 April 2025: https://www.lennysnewsletter.com/p/kevin-weil-open-ai and Lenny Rachitsky on X quoting the episode: https://x.com/lennysan/status/1909636749103599729
5. Anthropic Engineering, "Demystifying evals for AI agents", Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares and Jiri De Jonghe, 9 January 2026: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
6. OpenAI, "Evaluation best practices", API documentation, accessed September 2026: https://developers.openai.com/api/docs/guides/evaluation-best-practices
7. Lianmin Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023 Datasets and Benchmarks: https://arxiv.org/abs/2306.05685
8. Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya Parameswaran and Ian Arawjo, "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences", 2024: https://arxiv.org/abs/2404.12272
9. Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains", 2024: https://arxiv.org/abs/2406.12045
10. Anthropic, Claude API pricing (model and Batch API prices), checked September 2026: https://platform.claude.com/docs/en/about-claude/pricing
11. Abridge, "Product Lead, AI/ML (Evals)" job posting: https://jobs.ashbyhq.com/abridge/9c7ba6c3-7744-48b8-a5b3-dab55c22e4b3 (also in our jobs catalog)
12. AllthingsPM, "State of AI PM Hiring 2026": 604 PM postings from 95 companies, read 6 September 2026: https://www.allthingspm.app/blog/state-of-ai-pm-hiring-2026
13. AllthingsPM question bank: 95 of 4,122 questions match eval, evaluation, golden set, LLM judge or hallucination, September 2026 (chart above): https://www.allthingspm.app/question-bank
14. Maven, "AI Evals for Engineers & PMs" (Hamel Husain and Shreya Shankar), price, format and cohorts checked November 2026: https://maven.com/parlance-labs/evals
15. Douglas W. Hubbard, How to Measure Anything (summary): https://www.allthingspm.app/book-summaries/how-to-measure-anything