DEV Community

Cover image for Acceptance Checklist for AI-Generated Code: A PM's Review Guide
Fathy Shalaby
Fathy Shalaby

Posted on Originally published at specky.space

Acceptance Checklist for AI-Generated Code: A PM's Review Guide

Building got cheap; accepting the result did not. The four-rung Acceptance Ladder helps PMs review agent-built changes against the decision, the brief, the edge cases and the outcome.

An acceptance checklist for AI-generated code is the short list of checks a product manager runs before an agent-built change counts as done. Engineers review how the code is written; the PM checks whether it is the right change. We use four rungs, in order: trace it to a decision, compare it with the brief, probe the edge cases, and define how you will know it worked.

Key takeaways

  • Anthropic describes software development as shifting "from writing code to orchestrating agents that write code" (Anthropic, 2026). The scarce skill moves to the review seat.
  • 66% of developers say their biggest frustration with AI tools is solutions that are nearly right but not quite (Stack Overflow, 2025). Nearly-right changes pass a "looks good" review.
  • Fluent output distorts judgement: in one randomised trial, developers believed AI had made them 20% faster when it had made them 19% slower (METR, 2025).
  • Agents tend to over-build. The surplus pass (listing behaviour nobody asked for) is the check most teams skip.
  • The checklist does not replace code or security review: 45% of AI-generated code samples in one 2025 test introduced known vulnerabilities (Veracode, 2025).

Why is "looks good" the wrong bar for AI-generated code?

Most teams review agent output the way they reviewed a human pull request: skim the diff, click around the preview, approve. That works when a teammate carries context from the planning meeting and the customer calls. It fails when an agent carries only what the brief encoded.

If the brief was vague, the agent fills the gap with the most statistically plausible behaviour, and plausible isn't the same as correct. The change compiles, the preview renders, the happy path works, and the PM approves it. Two weeks later someone notices it quietly dropped a constraint nobody wrote down, or solved an adjacent problem instead of the one in the brief. Nothing was wrong with the code. The acceptance step simply never asked the right questions.

Three things make agent output harder to judge than human output:

  • Volume. Cheap generation means more candidate changes per week, so review time becomes the bottleneck.
  • Confidence. Agents describe their own work fluently. A tidy summary isn't evidence. METR's 2025 trial with 16 experienced developers and 246 real issues found that developers expected AI to speed them up by 24% and, after the work, still believed it had sped them up by 20%, while the measured result was 19% slower (METR, 2025). METR now says those early-2025 results are out of date, and its February 2026 follow-up found small estimated speedups that it calls an unreliable signal (METR, 2026). The lasting lesson is the perception gap, not the speed number.
  • Missing intent trail. Without a link from the change back to the decision it implements, you can't tell "built as specified" from "built as inferred".

The delivery data points the same way. Google's DORA 2024 research found that a 25% increase in AI adoption was associated with 7.5% better documentation quality, 3.4% better code quality and 3.1% faster code review. Yet as adoption rose, the researchers also estimated a 1.5% drop in delivery throughput and a 7.2% drop in delivery stability (DORA, 2024). The 2025 report sums it up: AI amplifies an organisation's existing strengths and weaknesses (DORA, 2025). A weak acceptance step is one of the weaknesses it amplifies.

What should a PM check, and what should an engineer check?

Split the review. Google's engineering practices list what a code reviewer looks for: design, functionality, complexity, tests, naming, comments, style, consistency, documentation and context (Google). Most of that is engineering work. One question on the list belongs to product: "Is what the developer intended good for the users of this code?"

Question Who owns it Where it's answered
Is the code well designed, readable and tested? Engineer Code review
Is it secure, and does it handle permissions correctly? Engineer (with security tooling) Code review, scanners
Which decision or evidence does this change implement? PM Acceptance Ladder, rung 1
Did it build what the brief said, and only that? PM Rung 2
Does it behave defensibly on empty, error, boundary, permission and repeat cases? PM and engineer together Rung 3
How and when will we know it worked for users? PM Rung 4

On a solo-founder team, one person wears both hats. Do the two reviews as separate passes anyway: reading code for correctness and reading behaviour for intent use different attention.

The Acceptance Ladder: four checks before an agent-built change is done

Each rung asks one question. You only climb to the next rung when the current one passes. If a change stalls on rung one, you don't spend time on rung four.

Rung 1: Traceability. Which decision is this implementing?

Before reading any code, answer one question: what decision or evidence does this change trace back to?

A change should point to a specific item: a decision record, a customer insight, an opportunity, an experiment. If you can't name it in one sentence, stop. Either the work was never justified, or the justification lives in someone's head and the agent never saw it.

In Specky, this is what the Product Graph is for. Decisions, insights, and tickets are linked nodes, so an agent run can pull the surrounding context before it plans anything. The reviewer's first check is simply whether that link exists and whether the cited evidence says what the change claims it says.

If the brief itself came from an AI draft, check its evidence before checking the code against it. Our Claim Trace for AI-written specs takes about 15 minutes.

Pass condition: one decision or insight, named, with its evidence reachable in two clicks.

Rung 2: Contract. Did it build what the brief said, and only that?

Now compare the change to the written brief, line by line. Not "does it feel right", but a literal diff between acceptance criteria and observed behaviour.

Do two passes:

  1. Coverage pass. For every acceptance criterion in the brief, find the behaviour that satisfies it. Mark each as met, partially met, or missing.
  2. Surplus pass. List behaviours the change introduces that no criterion asked for. Agents often add helpful extras: a new setting, an extra column, a fallback path. Each one is unreviewed scope. Keep it or cut it deliberately.

The surplus pass is the one teams skip, and it's where agent work differs most from human work. In our experience a person tends to under-build against a brief, and an agent tends to over-build.

Pass condition: every criterion marked met, every surplus item explicitly kept or removed.

Rung 3: Edges. What happens when the input is wrong, empty or hostile?

Agents are strong on the happy path because that's what prompts describe. Rung three goes where prompts rarely do. Write down five situations and try each:

  • Empty state: no data yet.
  • Error state: the upstream call fails.
  • Boundary: the largest and smallest realistic input.
  • Permission: a user who shouldn't see this tries to.
  • Repeat: the action runs twice in a row.

You don't need exhaustive QA. You need the five situations a real customer will hit in the first week. If the brief didn't specify behaviour for them, that's a brief defect, and the fix goes into the brief, not just the code.

The permission check matters more with AI-generated code. Veracode tested more than 100 language models in 2025 and found that 45% of generated code samples failed security tests, with Java at a 72% failure rate and 86% of relevant samples failing to defend against cross-site scripting (Veracode, 2025). A PM can't audit that, but can insist that "a user without access tries to see this" is a written, tested criterion and that security tooling runs on every agent change.

Pass condition: all five situations behave in a way you'd defend to a customer.

Rung 4: Evidence of outcome. How will we know it worked?

A change can pass the first three rungs and still be useless. The last rung asks what you'll measure and when you'll look.

Write the check before shipping: the metric or signal, the expected direction, and a date. "We'll see how it goes" isn't a check. This is also where hype gets filtered out: a feature that ships fast but has no outcome check is a guess with a deploy date.

Pass condition: a named signal, an expected direction, and a calendar date to read it.

Copyable acceptance checklist for AI-generated code

Paste this into the pull request template or the ticket. One block per agent-built change.

ACCEPTANCE CHECKLIST: [change / PR link]
Reviewer: [name]   Date: [date]

RUNG 1 TRACE
[ ] Decision or insight this implements: [link]
[ ] Evidence behind it is reachable in two clicks and says what the brief claims

RUNG 2 CONTRACT
Coverage (one line per acceptance criterion):
  [criterion 1]  met / partial / missing
  [criterion 2]  met / partial / missing
Surplus (behaviour no criterion asked for):
  [item]  keep (criterion added) / cut (filed separately)

RUNG 3 EDGES
[ ] Empty state      [ ] Error state      [ ] Boundary input
[ ] Permission       [ ] Repeat action
Brief defects found (add to brief, not just code): [list]

RUNG 4 OUTCOME
Signal: [metric]   Expected direction: [up/down]   Read on: [date]

ENGINEERING (not the PM's call, but confirm it happened)
[ ] Code review done   [ ] Tests added/updated   [ ] Security checks ran

RESULT: accept / send back with brief changes
Enter fullscreen mode Exit fullscreen mode

What does the checklist look like on a real change?

Suppose your team asked an agent to add a "bulk archive" action to a feedback inbox.

Rung 1. The brief links to an insight: support spends time clearing stale items, and three customer conversations mention it. The evidence is reachable. Pass.

Rung 2. The coverage pass finds all four criteria met. The surplus pass finds the agent also added an "undo" toast and a new "archived" filter tab. The undo is worth keeping and gets a criterion retroactively. The filter tab changes navigation and wasn't asked for, so it's cut and filed as a separate opportunity. Pass after one edit.

Rung 3. Empty state shows a blank panel with no explanation. Fails. The fix goes into the brief as a missing criterion, then back to the agent.

Rung 4. The signal is median time to clear the inbox, expected to drop, read in two weeks.

The review took one fail and two edits, caught before release. A "looks good" review would have shipped the blank panel and the unrequested navigation tab together.

Filled into the checklist, the result line reads: 4 of 4 criteria met, 2 surplus items (1 kept, 1 cut), 1 of 5 edge cases failed, outcome read scheduled for day 14. Send back with one brief change.

How do you write acceptance criteria an agent can't misread?

The ladder is only as good as the criteria in rung two. Atlassian defines acceptance criteria as the conditions a piece of work must satisfy to be complete, and lists the traits of good ones: clear, testable, focused on the outcome rather than the method, measurable where possible, and independent of each other (Atlassian). For agents, testable matters most: a criterion an agent can't verify is a criterion it will interpret.

Given-When-Then, developed by Daniel Terhorst-North and Chris Matts as part of behaviour-driven development, is a good shape for this (Fowler, 2013). Given is the state before, When is the action, Then is the change you expect.

Vague (agent will guess) Testable (agent can check)
Archiving works well Given 3 selected items, when the user clicks Archive, then all 3 disappear from the inbox list within 1 second
Handle empty states Given no items in the inbox, when the page loads, then show "Nothing to review" and a link to settings
Only admins can archive Given a user with the Viewer role, when they open the inbox, then the Archive button is not shown and the API returns 403
Make it fast Given 500 items, when the user selects all and archives, then the action completes in under 3 seconds
Don't break anything Given an archived item, when the user opens its direct link, then it opens read-only with an "Archived" label

If you are breaking a larger spec into tickets, our guide to breaking a PRD into user stories shows how to attach criteria like these to each one. For the brief itself, the AI coding agent PRD template has a section for them.

Make the ladder automatic where you can

The ladder works best when part of it isn't manual. Two practical moves:

Validate the plan before the agent writes code. Specky exposes this through its MCP server for coding agents: an agent calls validate_agent_plan against your engineering standards before it starts, and validate_agent_change before it reports success. That moves rung two earlier, so the plan gets checked against the rules while it's still cheap to change.

Keep the evidence attached. When the decision, the brief, the agent run, and the outcome check are all nodes in one graph, rung one stops being a hunt. The reviewer opens the change and the reason is already beside it.

Running the ladder in a small team

You don't need a review board. For a solo founder or a three-to-ten person team, the ladder is a five-minute habit per change, and the order matters more than the ceremony.

Start by writing acceptance criteria as short, checkable sentences in the brief itself, such as "archived items disappear from the inbox list" rather than "archiving works well." Sentences like that are what make the coverage pass mechanical instead of a matter of taste. Next, keep a standing list of your five edge situations so rung three isn't reinvented each time. Finally, make the outcome check part of the definition of done: if the change has no signal and no date, it isn't ready to merge.

When a change fails a rung, resist the urge to patch it in the review thread. Send it back with the missing criterion added to the brief. The brief is the durable artefact; the thread disappears. Over a month, your briefs get sharper and your failure rate on rungs two and three drops, which is the real payoff.

What to measure on your own review process

Treat the ladder as a process you can inspect:

  • Rung-fail distribution. If most failures land on rung two, your briefs are the problem. If they land on rung three, your prompts skip edge cases.
  • Surplus rate. How often does an agent add unrequested scope? A rising rate means your briefs leave too much room.
  • Outcome-check completion. How many shipped changes got their rung-four read on the date you set? This is the number that shows whether you're learning or only shipping.

You don't need a dashboard for this. A table with one row per reviewed change is enough for the first month.

When doesn't this checklist work?

  • It is not a code or security review. The ladder checks intent and behaviour. With 39% of developers reporting little or no trust in AI-generated code in DORA's 2024 survey, engineering review and automated checks stay mandatory (DORA, 2024).
  • It is overkill for trivial changes. A copy fix or a colour change needs rung two at most. Save the full ladder for anything a customer will use or that touches data or permissions.
  • It can't rescue a bad decision. Rung one checks that a decision exists, not that it was right. If the underlying problem is unclear, go back to a problem statement before reviewing more code.
  • It depends on the outcome read actually happening. For AI features specifically, where the output itself varies, pair rung four with an AI evaluation framework rather than a single metric.

Frequently asked questions

What is an acceptance checklist for AI-generated code?

It is a short list of product checks run before an agent-built change counts as done: which decision it implements, whether it matches the brief and nothing more, how it behaves in five edge situations, and how you will know it worked. It sits alongside code review, not in place of it.

Should product managers review code written by AI agents?

PMs should review behaviour, not code style. Read the brief against what the change does, run the edge cases in a preview, and check the outcome plan. Leave design, tests and security to engineers and tooling. On a solo team, do both, but as separate passes.

How long does an acceptance review take?

For a typical feature-sized change, about five minutes once your briefs contain testable criteria and you keep a standing list of edge cases. The first few reviews take longer because they expose missing criteria. That time moves into writing better briefs rather than disappearing.

What is the difference between acceptance criteria and a definition of done?

Acceptance criteria are specific to one change: what this feature must do. A definition of done applies to every change: code reviewed, tests pass, security checks ran, outcome check scheduled. Atlassian notes the two terms are sometimes used interchangeably, but keeping them separate makes agent reviews easier.

Why do AI agents add features nobody asked for?

Agents fill gaps in the brief with plausible behaviour, and extras like undo buttons, filters or settings are plausible. Some are useful. The problem is that each one is unreviewed scope. The surplus pass in rung two makes you keep or cut each extra on purpose, and add a criterion for anything you keep.

Can an AI agent check its own work against acceptance criteria?

Partly. Agents can run tests and compare behaviour with written criteria, and tools that validate a plan before coding catch mistakes early. But agents describe their own work fluently, and fluency isn't evidence. A person still runs rung one (is this the right decision?) and rung four (did it help users?).

Sources

  1. Anthropic — 2026 Agentic Coding Trends Report
  2. Stack Overflow — 2025 Developer Survey: AI
  3. METR — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (July 2025)
  4. METR — Uplift study update on late-2025 AI tools (February 2026)
  5. Google Cloud — Announcing the 2024 DORA report (October 2024)
  6. DORA — State of AI-assisted Software Development 2025
  7. Veracode — 2025 GenAI Code Security Report (July 2025)
  8. Google Engineering Practices — What to look for in a code review
  9. Atlassian — What is acceptance criteria?
  10. Martin Fowler — GivenWhenThen (2013)

This story was originally published on 10 October 2026 at https://www.specky.space/blog/review-what-ai-agents-build-pm-acceptance-checklist.

Top comments (0)