Your agent produced 400 lines in nine seconds. It compiles. The tests pass. The demo works.
None of that tells you whether it's correct.
I evaluate agentic AI coding output against structured rubrics professionally - correctness, instruction adherence, edge-case handling, the whole grid. Separately, I've spent seven years shipping production systems: high-traffic e-commerce, multi-tenant healthcare data, a national eLearning platform handling tens of thousands of concurrent applications. The checklist below is what happens when those two things collide. It's the pass I run before I'll put my name on anything an agent wrote.
None of it is exotic. All of it is stuff agents get wrong constantly, and reviewers skip because the code looks fine.
1. Does deleting the error handling break any test?
If not, your error paths are untested decoration. This is the single fastest way to find out whether a test suite is real. Comment out a catch block and run the suite. Green? You have no coverage of the thing most likely to hurt you.
2. What happens on the second call?
Agents write beautiful single-execution logic. Ask: if this runs twice - retry, duplicate message, user double-click - does it produce one side effect or two? Idempotency is seldom in the generated code unless you asked for it explicitly, and it's almost always required in production.
3. Is every dependency pinned, and does the lockfile exist?
"Latest" is not a version. I've root-caused a bug where a minor version bump silently collapsed TypeScript types to never across a monorepo - semver protects runtime behaviour; it promises nothing about type inference or subtle behavioural edges. Agents love unpinned ranges. Pin them.
4. Is it mocking the thing it's supposed to be testing?
The most common fake-coverage pattern I see: mock the database, assert the mock was called correctly, declare the data layer tested. If the correctness property is enforced by the database - constraints, row-level security policies, transaction isolation - a mock verifies your assumption about the policy, not the policy. Run against a real, disposable instance. I maintain a security suite that validates RLS policies against actual Postgres for exactly this reason.
5. Does it check what should be invisible, not just what's visible?
Access-control tests written by agents almost always assert "the authorised user can see their data." The security-relevant half is "the unauthorised user cannot." Test negative cases explicitly - and remember that some systems (RLS being the classic) filter silently rather than erroring, so "no exception raised" is not the same as "correctly denied." Assert the actual result set.
6. How many database round-trips does the happy path make?
ORMs plus agents produce N+1 queries at an impressive rate, because each line looks perfectly reasonable. Log the queries for one request. Count them. The number is usually higher than anyone guessed, and it's where a large share of real latency wins hide.
7. What are the timeout and retry policies - explicitly?
Not "does it retry," but: how many times, with what backoff, and what happens when retries are exhausted? Unbounded retries against a struggling downstream service is how a partial outage becomes a full one. Agents default to either no retries or naive infinite ones.
8. Does it handle partial failure, or only total failure?
Total failure is easy - the call throws, you catch it. Partial failure is the hard case: three of five writes succeeded, the response timed out but the operation actually completed, the queue delivered twice. Generated code is overwhelmingly written as though operations either fully succeed or fully fail.
9. Are the IAM permissions scoped, or is it wildcards?
Agents reach for permissive policies because permissive policies make the demo work. Any wildcard in a generated permission set is a finding, not a default. Same for over-broad database roles - a correct policy can still leak data if the connecting role has privileges that sidestep it.
10. Is there anything in here that only works because of a race the tests never trigger?
Concurrency is where "semantically wrong about failure" gets most expensive. Check-then-act patterns, non-atomic read-modify-write, missing transaction boundaries. Tests run sequentially; production doesn't.
11. Can someone else reproduce this from a clean clone?
One command, fresh machine, same result. If setup requires tribal knowledge or an undocumented sequence of steps, your correctness is unverifiable by anyone but you - which, for anything that will outlive your attention span, means it's unverified.
12. Is the reasoning written down anywhere?
Which ambiguity did you resolve, and how? What did you deliberately not handle? Agents produce code without provenance - no record of what was considered and rejected. That gap is a real maintenance liability, because the next person can't distinguish a deliberate decision from an accident. Write down the why, especially for the non-obvious calls.
The pattern underneath all twelve
Every item is a variant of the same thing: agents are excellent at code and unreliable about consequences. They handle the path you described and quietly assume the paths you didn't. The failure mode people complain loudest about - hallucinated APIs - is the easy one, because it's loud and any test catches it. The dangerous one is syntactically perfect code that's confidently wrong about what happens when something breaks.
Which means the review skill that matters now isn't "can you spot bad code." It's "can you enumerate the failure modes nobody wrote down." That's not a new skill. It's the thing senior engineers have always done. It just got a lot more load-bearing.
What I'm building from this
I'm turning this checklist into something executable - a fault-injection harness that stress-tests agent-generated infrastructure code against realistic failure conditions (retries, partial outages, IAM misconfigurations, concurrent access) with deterministic pass/fail checks instead of eyeballing. Chaos engineering, pointed at AI output. It'll go up on my GitHub and portfolio as I build it in the open, along with what breaks and why.
What's on your list that isn't on mine? I'm collecting failure patterns for the harness's scenario set, and the ones that come from people who've been burned in production are worth more than anything I can invent. Drop them in the comments.
Part of an ongoing series on production debugging, performance engineering, and evaluation infrastructure for AI systems.
Top comments (0)