An AI coding agent implements a feature, generates tests, runs them, and opens a pull request with a green build.
That looks like independent veri...
For further actions, you may consider blocking this person and/or reporting abuse
Great article, John! πΈ
I really liked the point that AI-generated code and tests can share the same misunderstanding, so a green build isnβt necessarily independent evidence.
The idea of using a different source of truth, like contracts, invariants, boundary cases, or a fresh adversarial context, feels much stronger than simply asking another model to generate more tests.
I especially liked the focus on asking where each expected value actually came from. That feels like the real test of whether the suite can challenge the implementation rather than just agree with it. π
The oracle framing is the right one, and the trap you name under "asserting the implementation against itself" is the one I keep meeting in agent-written suites. One thing in the worked example does not hold, and it is checkable in a minute.
Take the two functions as printed. Intended: discount first, then tax on the discounted base, which gives 88.00. The snippet does this:
90.00. So the test asserting
.toBe(88)goes red against the code it is paired with, and the example ends up showing a test that disagrees with the bug rather than agreeing with it. It is also not a values problem: the difference is exactlysubtotal * taxRate * discountRate-- the tax charged on the discount, 2.00 here. I ran both formulas over 90,000 strictly-positive (rate, discount) pairs and found zero where they agree; they can only coincide when one of the two rates is 0. (88.00 is what a third implementation returns, the one that takes the discount on the tax-inclusive amount: (100 + 10) * 0.8.)The useful version of that fact is that the gap scales as the product of the two rates. At 10% and 20% it is 2.00 on a 100.00 order -- the size of a rounding difference, which is why a mismatch of this kind survives review. Your decision-table example is the right repair and this gives it a reason: push both rates up and the interpretations separate, 108.00 against 110.00 at 125.00 / 8% / 20%.
One addition to the mutation section, because there the correlation reappears one level down: a mutant set drawn from the implementation cannot see an oracle that drifted, since the mutants and the oracle read the same code. I scored two versions of one test against a battery in your vocabulary -- three syntactic mutants of the correct implementation, then one more drawn from the claim ("the discount is taken after tax"):
The drifted oracle there pins 110.00, which is what the buggy implementation produces at that point -- an expected value that came from the code. So step 3 of the workflow, listing plausible incorrect interpretations, is not a nicety: it is the only source of mutants that can separate a right expected value from a plausible wrong one, and it is why the adversarial tester should be asked for interpretations before it is asked for test cases.
To your closing question, the most convincing green suite I have seen reported this year was 42 integration tests that skipped themselves on every dependency pull request, with the required check satisfied by the skips.
Brilliant breakdown - especially pinpointing that 88 came from the tax-inclusive sequential math (100 + 10) * 0.8. Fixing that snippet!
βYour point on mutation testing is pure gold: if mutants are only drawn from the implementation, they inherit the exact same blind spot. Generating mutants from competing interpretations (claim-drawn) rather than just syntax flips is crucial to preventing oracle drift.
βAlso, 42 self-skipping tests satisfying CI is equal parts terrifying and hilarious. Thanks for such a high-signal contribution!
Love the core point, correlated output isn't independent evidence. Fun twist though: I ran the example by hand and the "wrong" calculateTotal(100, 0.1, 0.2) returns 100 + 10 - 20 = 90, not 88, so that generated test would actually fail π Which kind of proves your point: it looked right, and I almost scrolled past without checking the math, same as a green build.
Eagle eye! Youβre completely right β 100 + 10 - 20 = 90. π
Peak irony that a post about not blindly trusting plausible assertions had an assertion slip through because 88 just "felt" right. Thanks for being the independent oracle the article was advocating for - fixing the snippet now!
Ha, it happens to all of us. If anything, the slip made the point even stronger π Great post.
The leak I kept hitting when splitting the workflow into an adversarial test agent was giving that second agent access to the pull request diff.
Even when the tester prompt says to act as an adversarial tester and find bugs in this change, handing it the diff immediately primes the model on the implementer's variable names, error codes, and edge-case boundaries. It ends up writing tests tailored to the shape of the patch rather than the shape of the domain requirement.
The setup that actually caught mismatches for me was keeping the diff completely out of the tester's context. The tester gets only the issue description and the repo at the merge-base commit. It has to write a test that fails on main first, proving that the test is testing the missing requirement, before the candidate patch is merged to run against it. If the test cannot fail against the pre-patch commit, the test is usually either vacuous or asserting an internal helper that did not exist yet.
For the merge base check, I'd ask the tester to include the failure itself. An import error for a new public API makes the run red without exercising its behavior. For a fix behind an existing interface, the useful check is an assertion that runs on both revisions, fails before the fix and passes after it. If the interface is new, its absence on the old revision should be reported separately from a reproduced behavioral failure.
Critical nuance - an
ImportErrororTypeError: undefined is not a functionis not the same as a red behavioral test.Distinguishing compilation/interface absence from a true assertion failure on existing behavior prevents the agent from gaming the merge-base check. Requiring the failure output/stack trace as explicit evidence is definitely the missing piece for new APIs. Thanks for adding this!
Spot-on critique, including the diff in Step 3 is definitely an information leak.
Handing over the patch primes the tester to mirror the author's abstractions and assumptions, which partially defeats the fresh context.
Enforcing agentic TDD (requiring the test to fail on the merge-base before running against the PR) is a much cleaner filter. It proves the test actually has teeth and eliminates vacuous assertions or leaks of internal helpers. Stealing this workflow tweak!
The billing example is a good one because the gap is small enough to pass a skim. A $2 difference on a $100 order looks like rounding, not like tax applied in the wrong order.
The fix you describe, deriving acceptance tests from a separate source, matches what has worked for us on client builds. The expected values come from the written requirement before any code exists, ideally checked by the person who owns that requirement. The agent can then generate as many extra tests as it likes, but it cannot edit the ones that define done.
Mutation testing is the part I see teams skip because it feels slow. In your experience, is a targeted run on the changed functions enough, or do you need the full suite to catch this kind of shared assumption?
Targeted runs on changed functions (or diff-scoped) are more than enough β and practically the only way to make mutation testing viable in daily CI/PR workflows. Running it across the entire suite produces too much runtime overhead and equivalent-mutant fatigue for marginal extra gain.
The key to catching shared assumptions isn't test breadth across other files, but the oracle anchoring the mutated code:
Targeted AST mutations (e.g., swapping operators, zeroing discounts, flipping relational signs) on the modified functions immediately expose whether tests actually assert business logic or just mirror happy paths.
Locked acceptance tests: As you pointed out, the real safeguard is ensuring those independent acceptance fixtures can't be mutated or rewritten by the agent to match the implementation. As long as the targeted mutation runner executes against those locked fixtures, an inverted or missing requirement gets caught immediately.
Love that policy of yours: "the agent can generate extra tests, but cannot edit the ones that define done." That single constraint eliminates half of the prompt-drift risks.
The review question 'where did this expected value come from' is the one I will actually carry into code reviews from now on. It cuts through everything. One cheap habit that has helped me: generating the tests in a fresh session that never sees the implementation diff, so the tester cannot mirror the author's abstractions. It does not replace an independent oracle, but it catches the correlated-assumption problem more often than I expected.
That fresh-session habit is fantastic. The moment a model sees the implementation diff, it inevitably anchors to the author's abstractions, naming, and mental shortcutsβtesting how it was built rather than what should actually happen.
Keeping the diff out of the prompt is such an elegant, low-cost way to break that confirmation bias and force independent reasoning. Really glad the review question resonated with you!