Green and worthless: what verifying one small application taught me, when an AI agent wrote most of the code.
The Sanity DEV Challenge wrapped up ...
For further actions, you may consider blocking this person and/or reporting abuse
The paired-assertion rule is the one I'd steal — a test that X has no effect means nothing without the companion proving the effect exists when X is enabled. Your mutation check (delete the clause, watch it finally fail) is the honest version of coverage. Do you run that check across the whole suite now, or only on the tests you already suspect?
Targeted, not suite-wide. I haven't automated mutation testing across the full suite, and the checks I describe were performed manually during the audit.
I started with tests that looked suspicious, but I wouldn't want suspicion to be the permanent selection criterion. Some of the most dangerous vacuous tests look perfectly ordinary until you deliberately remove the behavior they're supposed to protect.
My longer-term preference would be to require positive controls and targeted mutation checks for load-bearing invariants, particularly authorization, state integrity, and irreversible operations. That gives the practice a defined scope without pretending every assertion needs the same level of scrutiny.
That scope is the honest middle — requiring positive controls plus mutation checks only where the invariant is load-bearing, rather than pretending every assertion deserves the same scrutiny. The selection problem you flag is the interesting one though: if suspicion can't be the permanent criterion, what replaces it — is it the invariant's blast radius, or do you rotate the target set so the ordinary-looking tests get their turn too?
I'd use both, but for different purposes. Blast radius would determine which tests get mandatory positive controls and mutation checks. Anything protecting authorization, data integrity, irreversible operations, or a central domain invariant should qualify regardless of whether the test looks suspicious.
But that still leaves the selection problem you identified. We don't necessarily know which ordinary-looking tests are protecting assumptions that turn out to be important. I'd use a rotating sample of the remaining suite, deliberately including tests that have been green for a long time and haven't attracted much attention. When one fails a mutation check, I'd look for other tests built around the same assumption or fixture pattern rather than treating it as an isolated defect.
I haven't implemented that rotation in Cellar, so this is how I'd approach scaling the practice rather than something I've already validated. The original audit taught me that suspicion is useful for finding the first problem, but it's not a reliable coverage strategy.
The goal wouldn't be to mutation-test everything eventually. It would be to make the selection criteria explicit and use what the sampled tests reveal to improve them.
Using both is the honest answer — blast radius for the mandatory set, rotation for everything else. The part I'd underline is treating a failed mutation check as a signal to audit the fixture pattern, not just the one test: that's where a single vacuous test becomes a systemic find. Do you rotate on a schedule, or does a change to a shared fixture trigger its dependents' turn?
I'd favor a change to a shared fixture triggering a review of its dependents, with periodic sampling to catch problems that don't announce themselves through a change. But that's a proposed approach, not something I've implemented in Cellar.
The larger lesson from this audit was that one vacuous test can reveal a shared assumption affecting several others. That's the part I'll carry forward into future projects.
Thanks for the thoughtful discussion!
The positive and negative control framing is a much better vocabulary than just calling these weak tests. The first example is the one I recognise most: a test that was measuring the fixture rather than the rule, because the competing claim lost on recency regardless of its review state. Deleting the clause the test claims to cover is such a cheap mutation check, and it is telling that it found two more cases of the same shape. The always-failing grep is the mirror problem I worry about in CI, because once a team learns to dismiss a check as a known false positive, the check has effectively stopped existing. Do you apply the mutation step to every new inertness or alarm test as a routine gate, or only to the load-bearing ones, given the time it would take to do it across a whole suite?
Only the load-bearing ones, at least for now. I haven't made mutation testing a routine gate across the entire suite, and I wouldn't want to suggest otherwise. In this project, I used it where a passing test was supposed to establish an important behavioral boundary, particularly assertions that something cannot affect the result.
The proposed-claim example is a good illustration. The negative assertion was green, but deleting the review-state filter left it green. The paired positive assertion exposed the problem, and mutation confirmed that the repaired test actually depended on the rule it claimed to cover. I found two more cases with the same basic problem.
I think the practical approach is risk-based. If a test protects an authorization boundary, an irreversible operation, or a central domain invariant, I want evidence that it fails when the relevant protection disappears. For less consequential tests, applying that process universally could cost more than it returns.
And I agree about the always-failing grep. Once a team learns to ignore a red check, the check becomes operationally equivalent to one that never runs. Both create the appearance of verification without providing much of it.
I agree
The proposed-claim fixture is a great example of a test passing for the wrong reason. A useful companion is to mutate the review-state filter and require the test to fail; that checks the test can actually detect the rule it claims to protect.
Exactly, and that's what finally gave me confidence in the repaired test. Removing the review-state filter had to make the test fail. Before the repair, it stayed green because the fixture's recency ordering masked the missing rule.
I ended up finding two more tests with the same underlying weakness, and all three repairs were verified by deliberately removing the logic they claimed to protect. It's a simple experiment, but it changes the question from "does this test pass?" to "can this test actually detect the failure I care about?"
That is exactly the test I trust most: deliberately break the protected rule and watch the test fail for the expected reason. It is a small mutation test, but it catches fixtures that accidentally make a green result meaningless.
Your first one has a cousin I keep finding on client sites: an accessibility audit that's green because every image has alt="image". The check asks whether the attribute exists, and nothing in the pipeline ever reads what it says.
Do you have a control for that kind of check, or is that simply the point where a human has to look?
That's a great example of a check answering a much narrower question than the one we actually care about. The pipeline proves that an attribute exists, not that someone using a screen reader gets useful information from it.
I'd approach it with two layers. Automated checks can reject empty or generic values where descriptive alt text is expected, flag repeated descriptions, and identify suspicious patterns. But those are still heuristics. A human needs to assess whether the description communicates the image's purpose in context, including whether the image is decorative and should have empty alt text.
I'd want a known-bad control example, such as a meaningful image with alt="image", that the audit must reject. That at least proves the check can detect the failure it's supposed to catch. It doesn't eliminate the need for human review, but it makes the automated part more defensible.
A known-bad control is such a cheap fix too. One alt="image" left on a staging page on purpose, and the audit has to go red every time, or the audit is the thing that's broken.
A fifth one from container scanning: the gate that cannot tell "nothing critical" from "nothing read".
trivy image --exit-code 1 --severity CRITICAL busybox:1.37exits 0 with no warning. Trivy detects no OS in that image, so it scans no packages, and the report has noResultsat all. The signedcosign-vulnattestation carries the same absence, and it reads as clean. A positive control catches it only if the control image is one the scanner can read:alpine:3.10fails on CVE-2021-36159.The clause I added is your "not verified" outcome written into the admission policy: no severity is counted until the report has a
Resultsentry of classos-pkgs. An empty scan is a missing scan. I reported the exit code upstream: trivy discussion #11403That's an excellent fifth example. The distinction between "no critical vulnerabilities found" and "no OS packages examined" is exactly the kind of evidence boundary I was trying to get at.
What's particularly interesting is that signing the attestation doesn't resolve the problem. It can establish the provenance and integrity of the report without establishing that the report actually covered the intended material.
I like your admission-policy requirement for an
os-pkgsresult. It makes successful coverage a prerequisite for interpreting the severity findings, rather than allowing an empty result to masquerade as a clean one.Thanks for sharing the concrete reproduction and upstream discussion. This is a particularly useful example of why verification has to establish what was examined before interpreting what was found.
The paired-assertion rule deserves to be mechanical: if a test asserts "X causes no change", auto-generate its sibling that enables X and asserts the change exists - if you cannot write the sibling, the first test is vacuous. Mutation testing is the batch version: delete the review-state filter on purpose and count how many of the 530 tests go red. The scary number is not the 3 bugs that slipped through, it is how many of the 530 would stay green with their feature deleted. Did you ever run that count?
I didn't run a suite-wide mutation count, so I don't have that number. The mutations I performed were targeted: remove the review-state filter, confirm the relevant test fails, and repeat for the other cases uncovered during the audit.
I agree with making positive controls a more explicit requirement, although I'd distinguish generating a sibling assertion from proving that the sibling exercises the intended behavior. The original problem was precisely that the fixture made the rule irrelevant, so automatically generating another assertion wouldn't necessarily solve it.
I'd also be careful with the denominator. If removing the review-state filter leaves 500 unrelated tests green, that's not inherently concerning. What matters is how many tests claiming to protect that rule remain green after it's removed.
That's the measurement I'd want from a more systematic mutation-testing pass: not simply how many tests fail, but whether the tests responsible for each invariant can actually detect its absence.
This is clearly an instance of a positive-control bug. I executed a version of your first lie in my own testing framework last night. A security policy can only be applied on Windows operating system and therefore one of the tests was written by simply replacing the check with Windows with a test that returns True. This meant that the branch was tested but not the switch used to activate it. Also, the tests executed in the Windows environment do so from within the home folder which is not covered by the security policy. This means that the remaining 173 tests that used the switch did not detect the bug. Flicking the switch brought about no changes. This is clearly a case of positive control failing to work.
That's an excellent example, particularly because there are two independent reasons the tests could remain green. Replacing the Windows check with an unconditional True removed the platform condition, while running the tests from a directory outside the policy's coverage meant they weren't necessarily exercising the protected behavior in the first place.
The detail that stands out is that 173 tests used the switch, yet toggling it made no difference. That's a pretty compelling indication that the tests weren't establishing the behavior they appeared to cover.
I'd be interested in whether moving a representative test into a policy-covered directory and explicitly toggling the switch produces the expected behavioral difference. That would give you both a positive control and evidence that the test environment actually exercises the enforcement boundary.
It's a great illustration of why test count and branch coverage alone don't establish that a security control works.
Absolutely; that is the test that I introduced after this commit (github.com/DeChuks-hub/llmshift/co...). In that test, the actual switch is kept, but the home directory is shifted so that the test directory is the target directory for the rule. A config above that is ignored on Windows and loaded on Linux and macOS. With that configuration, a switch that is always turned off fails on Windows, not passing all 173 tests. Right; it is the number of tests that is deceiving here.
The control framing is what makes this more than a testing cautionary tale. A check with no positive control — nothing proving it can actually detect the thing it guards — produces green results that are really just unexamined results, and a suite of 530 of those feels safer than a suite of fifty honest ones while being the opposite. The detail that three of the four real bugs lived outside the view of the checks designed for their own feature is the sharpest part: the checks were not lazy, they were pointed at a model of the feature, and the bugs lived in the gap between the model and the thing itself.
I think you've captured the uncomfortable part of this really well. The tests weren't necessarily poorly written, and many of them were doing exactly what their assertions said they should do. The problem was that those assertions didn't establish the behavior I thought they established.
That's what made the three missed bugs so revealing. I had a substantial suite and plenty of green results, but the verification model had blind spots that the test count couldn't expose.
The distinction between testing our model of the feature and testing the actual behavior is one I'll be carrying into future projects. Thanks for putting it so clearly.
Three of four real bugs slipping past every check built for the feature they lived in, that stat stuck with me. When the agent writes both the code and the tests, they can share the same wrong assumption, so green stops meaning anything. Did you ever break a function on purpose just to confirm something would turn red, or did that idea come later?
Yes, but that came later. It wasn't part of my original testing strategy, and that's probably the most important part of the story.
The turning point was a test asserting that a proposed claim couldn't change the cellar's state. It passed, but when I removed the review-state filter entirely, it still passed. The fixture had been constructed so the proposed claim would lose on recency anyway, regardless of whether the review rule existed.
After fixing that, I deliberately removed the relevant logic again to confirm the test would turn red. I applied the same approach to two other tests discovered during the audit.
That changed the question I was asking. Instead of just asking whether the implementation passes the test, I started asking whether the test can actually detect the failure it's supposed to prevent. It's a small wording change, but it makes a substantial difference in what a green result means.
The fourth lie has a cousin on the agent side. In a recorded session of ours, two of twelve domains resolved to the wrong company record and the agent reported all twelve as fine, because every call succeeded. The only thing that caught it was a column nobody asked it to check: the headcount band against the company we knew. Verify the target before the result, same habit.
That's an excellent example of the same failure showing up in an agent workflow. All twelve calls succeeded, but success only established that the tool returned records, not that those records belonged to the companies you intended to retrieve.
I particularly like the headcount-band detail. It gave you an independent signal that the identity mapping was suspicious, even though the agent wasn't asked to examine it. Of course, headcount disagreement doesn't necessarily prove a mismatch, but it gives you a reason to stop and investigate rather than accept twelve successful calls as twelve correct resolutions.
It makes me wonder whether entity resolution should have its own explicit verification step before the agent is allowed to make claims about the retrieved company. Otherwise, every downstream conclusion can be internally consistent while referring to the wrong organization entirely.
The fourth lie is the one that scared me most — perfect assertions certifying the wrong cellar because the app silently ignored the dataset variable. 'Verify the target before the result' is the habit I'd steal, starting with just showing the dataset name in the header. Did the hand-edited literal survive contact with a real team workflow, or does someone eventually 'fix' it back into an env var?
That's a fair challenge, and I don't have evidence that the hand-edited literal would survive a larger team's workflow. This was a solo project, so I could make that tradeoff without negotiating with anyone else.
The important thing was eliminating the silent fallback. The environment variable appeared to select staging, the build succeeded, and the application still pointed at production. Making the dataset explicit in the source and visible in the UI made that mismatch harder to overlook.
For a team, I'd want explicit build-time selection that fails on missing or invalid values, plus verification of the dataset actually embedded in the built artifact. I'd keep the visible indicator, too. The invariant isn't that configuration must be hand-edited; it's that the target cannot silently change while every test continues to pass.
The mutation check is the part I would make non-negotiable, because a test that stays green after its target code is removed is not evidence at all. I would also apply the same control logic to alarms and data checks, since a warning that cannot stay quiet on clean input teaches a team to ignore it. Where do you draw the line in practice between a load-bearing test that deserves this treatment and an ordinary test where the extra control would cost more than it saves?
I draw that line based on the consequence of a false green result, rather than the complexity of the test itself. Authorization boundaries, irreversible operations, state transitions, and rules that protect the integrity of the underlying data are where I'd want mutation checks and positive controls as a matter of course.
For something like presentation formatting or a low-impact helper function, the additional verification may not justify its cost. That doesn't mean those tests shouldn't be reliable, just that I'd prioritize the ones where a misleading pass could cause substantial damage or invalidate a larger set of conclusions.
I also think your point about alarms is important. A warning that fires on everything eventually becomes background noise. Both controls matter: can it detect the condition when present, and can it remain quiet when the condition is absent? Otherwise, we haven't really established what the alarm means.
The recency example is the one that stays with me. The review state filter could be deleted and the test still passed, because the proposed claim lost on recency anyway. Nothing was wrong with the assertion. The setup simply made failure impossible.
Pairing every "X has no effect" check with a second test proving the effect exists when X is on is a cheap habit, and the targeted mutation idea from the comments makes it concrete: delete the clause, and if nothing goes red, the test was decoration.
Lie four worries me more on client work. An env var quietly pointing at production turns a green run into a statement about the wrong system. Kevin Hall's headcount band catching two of twelve domains mapped to the wrong company shows the same idea at the data level.
Your line about choosing tests by the consequence of a false green is how I would explain test budgets to a non technical owner. Do you keep a short list of the known bad controls per project?
Not as a formal project-wide catalog yet. In this audit, the known-bad cases were targeted at particular invariants, including removing the review-state filter and confirming that the repaired tests detected its absence.
I do think a small collection of known-bad controls would be useful, especially for authorization boundaries, data integrity rules, and other places where a false green has substantial consequences. It would also make those controls easier to preserve as the implementation evolves.
The important part would be keeping the controls tied to the behavior they demonstrate, rather than collecting fixtures that eventually become another set of assumptions nobody challenges. Thanks for connecting that back to the cost of a false green!
the test that stayed green with the whole review state filter deleted is the failure i see most in generated tests. what changed things for us was a mutation rule in review: delete the clause the test claims to cover, and if it does not go red the test gets rejected. it caught fixtures that were quietly inert more than once. did you automate that mutation pass in CI or is it still a manual check per test?
Still manual, and targeted rather than applied to every test. I haven't built an automated mutation pass into CI for this project.
The first discovery came from deliberately removing the review-state filter and finding that the test stayed green. After repairing it, I repeated the mutation to confirm the test would fail. I then found two more tests with the same underlying problem and verified those repairs the same way.
I like your review rule because it makes the test demonstrate that it actually depends on the behavior it's supposed to protect. For a larger codebase, I'd probably start by applying that requirement to load-bearing invariants and authorization boundaries rather than every assertion, then look at automation where the patterns are repeatable.
The important change for me was treating a passing test as the beginning of the verification question, not the end of it.
This is an excellent reminder that having a large test suite is not the same as having reliable verification. The distinction between a test that passes and a test that actually proves something is especially important in AI-assisted development, where an implementation and its tests can inherit the same incorrect assumptions and end up validating each other.
The concept of positive and negative controls is particularly valuable. Whenever we test that something should have no effect, we should also demonstrate that the effect can be detected when the relevant condition changes. Similarly, every security check needs a known-clean input to expose false positives and a deliberately unsafe input to prove that the detection mechanism actually works. Otherwise, we risk confusing activity with evidence.
I also appreciate the distinction between testing the implementation and validating the contract. Even a perfectly constructed test can faithfully enforce the wrong requirement. Property-based testing, mutation testing, and independently generated oracles help uncover different classes of defects, but none can completely compensate for a flawed specification. Independent implementations are especially useful because they are less likely to share the same implementation mistakes, although they can still share incorrect assumptions about the expected behavior.
The fourth example highlights another often-overlooked issue: verifying the environment before trusting the result. A test can be logically correct, execute successfully, and still provide false confidence if it targets the wrong dataset, configuration, or deployment. Making the target environment visible and verifying it explicitly should be part of the test contract, not an optional debugging step.
With AI agents, I think the most important improvement is to make uncertainty an acceptable outcome. An agent should be able to report “not verified” when it lacks the evidence needed to confirm a result. That is far more valuable than a confident green check based on an assumption. Verification should rely on observable outcomes, independent evidence, and reproducible experiments—not simply on an agent reporting that its own work succeeded.
Ultimately, tests are not proof that software is correct; they are evidence whose strength depends on the quality of the oracle, the independence of the checks, and the relevance of the environment. AI can accelerate both implementation and testing, but it also makes it easier to produce convincing evidence that is fundamentally misleading.
The question I keep coming back to is this: when an AI agent writes both the implementation and the tests, how often should we require an independent verification path before considering a critical behavior genuinely tested?
Always. 😏
At least for critical behaviors where a false green could have serious consequences. Independence doesn't necessarily require a separate implementation, but it does require a verification path that isn't simply repeating the assumptions of the code being tested.
And that's really the distinction I was trying to explore in the article: a test passing is an observation; deciding what that observation actually proves is a separate exercise.
Thanks for the thoughtful response!
Giving the agent a third answer, not verified plus the command and expected numbers, fits my eval judges too. Mine still have to pick pass or fail when the evidence they'd need isn't in what they were shown, and I suspect some of my green comes from that rounding up. Once your agent had that option, how often did it use it?
I don't have a reliable frequency to give you, and I wouldn't want to turn a workflow change into a quantitative claim I haven't measured.
The important change was allowing the agent to distinguish between evidence that supports a pass, evidence that supports a failure, and evidence it hasn't actually obtained. Previously, the workflow encouraged a binary conclusion even when the required verification hadn't happened.
The NOT VERIFIED outcome gives it a legitimate way to stop and identify the missing evidence, including the command that should be run and the expected result. But I haven't instrumented that behavior sufficiently to say how often the agent selects it or whether it consistently chooses it when it should.
Your point about eval judges rounding up is exactly why I think that third outcome matters. Forcing a verdict when the evidence is missing doesn't make the evaluation more decisive. It just hides the uncertainty.
the fixture-measuring-itself one keeps biting. agent writes the code and the tests, they share one wrong assumption, green means nothing.
Exactly. When the implementation and tests inherit the same assumption, they can agree perfectly and still be wrong. That's why I found the deliberate mutations so useful: they forced the tests to demonstrate that they could actually detect the behavior disappearing.