I've spent eleven years building test automation for enterprise systems. The whole job rests on one quiet assumption: a passing test is evidence. Green means safe. We ship on green.
Lately that assumption has been getting shakier, because more and more of the test itself is being produced by AI:
- AI generates the test oracle (the assertion that decides pass/fail).
- AI heals broken UI locators when the page changes (self-healing frameworks).
- AI adjudicates whether a run actually passed or was "just flaky" (LLM-as-judge).
Each of these can hand you a GREEN result that isn't actually sound:
- An AI-written assertion can be too weak to catch a real bug — it checks the type but not the value, so the test passes even when the output is wrong.
- A self-healing locator can "repair" itself onto the wrong element and sail through, silently testing nothing.
- An LLM judge can wave a genuine failure through as a flake.
In all three cases the dashboard stays green while the safety net quietly grows holes. I wanted to stop hand-waving about this and actually measure it. So I built a small, open research prototype called TrustLab.
The idea: the False-Confidence Gap
The core concept is simple:
False-Confidence Gap = how often tests appear to pass − how often they are actually sound.
To measure it honestly you need ground truth for "sound." I use mutation testing: an engine injects real faults into a program (flip a + to a -, a < to a <=, and so on), and a verdict only counts as sound if it actually catches those faults. No opinions, no self-report — a test either kills the mutant or it doesn't.
I ran this across the three modalities above, on a set of small subject programs, repeated over 15 random seeds with held-out evaluation.
What came out
- AI-generated oracles: ~62% gap. This is the scary one. A huge share of "passing" came from assertions that don't actually catch injected faults.
- LLM-as-judge: ~41% gap.
- AI self-healing: ~11% gap — lower, but with silent mis-heals hiding in the tail (the worst kind, because nothing looks wrong).
A detail I didn't expect: for AI-generated oracles, the danger tracks code structure. Branch-heavy logic like a grading function (99.7% gap) or a triangle classifier (93.6%) was far more dangerous than a simple BMI bucket (5.2%). The more branches your code has, the more room a weak AI oracle has to look green while missing faults.
Can we do anything about it?
Measuring the problem is only half of it. The other half: can we predict which green verdicts to believe, using only signals we can actually observe at runtime (not the ground truth, which in real life we don't have)?
I trained a small, calibrated trust metric on observable features and compared it to the naive policy everyone uses today — "it's green, trust it."
If you keep the 70% most-trusted green verdicts and set aside the least-trusted 30%, the silent false-pass rate drops hard:
- Self-healing: 11% → 0.5%
- LLM-judge: 40% → 19%
The naive "trust any green" baseline can't separate sound from unsound at all (ROC-AUC 0.50 by construction). The trust metric reaches ROC-AUC up to ~0.95, with calibration (Expected Calibration Error) reported so the scores actually mean something. Effect sizes are "large" (Vargha–Delaney A12 0.76–0.95), Mann–Whitney p < 0.001.
Being honest about what's real
I think credibility matters more than hype, so I'm explicit about this in the repo too:
- The oracle results are real: a genuine AST mutation engine injects faults, and runtime coverage decides whether a verdict is sound.
- The self-healing and adjudication results are empirically-shaped models of documented tool and LLM behaviour, placed behind an adapter interface. That means you can drop a real Playwright self-healer or a real LLM judge into the same seam and re-run the exact experiments without touching the metric. (That's literally one of the open issues on the repo — contributions welcome.)
I'd rather you know exactly where the solid ground is.
Why it matters for the rest of us
If your pipeline uses AI to generate assertions, heal locators, or judge flakiness, "green" is no longer self-evidently trustworthy. Two practical takeaways:
- Weak oracles are the biggest risk, and they're worst exactly where your code is most complex. AI-generated assertions deserve review in proportion to branch complexity.
- A trust score beats a raw green/red. Even a simple calibrated metric over observable signals can tell you which passes to re-examine — turning a binary into a prioritised queue.
Try it / break it
Everything is open and reproducible — pure Python standard library core, 19 passing tests, the benchmark dataset, and the experiment outputs:
👉 github.com/rollamadugu/trustlab
I'd genuinely value feedback from people who build test oracles, self-healing frameworks, or LLM-evaluation pipelines — especially if you can poke holes in the methodology. There are a couple of "good first issues" open if you want to plug in a real tool. A fuller write-up (preprint) is in preparation.
I'm Krishnasai Rollamadugu, a test automation architect (11+ years) researching trust and measurement in AI-driven software testing. If this resonates, follow along — this is the first step of a longer research direction.


Top comments (5)
The thing I would most want to see separated out is the synthetic adapters versus the real tools. Right now both feed into the same headline number, and a reader cannot tell how much of the result comes from a model of behaviour rather than from behaviour.
Two suggestions if you do the real-tool run. Write down the pass/fail criteria before you look at any results, and publish them with the run, so nobody has to take your word for it later. And report a confidence interval, because with the mutant counts in this range the difference between, say, 61% and 68% can easily be noise.
It would also help to break the escape rate down by mutant class. "Tests miss off-by-one errors far more often than they miss removed null checks" is a much more actionable finding than one combined percentage.
@botgauntlet This is exactly the kind of critique I was hoping for — and you're right on all four.
1.Separating synthetic adapters from real tools is the big one. The oracle modality is genuinely mutation-grounded, while healing and adjudication are empirically-shaped models behind an adapter seam. I say so in the README, but you're right that folding them into one headline number hides how much is behaviour vs model. I'll report the real vs modelled modalities separately and label each one's status explicitly.
2.Pre-registering pass/fail criteria before the real-tool run — agreed. I'll commit the hypotheses and thresholds to the repo before running, so nobody has to take my word for it afterward.
3.Confidence intervals — fair. With these mutant counts, the difference between 61% and 68% can absolutely be noise. I'll add CIs (across seeds and on the proportions) to the results in the repo, not just the post.
4.Escape rate by mutant class — this is the most actionable suggestion. The engine already tags operators (arithmetic / relational / logical / constant / unary), so I can break the escape rate down by class. "Tests miss off-by-one far more than removed null checks" is exactly the kind of finding worth surfacing over one combined percentage.
Folding these in. The real-tool run — a real LLM judge behind the same seam — is next on my list, and I'll pre-register and report CIs for that one. Thanks for taking the time to engage this carefully.
@botgauntlet Shipped all of this in v0.2.0 — thank you, this made the work better.
CIs: added bootstrap 95% CIs across seeds (and Wilson intervals for single-run proportions). You were right to push on it — the gaps are stable (e.g. oracle 61.9% [61.3, 62.6]), but now that's shown, not asserted.
Real vs modelled: the results table now has a "Grounding" column. Only the oracle modality is a real mutation-tested measurement; healing and adjudication are explicitly labelled as models behind the adapter seam, so no one has to guess how much is behaviour vs model.
Escape rate by mutant class: added as its own experiment — and it backs up your exact intuition. Relational / off-by-one faults escape most (44.1%), while arithmetic faults are caught far more often (13.7%). "Tests miss off-by-one more than removed null checks" turns out to be measurably true here. Far more actionable than one combined number, agreed.
Pre-registration: committed a PREREGISTRATION.md fixing the hypotheses and pass/fail criteria for the real-LLM-judge run before I run it, so nobody has to take my word for it afterward.
Repo's updated if you want to poke holes in the next round.
The self-healing locator fixing itself onto the wrong element is the sneaky one - everything stays green while the test quietly checks nothing. Putting an actual number on that gap is useful.
@abhinavs1920 Thanks Abhinav — that's the one that worries me most too. A failing assertion at least shows up red. A mis-heal keeps everything green while the test quietly checks nothing, so it hides in the exact place nobody looks.
One honest caveat on that number, since I'd rather you know: in the current benchmark the self-healing modality is a model of documented healer behaviour (behind an adapter seam), not a real framework yet — so treat 10.8% as the model's gap, not a measured tool. Dropping a real Playwright/Selenium self-healer into the same seam and measuring the true mis-heal rate is next on my list, and I've pre-registered the criteria so I can't massage the result afterward.
Have you hit silent mis-heals in a real suite — and if so, how did you end up catching them? That failure mode is exactly what I'm trying to make measurable.