TL;DR
- In May I published a saga benchmark whose central finding was wrong. I reviewed the harness, an AI reviewed the harness, an AI confirmed my explanation at draft stage, and the run passed the harness's own fairness gates. Nobody caught the defect for eleven weeks.
- It was finally found by an agent that wasn't reviewing anything. It was implementing the next version of the scenario contract, so it was working from different inputs.
- That is the point of this article: agreement between reviewers is not independent confirmation unless something makes their errors independent, whether the reviewer is a person or a model. Shared inputs and a shared frame are the usual way that goes wrong.
- I don't run a separate label for AI output. I just don't count anything as verified until something outside the frame that produced it has checked it: a primary source, a deterministic oracle, a gate in CI.
- The gate gets stricter as the cost of being wrong grows. And for one step, code review, where working alone I have no second person to review my code, AI made my checks better rather than weaker.
The error nobody caught
On May 14 I published a saga benchmark comparing an in-process Flow engine with two stacks that depended on a separate server process. The headline was a correctness gap. Under 3% configured payment failure, the in-process engine compensated 3.32% of sagas, while both other stacks reported zero compensations and a small share of "unresolved" sagas.
I had an explanation ready: async dispatch returns before the work is done, so the benchmark window closes on sagas that are still in flight. It was mechanical, it fitted both stacks, and it fitted the latency numbers too.
It was wrong. The poller that decided whether a saga had finished didn't recognise CANCELLED, which is the status a compensated saga actually wrote. The compensations had fired. My harness scored them as unresolved, and I read that as the frameworks dropping them. The fix landed on July 28. The retraction of the May article went up a month later, on August 27, inline, with the original text left in place. Most of that month, as far as I can reconstruct it, went into correctness measurements under the corrected contract, and the first campaign under it didn't survive either.
AI was in the loop at every step where this could have been caught. It reviewed the harness first, and it reviewed it the way I did: one implementation at a time. The three arms were built in different periods, so no review, mine or the model's, ever put the three pollers side by side. Later, at draft stage, the model confirmed my async-dispatch reading. It was a good explanation, and it agreed with it.
What found CANCELLED was an agent implementing contract v2 of the scenario. It wasn't asked to review the old finding. v2 replaced probabilistic failure injection with a deterministic one, because I needed to check every stack's compensation count exactly rather than within noise. To do that, the agent had to line up the terminal states each implementation writes with the ones the poller script recognises. They didn't match, and it filed the mismatch as an issue.
What bothers me in hindsight isn't that I made the mistake. It's that the review process looked thorough. The claim had a claim_scope, a reproducibility_status, raw artifacts, JFR files, fairness gates that reject a run failing its equivalence checks, and a second reader. It passed every check I had. All of them read the same harness, the same numbers and the same framing.
Why the AI review agreed with me
In the retraction I wrote about trusting the finding for three reasons: it reproduced, it had a mechanical explanation, and it appeared in two unrelated host runtimes. All three were true and none of them was evidence. A harness defect reproduces perfectly, it admits a mechanical explanation, and it appears in every stack the harness touches.
The AI review failed the same way, for the same structural reason. I gave it my harness one arm at a time, then my numbers and my framing. A model that reads the same inputs, in the same order, inside the same frame, is not a second system. Its agreement told me the explanation was internally consistent. It told me nothing about whether the poller was correct.
The agent that found the bug wasn't smarter than the reviewer. It had a different task, so it was looking at different inputs.
Figure 1: Four checks on the same inputs, and one check on different inputs.
This is the part I think gets lost in most "use AI, but verify" advice. Verification doesn't come from adding reviewers. It comes from adding reviewers whose errors don't move together with yours. Two runtimes agreeing is evidence about the code they share. A model agreeing with you is evidence about the context you gave it.
That doesn't make a second reviewer on the same inputs worthless. This article went through several review passes: mine, and models run under different configurations. Each one caught something the others didn't. One found that a figure in the draft didn't add up. Another found that an earlier version of the TL;DR stated this very thesis more strongly than the evidence allowed. Different configurations fail in different ways, so they catch different classes of error. What none of them could do is the thing the poller bug needed: see an input nobody had put in front of them. That needed different inputs, or a criterion outside all the reviewers.
The fix for the class of error was not a better reviewer either. It was removing judgment from that step. Contract v2 makes the declined subset deterministic, so the expected compensation count is an integer known before the run starts. observed == expected is a hard pass/fail assertion. There is nothing left for a reviewer to find plausible.
That assertion checks one property, and the first campaign under it showed how little that is. It passed the compensation gate on every repeat. I retired its report the day after publishing it, for five reasons, any one of which would have been enough:
| What was wrong | What it invalidated | What changed |
|---|---|---|
| No step ever waited on an external system | The framing itself: the workload was a transaction script with compensation, not saga orchestration | The contract now requires a step that parks |
| The Exeris arm's graph traversal matched nothing and was served from a Postgres fallback | Every cost comparison involving Exeris: most of the CPU gap between Exeris and Quarkus was spent scanning for rows that could never match | The graph path was repaired, then removed from the scenario |
| Container CPU samples were read in the wrong unit | Every CPU figure, and the ordering between stacks | Fixed in 97999c4
|
| The Quarkus arm persisted no saga state at all | Every cost comparison involving Quarkus | Nothing new: the contract already forbade comparing across durability tiers, and the campaign did it anyway |
| Postgres on the benchmark box was published on a public interface with default credentials, and a third party held superuser on it during the run | The provenance of the whole run | Remediated in 56e3e95: backend ports bound to localhost, the foreign superuser roles dropped |
The fifth one didn't look like a security problem at first. It showed up as flaky Postgres authentication that killed a few runs, and for most of an evening I treated it as infrastructure noise.
The integer was right, and the report was still void.
Where I use AI, and what each step has to pass
I use AI across most of my work on Exeris, which I build and maintain alone: looking things up, reviewing text, widening the option space for architecture decisions, writing code, reviewing code, and running multi-step verification. The useful question was never whether to use it. It was what a given output has to pass before I act on it. The answer depends on what a wrong output costs.
| Step | Typical failure | Gate before I act on it | What the gate costs |
|---|---|---|---|
| Search and lookup | Confident paraphrase of a source it didn't read closely | Primary source opened; claim worded to match it | Minutes per claim |
| Text review | Smoothing voice into generic prose; a plausible detail that never happened | Targeted patches, not rewrites; my voice checklist scored before and after | Slower editing |
| Specification and architecture | A coherent option that ignores a constraint it was never told | ADR checklist; every option traced to a constraint I can name | The option space shrinks back |
| Code | Tests written in the same session as the code, passing because they were written to pass | Mechanical gates first, then a multi-part review in CI with a schema'd verdict as a required check | Model runs, and a review routine I have to maintain |
| Autonomous action | Inheriting authority nobody granted on purpose | No destructive action without a human; permissions scoped to the task, not the operator | Agents stop and wait |
The first row is the one I hit most often, and this article has an example of it further down: the two incident summaries I started from were wrong in exactly that way. The second row is where the script for my tJUG talk went wrong, with a framing the model introduced that read well and wasn't true. The third has its closest example in the saga story above: the async-dispatch explanation was a coherent architectural reading built on an assumption nobody had stated, namely that the poller was right.
The code row is the one where my experience went against what I expected. AI-assisted code review is usually faster than reviewing by hand, and in my experience it finds more. That second part is an impression rather than a measurement, so by the rule this article is about, it stays exploratory.
Part of the reason is specific to working alone. Reviewing my own code, whether I wrote it by hand or with a model, is hard to do honestly. I know the shape I wanted the solution to have, and I read the diff against that shape instead of against the code. The review run doesn't have that problem. It runs under a different routine and different instructions from the session that wrote the code, and it has none of the authoring conversation. It shares less of my frame than anything else in the loop. Getting there took configuration rather than a second person, and it only goes so far.
What made it trustworthy wasn't the model, though. It was splitting the review so no single run carries the whole judgment, and putting mechanical gates in front of it that don't read prose at all.
In Exeris the mechanical layer comes first: an SPI compatibility diff, a scan of the shipped bytecode for preview features, the TCK, and tagged integration and stress suites that run in their own CI gates. The model review sits behind that. The organisation's review routine lives in one repository, and every repo checks out the same copy. A planning step decides which parts of the routine a pull request calls for (pr, docs, records, code-docs, code, plus the repo's own rules). Each part is a separate model run, and an aggregation step takes the worst decision across them. The result is a JSON verdict with a schema. The required check is red when the verdict is missing, invalid, BLOCKED, or rests on a gate that didn't run. The schema guarantees the verdict's shape, not that its content is right. It makes it impossible to merge past a review that didn't happen, and that is all it does. The ruleset that requires the check has no bypass actors, me included.
Two details there matter more than the rest. First, the review reads the agent instruction files as the base branch has them, so a pull request can't write the instructions its own review follows. That is the independence-of-inputs point, turned into configuration. Second, the docs lint has an error-level rule for retracted figures. The 0% compensation number from May can't come back into a document by accident.
Would this routine have caught the CANCELLED bug? Probably not. It reviews diffs, and the three pollers were never in the same diff. It might have, if all three implementations had gone through review in one pull request, but they were built at different times. What catches that class of error now is the v2 assertion. The review improves how I read a diff, but it has no expected value to compare a result against, and the poller bug needed one.
The bottom two rows each have a public incident that's worth reading in full rather than as a headline.
In July 2025, Replit's agent deleted a production database during Jason Lemkin's twelve-day experiment, despite a code freeze. The part that matters for the code row came after: Lemkin said the tool had been covering up bugs with fake data and fake reports, and had lied about the unit tests. When the same agent writes the code, the tests and the report on the tests, none of the three is checking the others.
In February 2026 the Financial Times reported that Amazon's Kiro agent, asked to make changes, chose to delete and recreate an environment, which caused a 13-hour interruption in December. Amazon's response narrows it to one service, AWS Cost Explorer, in one region, and puts the cause on a misconfigured role that any developer tool could have exercised. I don't think the two readings conflict. An agent that holds its operator's permissions does whatever that operator could do, without the hesitation the operator would have had.
I fact-checked both incidents for this article with an AI, starting from my own one-line summaries, which were wrong in instructive ways. "AWS's AI deleted production data" is not what was reported. The check worked because it went to the reporting and to Amazon's own statement, not to the model's memory of them.
The evidence bundle doesn't ask who wrote the sentence
Every number I publish carries a claim_scope: exploratory for a single run or a small N not yet cross-validated, comparison_eligible only with matched hardware, symmetric methodology and enough repetitions.
That is the sense of the title. Output from a model enters at exploratory the same way a single benchmark run does: it may well be right, but nothing outside the frame that produced it has said so yet.
I don't keep a parallel system for AI output, and I don't think one is needed. The checks attach to the claim, whoever drafted it. When I prepared the tJUG talk Every Benchmark Measures Two Systems (a lightning version ran at tJUG's 18th birthday on September 18), I wrote the script with heavy AI assistance. The evidence for it lives in an evidence bundle: a document in the benchmarks repository that lists every figure the talk was allowed to use, each in one of four states: CITABLE, UNARTIFACTED, RETRACTED, FORBIDDEN. A figure could reach a slide only from CITABLE, and it reached CITABLE only by being traced to a file in the repo or a run artifact.
It isn't a gate, though. Nothing runs it. The script and the deck were written on my machine, outside any repository, and the bundle only worked because I read the script against it.
The model made real errors in this kind of work. Across several sessions it produced GitHub traction figures that did not exist, misread a timeline from my own whitepaper, and framed the talk script around "two different teams" building the system, when there is one person. Each one was fluent, confident, and consistent with everything around it. None of them survived into the final script or the deck. The CI gates couldn't have seen them either: they run on the repositories, and articles and talk scripts are written outside them, from data that lives in them. I caught all three myself, by reading each sentence against the file it claimed to come from. What worked each time was a check that didn't care how consistent the sentence was.
So the working rule is simple, and it's the same one I use for numbers: nothing counts as verified until something outside the frame that produced it has confirmed it. AI output just reaches that line faster, and in larger volume.
Why the AI layer in Exeris starts by only watching
The same reasoning set the scope of the AI execution layer I'm adding to Exeris. The decision is ADR-086, accepted September 15, and it follows RFC-2026-09-08.
The obvious goal is routing: send each class of task to the cheapest model that is good enough for it. I can't do that yet, because I don't know which model is good enough for which class of task. Routing without that data would be one more confident claim with nothing behind it. The ADR also requires every stage up to recommendation to be worth building on its own, so the dataset has to be useful even if routing never ships.
So the ADR stages the layer by what each stage is allowed to emit, not by what it can technically do. V0 emits capture only: no routing, and not even a sentence saying a model looks suitable for a workload. V1 may describe, V2 may estimate complexity, V3 may recommend. Routing comes only at V4, and its exit criterion is a conjunction on purpose: the router has to reach a correct result at least as often as a frontier-model baseline and cost less. "Cheaper" on its own is something a router achieves trivially by being worse. "Correct" here means the task is actually done (TRUE_DONE) as judged by the domain's calibrated oracle, read from paired runs of the same task across models, with the hypothesis and its falsification condition written down before the data is queried. What the ADR doesn't fix yet is the sample size and the tolerance for that comparison. Both are owed before V4 means anything.
Part of this already runs. When capture is turned on for a repository, each review run writes a metadata-only record to the public exeris-ai-execution inbox. A separate job runs when the pull request closes, reads what I actually did with each finding, and appends a judgement record next to the run. The run record is never edited afterwards.
Here is what that dataset says after its first month, as of October 15: 163 run records (157 live reviews in CI and six runs of planned documentation tasks) and 73 judgement records from pull requests that have closed. Every one of the 157 review rows has the outcome UNKNOWN, and that's intended. The oracle that judges live reviews has no calibration suite, so the schema refuses it a pass or a fail. The only rows with a real verdict are the six planned runs, judged by the one oracle that has passed its own mutation suite, which means it fails when I deliberately break the output it checks.
The review rows are also split across 14 fences. A fence marks a change in what produced a row, and here that means the version of the client running the review: fourteen versions in one month, each its own set of conditions. Rows on either side of a fence are never summed into one figure. The rows carry 18 fence ids rather than 14, because the live producer stamps each id with the day the review ran, so four versions that spanned two days got two ids for the same conditions. That was undocumented until I checked the numbers for this article, and the fence register now says so. A month of capture has produced a dataset that is honest about the fact that it can't answer the routing question yet.
The trade-off is real. Capture-only gives me nothing I can feel this quarter. It costs storage, it leaves the line between metadata and content still to settle, and it postpones the part people actually want, which is "use the cheaper model when it's good enough." I'd still rather spend a quarter collecting rows that say UNKNOWN than route on intuition.
What this costs
Not where I expected. For code review, AI brought the time down, and as far as I can tell the multi-part routine made the review better than what I did by hand. I don't have a figure for the saving, and the capture data above deliberately can't give me one yet.
The cost is in maintaining the gates. A voice checklist, an ADR checklist, an evidence bundle, a review routine, a verdict schema and a capture pipeline are all artifacts I now own. The review workflows are called from main in every repository, so a merge to that one repo is a deployment to all of them, with no staging in between. That is a choice, not an oversight: a rule change reaches every repository at once instead of drifting across pinned versions. What partly stands in for staging is that the repository runs its own gates and its own review on itself, so a broken workflow is meant to break there first. For a solo project that is a real share of the week, and it is easy to underestimate when you only count the time the model saves.
I still think it's the right trade. The failures above didn't look like failures from the inside. A fluent wrong answer and a fluent right answer cost the same to read, and I can only tell them apart with something that doesn't read prose.
What this doesn't solve
- An exact oracle checks exactly one thing. The v2 campaign in the table above is the proof. A deterministic assertion closes one class of error and says nothing about whether the workload, the environment or the comparison were valid.
- Most architectural judgment has no oracle. The saga fix worked because the expected compensation count is an integer. "Should this be in-process?" has no integer. There, the best I have is tracing every option to a stated constraint, and that still depends on my having stated the right ones.
- Writing outside the repositories has no gate at all. Articles and talk scripts, this one included, are checked by me reading them against the data. That is the weakest check in this whole article, and it's the one that caught the model's errors above.
- The judgement step uses my decisions as the oracle. Grading the review by what I did with each finding measures whether the review matched my judgment, not whether either of us was right, and on my own pull requests I am the only judge. It also rewards timidity: a reviewer that reports a few safe findings is never overruled, while one that catches hard things and is sometimes wrong scores worse. That is why ADR-086 admits it as observational only, never as a label, and never in one figure with a calibrated oracle.
- An independent model is less independent than it looks. A fresh session or a different vendor drops my conversation, but not my framing if I pass it the same artifacts. Models trained on overlapping data can share blind spots too. I treat a second model as a cheaper second reader. It isn't a second system.
- A solo maintainer is one frame. A differently configured review run gets me out of my own frame for one step. It doesn't change the fact that I wrote the oracles, will write the router, and am the only annotator of the dataset. ADR-086 lists that as a trade-off rather than a solved problem. The bundle, the gates and the oracles reduce how often I fool myself. They don't replace a second person with different inputs, and Exeris doesn't have one yet.
-
Capture-only proves nothing yet, including about the review. After a month, every review row says
UNKNOWN, so this article can't show that the multi-part review beats a simpler setup. That claim stays exploratory. The measurement that would settle it is specified but not run: a seeded corpus of pull requests with planted defects and clean controls, judged by an oracle that has to pass its own self-test before it may label anything. Planted defects aren't real ones, so even that will be an approximation. V0 might also show that routing isn't worth it for most task classes. That would be a fine outcome, and it's the reason the layer starts by watching.
If you're setting an agent policy for a team, the first thing I'd check is not which model you use. It's which of your reviews share their inputs with the thing they're reviewing.
The retraction and the contract that replaced the finding: 🔗 compensation-correctness-saga-benchmark · 🔗 CONTRACT-v2
The review routine, verdict schema and gates: 🔗 exeris-systems/.github · 🔗 exeris-kernel
The decision and the dataset: 🔗 ADR-086 · 🔗 exeris-ai-execution
Top comments (0)