The short version, for anyone who does not benchmark models for a living
Every few weeks a new model tops a leaderboard and the advice is to switch to it.
We have been measuring our own stack for a while, and two things keep coming back that are worth separating carefully, because the second one is easy to oversell and we have oversold it twice.
The first is measured and we are confident in it. Models that a leaderboard ranks two points apart are not doing the same work. On the problems where they differ, the difference is stable: run them again and the same models are right and wrong on the same specific problems. That is a property of the models. Which one tops the column total is substantially a property of the run.
The second is what we would like to be true, and we have not earned it. It would follow that pairing two different models beats upgrading to one better one. We have never run that experiment. We built the obvious mechanism to capture the available headroom and captured none of it. So take the title as the hypothesis this piece is about, not as a result it delivers.
What we can hand you is the measurement underneath it, a ceiling we could not reach, and twenty-one days of a ledger recording every time our own second model told us something confidently false. Including once, today, about this article.
What "a better model" actually buys you
Start with a number that surprised us, on our own production path.
We ran the same fixed 800-item tool-calling benchmark through our live system five separate times. Nothing changed between passes. Same items, same harness, same endpoint.
The five totals were 752, 750, 750, 749, 749. A mean of 93.75%, and the whole spread is three items.
That looks like a beautifully stable system, and here is the part that matters: 38 distinct items, 4.8% of the set, flipped somewhere across those five passes. They cancel out. The worst pair of passes disagrees on 23 individual items while their totals differ by one.
Two numbers, and they say different things. The score is extremely stable: 0.375 points across five passes. The answers underneath it are not: on any given pair of passes, 2.00% to 2.88% of individual items come back differently.
We had a rule of thumb here that doubled the second number and used it as a threshold on the first. Our own reviewer killed it while reviewing this article, and it was right to: an item-flip rate and a score gap are different quantities, and the whole reason the score is stable is that the flips cancel. You cannot derive one from the other, and we have retracted it. The section further down shows that exchange in full, because it is a better illustration of the method than anything we could have written on purpose.
What the five passes do license is narrower and still worth having. If you are going to quote a single benchmark number, run it more than once, publish the spread, and publish how many individual items moved. A stable total is not a stable measurement, and only the second of those two numbers tells you whether the thing you are ranking sits still.
There is a second way the level story gets overstated, and it is worse. We took one model, kept everything identical, and changed only which company served it. On one model-and-provider pairing that moved the score by 17.3 points, from 78.2% to 95.5%. On a different pairing the same experiment moved 0.6 points, so this is a property of the specific pair rather than a law. But it means a cross-vendor ranking built on unpinned routing can be measuring the plumbing rather than the model, at a magnitude larger than the gaps it is reporting.
What a second model buys you instead
Now the other half, and this is the one we would defend hardest.
We put three models over the same 160 coding problems. Same prompts, same scorer, one run. They scored 148, 146 and 144 of 160, which is 92.5%, 91.25% and 90.0%. A 2.5 point spread, and by the logic above you should treat that ranking as noise.
Then we stopped reading the column totals and partitioned by problem instead.
Across two runs the three models disagree on roughly 15 to 18 percent of the problems, and 83% of the set is uncontested by all three. Most of the work is genuinely settled. On the remainder, models that a leaderboard calls equivalent are answering differently.
The obvious objection is that this is the same flicker wearing a costume, and it has to be answered on this set rather than by importing a number from a different benchmark. On these 160 problems the same model on the same problem across two identical runs flips 3.7%, 4.9% and 5.2% of the time. So the honest comparison is a contested rate of 15 to 18 percent against a self-flip rate of up to 5.2 percent, which is a factor of roughly one and a half to two and a quarter depending on which pair you take. Real, and considerably less dramatic than it would look if you reached for a flip rate from some other benchmark, which is a thing we have now done twice and been caught doing twice.
So the size of the disagreement is not the argument. The argument is that ten problems are contested in both runs with zero flips by any model. Each model is stably right or stably wrong on them, run after run.
We had a bigger-sounding version of this, resting on how improbable the overlap of contested sets was under a null where every problem is equally likely to be contested. Our reviewer killed that null, correctly: easy problems are essentially never contested, both runs draw from the same medium band, so the sets overlap with no family specialisation at all. It supplied a difficulty mixture reproducing most of the overlap. We checked the arithmetic and it holds, so that p-value is gone.
Then we ran the test it should have been, because the objection is answerable rather than merely conceded.
Hold each problem's difficulty fixed and destroy only the model-specific part. For every problem take its observed success rate across all three models and both runs. That number is its difficulty, measured, with no mixture assumed and no distribution fitted, which makes it immune to the objection above whatever the true spread turns out to be. Then resample every cell as an independent coin weighted to that rate, and count the stable-contested problems again. Twenty thousand draws.
| | |
|---|---|
| observed, contested in both runs with zero flips | 10 |
| same statistic under the difficulty-preserving null, mean | 2.0 |
| the most the null ever produced in 20,000 draws | 9 |
| p | 0.00005 |
Difficulty makes all three models wobble on a problem. It does not make one model reliably right and another reliably wrong on that same problem, twice. The claim survives a null built specifically to satisfy the objection to the previous one.
Here is the whole residue, named, on 154 problems both runs scored:
| problem | gpt-oss-120b | qwen3-235b | qwen3.8-max |
|---|---|---|---|
| HumanEval/9 | pass | fail | fail |
| HumanEval/10 | pass | fail | pass |
| HumanEval/22 | pass | pass | fail |
| HumanEval/26 | pass | pass | fail |
| HumanEval/86 | fail | pass | pass |
| HumanEval/115 | pass | fail | pass |
| HumanEval/125 | pass | fail | pass |
| HumanEval/132 | fail | pass | fail |
| HumanEval/145 | fail | fail | pass |
| HumanEval/160 | pass | fail | pass |
Read down the columns. Every model is stably right somewhere another is stably wrong, and every model is the sole failure somewhere. gpt-oss-120b owns five of these and is the lone loser on two. qwen3.8-max is the only one that solves 145. None of the three is redundant, and none of them dominates.
That is ten problems out of 154, which is 6.5%. It is a much smaller number than the disagreement rate, and it is the one we can defend. The self-flip rates on this same set are 3.9%, 4.5% and 5.2%, measured here rather than imported.
The part we have not earned, and will not claim
If three models disagree on that many problems, then something that could always pick the right one would score above any of them individually. On the 154 problems both runs scored, the best single model gets 145 and any-of-three gets 152, so that ceiling is 4.5 points.
We are not claiming those six points. Reaching them requires an oracle that knows which model was right, and an oracle is precisely the thing you do not have at the moment you need it.
We know this the expensive way. We built the obvious mechanism to capture that headroom, ran it end to end, and captured none of it. The cheap arm passed its own checks almost every time, so the second opinion was rarely consulted, and the result landed exactly where that first arm already scored. The headroom was real and our design could not reach it by construction rather than by tuning.
So the honest claim is narrower than the exciting one, and it is still the useful one: the gap between models is a real, reproducible, addressable thing, and the value shows up when a second model actually looks at the first one's work. Which brings us to the part we can put numbers on from our own three weeks.
So what do we actually run, and why in those seats
Three models, fixed roles, and the roles were chosen by failure mode rather than by tier.
Claude orchestrates. Specs, review, decisions, and the commits. It holds a very large context, which means it can carry the codebase, the written constitution and the current spec at the same time and notice when two of them disagree. That is the job.
Where it is weak, measured on itself: it agrees with itself. Given a shortcut it will produce a reason for the shortcut, and the reason will be fluent. Our own standing rule says substantive work runs as three, and the rule exists because the orchestrator twice decided a piece of work was "small and mechanical" and did it alone. Both times the cost showed up within the hour, in public once. An orchestrator that also implements ends up grading its own homework, which is why it does not implement.
Codex implements. It works from a written spec inside a sandbox, and it is good in direct proportion to how exact the spec is. Vague spec, confident wrong build.
The interesting part is a limitation we stopped trying to remove: it cannot commit. Its sandbox has no path to our history. That forces a review step between "the code exists" and "the code is in the repo", performed by something that did not write it. We could grant it commit rights. We have chosen not to, because the gap is where the checking happens.
Grok reviews, adversarially, and verifies across multiple hops. Different family from both of the others, which by the argument above is the entire point. It is strongest before something is built, when the thing under review is a design and there is still time for the answer to be "do not build this". Three times in three weeks it stopped something broken from shipping, and one of those was a token minter where we had handed the verifier its own minting key.
Where it is weak is documented below in detail: it is confidently wrong at a rate we can measure, and it attacks whichever leg you failed to brief it on, with exactly the same confidence it brings to the legs it has evidence for.
And the fourth seat is a person. Every pivot that mattered over three weeks came from the founder rather than from any model: ask another mind, is that apples to apples, what does the source actually say. The models are good at execution inside a frame. Choosing the frame, and noticing when the whole frame is wrong, has not once come from them. Any description of this setup that leaves the human out is describing something that would have drifted weeks ago.
The brief does more work than the model choice
This is the part we would keep if we had to throw everything else away.
Ask any model to "review this" and you get a compliment. It will find something small, agree with your framing, and hand the whole thing back improved by three percent. The model is not the problem. The request is.
So our brief names what we want destroyed, in those words. Here is the shape of a real one, from a review that came back DO NOT BUILD:
You are reviewing a design. Your job is to attack it, not to compliment it.
The five things I most want destroyed, in these words:
- The central claim may be oversold. [the claim, and the measurement behind it]
- [The load-bearing beam.] This is the strongest argument; if it breaks, tell us.
- The evidence may prove a different thesis than the one claimed.
- [The weakest data.] Make the strongest possible case that this is worthless.
- [The newest, least-reviewed part.]
Also answer: what is missing, what should be cut, and where are you systematically weak and
should not be trusted.Rules: re-derive any step you challenge rather than asserting it. Separate findings from
insinuations. If you cannot check something from what is here, say "I cannot check this from
the brief" rather than guessing. Name which of your objections you are least sure of.
Every line there is load-bearing. Naming the load-bearing beam invites the review to aim at the thing you would least like to lose. Asking where the reviewer is weak gets you a calibration you cannot get any other way. And "name which objection you are least sure of" is the one that turns an undifferentiated wall of confident findings into something you can triage.
That last instruction exists because of the failure described in the next section: a reviewer attacks the leg you failed to brief it on, with exactly the confidence it brings to the legs it has evidence for.
A small, dull example, because the dramatic ones prove less
We would rather show you the cheapest possible case than a war story, because the cheap case is the one you will actually recognise.
Today the task was posting four comments. No design, no architecture, no judgment. The kind of chore where running a review feels absurd.
We skipped the check and did it alone. What that produced, measured from the logs:
| replies intended | 4 |
| posting attempts made | 17, over 11 minutes |
| landed correctly | 1 |
| landed and should not exist | 1 |
| never posted | 2 |
| permanent public artifact | 1 comment now reading "posted by mistake" |
The queue was shared and held old rows that had already been posted days earlier. Draining it fired those too, and one went out as a word-for-word duplicate of something we had already said in that thread. Fixing it took a browser session, an edit to a live comment, and an API check to confirm the edit landed.
The check that would have prevented all of it was reading the queue before draining it. One command. Under a second.
Two honest caveats, and they matter more than the anecdote.
That is not a typical solo result. Most shortcuts cost nothing at all, which is exactly why they keep getting taken. Anyone claiming their process catches an error every time is selling something.
And the comparison is not "team good, solo bad" on average. It is about the shape of the distribution. The check is cheap, bounded, and always the same price. The failure is rare, unbounded, and in this case irreversible and public. You are not buying a better average. You are buying a shorter tail, and the reason it is worth doing on a chore is that chores are exactly where the check feels too expensive to bother with.
The guard now exists, so this specific hole is closed. That is the other half of the discipline: every manual finding leaves something automatic behind, or you will find it again.
And one seat is not filled by a model either. Every finding gets re-derived against the source by whoever is about to act on it. That step is the product. Everything above it is logistics.
We kept a ledger of every time the reviewer was wrong
Cross-model AI review works, and that part is settled by people with bigger samples than ours.
Greptile ran 1,000 pull requests in July 2026 and found that Claude Opus catches 53.7% of bugs in its own code and 62.0% in GPT-authored code, with GPT-5.5 showing the same asymmetry in reverse. Milvus put five different-family models through five rounds of adversarial debate on 15 real production bugs and moved detection from 53% to 80%. Cloudflare runs a role-tiered, cross-vendor reviewer in production. If you want evidence that a different-family reviewer catches what the author model misses, go read those.
We have something smaller and, we think, differently useful. For the last three weeks we have kept a ledger. Every time our reviewer returned a finding, we re-derived it against the source before acting, graded it, and wrote down the grade. Including the times the reviewer was wrong.
That last column turns out to be the interesting one.
The numbers
Sixteen adversarial reviews between 2026-07-27 and 2026-08-16. Fourteen ran before the thing shipped, two after.
| count | rate | |
|---|---|---|
| reviews on file | 16 | 21 days |
| ran pre-ship | 14 | 87.5% |
| changed the artifact | 14 | 87.5% |
| contained a finding that was wrong on re-derivation | 13 | 81.3% |
| would have shipped something broken without the review | 3 |
Read those two bold rows together, because separately each one tells a lie.
The first says the review is worth running. Fourteen times out of sixteen, the artifact that shipped differed from the artifact we brought to review.
The second says the review cannot be obeyed. Thirteen times out of sixteen, at least one finding fell apart the moment we went back to the source. Not stylistic disagreements. Confident, specific, technical claims that were false.
What the wrong ones looked like
A P0 that died to one curl. A security review ranked identity forgeability as an unverified critical issue. We sent an anonymous request and a forged-JWT request at the live endpoint. Both returned the sign-in page, zero data. The finding was ranked highest and was empty.
A missing measurement we had already taken. A pre-publication review objected that we had never measured end-to-end task latency. We had: p50 0.85s, p90 1.36s, same 800-task run, already in the ledger. Deferring to that finding would have made us withdraw a defensible public claim. The reason it happened is worth naming, because it was our fault: we had not briefed that leg. A reviewer briefed on a subset will attack the gap you made and will sound exactly as confident there as everywhere else.
A mechanism that was never there, hiding a worse bug that was. A migration review's central high-severity finding described a specific line of code doing a specific thing. That line does something else entirely. But going to the source to check produced a different and more serious defect: two functions resolve a caller's tenant by reading different database columns, so a partial migration would put one principal in two tenants at once. Nobody had that going in. The wrong finding was worth more than a right one would have been, because chasing it down led somewhere real.
What the right ones looked like
Three times the review stopped something broken from shipping.
A pre-build review caught that consolidating keyboard handling by focus would steal the Enter key from the terminal, so the shell would get no newline or a doubled one. The correct shape came back in the reviewer's own words and went into the spec verbatim.
A migration review said do not flip one tenant flag: all 8,116 cache rows were world-scoped, so flipping it would stop them matching. The important part is the failure mode. That regression surfaces as a bigger bill, not as an error, so monitoring that watches error rates stays green throughout.
A pre-ship review of a token minter found we had handed the verifier the minting key. The binding we were proud of was policy on one writer, while the cryptography permitted many. The patch was held. The design went asymmetric.
One worked example, start to finish
Here is a full instance, so the protocol is visible instead of merely described.
The problem. Our own standing rule says substantive work runs as three. One Sunday the orchestrator went solo on two implementation items anyway and told itself they were "small and mechanical", which is the exact rationalisation the rule exists to block. The founder asked: "How do we make the team use happen all the time? Is it something I have to continue to remind you of?"
A standing rule a human keeps re-supplying was never installed. It is only being remembered, and remembering is the thing that resets. So: build a gate.
Step one, refuse to guess. A hand-written test restates your own mental model as data and can only confirm it. So we extracted every file-editing action from 163 sessions of transcript history, 210,438 lines, giving a corpus of 5,037 real edits, and tested candidate definitions of the lane boundary against what the agent had actually done.
| candidate definition of "implementer lane" | share of 5,037 edits blocked |
|---|---|
| anything not clearly orchestrator work | 51.9% |
| restricted to implementer roots and code file types | 51.0% |
| the same, minus measurement and benchmark scripts | 42.2% |
| the same, plus "no other agent was dispatched this session" | 27.0% |
| the same, counted once per file instead of once per edit | 6.2% |
The mental model was wrong. One scripts directory held 34% of all edits across 369 distinct files, mixing production serving code, guards, benchmarks and ops automation with no prefix separating them.
The obvious rival is that product code drives that rate and the scripts folder rides along. We tested it by dropping that directory from the definition: 51.0% falls to 16.9%, so it supplies 67% of the block volume by itself. One minute of work, and now the claim has a test under it.
The last row felt like the answer. 178 edits to one file is one decision to go solo, not 178. Change the accounting unit and an unshippable rate becomes about two prompts per session.
Step two, the review. The design went out with the measurements attached and an instruction to attack it. Asking a model to "review this" gets you a compliment, so the brief listed the five things we most wanted destroyed, in those words.
It came back DO NOT BUILD, with two findings we graded correct.
The dispatch condition was a laundering mechanism: survey, then decide, then implement is the normal shape of a working session, so after any unrelated dispatch the gate goes dark for the day. We measured it before agreeing. Removing that condition moved the rate from 1.91 to 6.03 fires per session. It had been hiding 357 of 669 events, which is 53% of the entire livability argument. The number proving the design was safe rested on the component we trusted least. Circular, and we missed it.
The second finding was about us. We picked the counting unit after looking at the rate table. The principle behind it holds, a decision does happen once, but we arrived at it while hunting for a number that worked.
Step three, the reviewer was wrong too. Its prescription was to stop inferring the boundary and block a fixed list of unambiguous implementer directories instead. Decidable, mechanical, no judgment.
We ran that against the same corpus. It fires 4.78 times per session, and it misses one of the two real incidents that caused the task to exist.
That is the finding of the day, and neither side had it going in:
The failures live precisely in the zone that no decidable rule reaches. A boundary crisp enough to be
mechanical cannot cover the ambiguous middle, and the ambiguous middle is where things actually go wrong.
Two designs, both measured, both dead. We built an observer instead: it blocks nothing, logs every event it would have flagged, and pre-registers all the candidate counting units so that post-hoc selection stops being available to us.
We also owe the reviewer a note on what we left out of the brief. We never told it our hooks support a shadow mode, which materially weakens its strongest objection, because livability then stops being a matter of argument and becomes something you measure in production for free.
The rule underneath all of it
One line, from the founder, in July:
"Don't trust someone else's work, because ours still has issues, why should we think theirs doesn't."
That became a constitutional rule here: an outside review is evidence, not a verdict. You re-derive the step the conclusion turns on. You separate the finding from the insinuation. You name the legs you did not brief, before you decide whether an objection is real or an artifact of your own briefing.
The asymmetry it exists to correct is the easy one to fall into: hostile scrutiny for our own numbers, courteous acceptance for someone else's. We had three instances of that in a single hour once, which is how the rule got written.
And it runs both ways. A review that agrees with you earns the same scrutiny as one that argues. External agreement feels like independent verification, which is exactly how it slips past the check it resembles.
We reviewed the review
Everything above describes a protocol. Here it is running, on this article, with the grades.
We sent this draft to the reviewer with a brief telling it to destroy five named things. It came back with a verdict: do not publish this thesis with this evidence. Then it listed findings.
The temptation at that moment is the whole subject of this piece. A confident, specific, well-argued rejection from a different model feels like a verdict, and the pull is either to obey it or to dismiss it. Both are the same mistake, which is treating the review as an answer rather than as input.
So we graded it. Every finding, re-derived against the source before deciding.
The findings we accepted
It found a defect that reached past the draft into our own source of truth. The article claimed a benchmark gap under about 5.8 points "is not a finding." The reviewer called that a category error: the number is twice an item-level flip rate, being used as a threshold on a score gap, and those are different quantities.
We opened the script the rule comes from. Its two-times rule governs the ratio of between-model disagreement to the within-model noise floor, and it prints a refusal when that ratio fails to clear two. It has nothing to say about leaderboard gaps. And our own paragraph refutes the claim without any help: across five passes the score moved 0.375 points while 4.8% of items flipped, because flips cancel.
Correct, and worse than the reviewer knew. That sentence did not originate in the draft. It came from a row in our verified-claims ledger, a row that had been cleared for public citation. The reviewer was attacking a line of prose. The defect was upstream, in the thing the prose was quoting, and it needed a correction at the claim.
It caught us committing an error we had already retired. The opening contrasted between-model disagreement of roughly 15 to 18 percent against within-model flicker of 2.00 to 2.88 percent. Those come from different benchmarks with different task distributions and different arity: one is a three-model contested rate on 160 coding problems, the other a pairwise flip rate on 800 tool-calling items.
That is the same error class we struck from the ledger in August, when an earlier version of this comparison divided a three-way rate by a pairwise one. We own the correct figure. On the same 160-item set the within-model flip rate is 3.7, 4.9 and 5.2 percent, which puts the like-for-like comparison at 1.50 to 2.25 times rather than something dramatic, and our own adversarial pass had already noted that this straddles the refusal line.
The reviewer re-derived that with no access to our ledger, our scripts or our artifacts. It had the article and nothing else.
It broke the load-bearing beam. Our strongest argument was that the contested problems reproduce across runs, 17 overlapping against 3.6 expected by chance, p = 1.5e-12. The objection is that the null assumes every problem is equally likely to be contested, which is false on arrival, because easy problems are never contested and both runs draw from the same medium band.
It supplied an existence proof. Thirty problems contested at probability 0.7 plus 130 at 0.023 yields the same overall rate and an expected overlap of 14.8 against our observed 17. We checked the arithmetic. It holds.
What survives is the smaller claim we already had: ten problems are contested in both runs with zero flips by any model. Difficulty alone cannot produce stable opposite answers. Only an interaction between a specific model and a specific problem can. That is the real residue, it is 6.25% rather than 15 to 18, and it should have been the beam all along.
The findings we rejected, and why
"91.2% is impossible on 160 binary items." The arithmetic is right, 91.2% of 160 is not an integer, and the conclusion drawn from it, that either the scoring is not binary or n is not 160, does not follow. It is 146 of 160, which is 91.25%, printed to one decimal. The repair is to publish counts alongside percentages, which we should have done anyway. The reviewer reasoned correctly from the page and reached past what the page could support.
"You never report the within-model flip rate on that set. You have the data." True as an observation, and the word "hiding" that followed it is an insinuation rather than a finding. That number is published in our own ledger, and it was withdrawn from public use by our own adversarial pass months before this draft existed. Leaving it out of the article is an editing failure. It is not concealment, and the difference matters, because one of those is a bug and the other is an accusation.
And a large share of what it could not check is our fault, not its. We sent the article and nothing else. No ledger rows, no scripts, no artifacts, no methods appendix. Its repeated "I cannot check this from the brief" is an accurate report about a brief we wrote badly, and its demand for a methods section is partly an artifact of that gap. A reviewer attacks the hole you left. Ours left several.
What grading actually costs, and what it buys
Settling two of those findings meant opening a script and a ledger row. That is perhaps twenty minutes. It is also the entire value of the exercise, because the accepted and rejected findings arrived in the same voice, at the same confidence, in the same list.
That is the point we would leave you with, and it is the one thing here that the larger studies do not measure. The published work on cross-model review reports how many real defects a second model catches. It does not report what the second model said that was wrong, because that column is only visible if someone re-derives every finding before acting on it.
We keep that column. Today it read five accepted, three rejected, and one of the five reached further than either of us expected.
And the recursive part is not a joke at our expense. The error the reviewer caught in the opening section was written today, by the orchestrator, inside the section arguing that a second model is worth the trouble.
What does not need three, which is most things
If the previous sections read as an argument for running everything past a committee, we have written them badly.
Most work does not need this. Renaming a variable, adding a log line, writing a test for behaviour you just specified, fixing a typo in a doc, any change whose failure mode is that it does not compile. A second model on those buys nothing and costs tokens, latency, and a small amount of your patience, and the practice will be abandoned within a week if you apply it there.
The filter we use is not size, and it is not risk in the abstract. It is this:
Would being wrong here be visible?
Work where the failure announces itself, a crash, a red test, a page that will not load, does not need a second opinion. You already have one, and it is faster and cheaper than any model.
Work where being wrong looks exactly like being right is where the whole practice earns out. Concretely, in three weeks, that meant: anything whose failure surfaces as a bigger bill rather than an error. Anything where a check might be passing because it cannot fail. Any security or identity boundary, where the successful case and the compromised case return the same status code. Anything irreversible or public. And any claim that is about to be published with a number attached.
That list is short on purpose. Rigour costs real money and real time, and the lever is never to be less rigorous. It is to be rigorous about fewer things, chosen deliberately, and then to go all the way on those.
What we are not claiming
We make no claim that three models beat one in general. We have never run that experiment.
We claim no novelty for cross-model review. Greptile and Milvus measured it first and at larger scale, and if the aggregate case is what you need, theirs is better evidence than ours.
Two honest limits on our own numbers. The corpus is self-selected: these are the reviews we chose to write up as nodes, so reviews that changed nothing may simply never have been recorded. And "artifact" here spans specs, designs, public claims and marketing copy, so the strict code-or-design cut is 9 of 10, against 14 of 16 overall.
And the premise underneath the worked example belongs to someone else too. "A blocking gate teaches where an advisory rule fails" was published by TRACE in June 2026, measuring advisory rules in context at 55.0% against hook-enforced blocking at 70.1%. Ours would have been a replication at best. What we actually found is that for a boundary requiring judgment we could not build the instrument at all, which is a smaller and stranger result than the one we set out to get.
The cost is real. Reviews take tokens, briefs take care, and re-deriving a finding means doing the measurement twice. Cutting rigour is the wrong lever. The right one is choosing fewer things to be rigorous about, deliberately, then going all the way on those.
The two lines worth keeping
On building guards: before you narrow a noisy guard's matcher, ask whether its unit is wrong. Everyone tunes the pattern. Almost nobody asks what one event is supposed to represent.
On running a team of models: the value is in the re-derivation, not the verdict. A reviewer you obey is an oracle you have not tested, and ours was wrong in 13 of 16 reviews while still being worth running every single time.
Top comments (0)