Every team I've worked with keeps a mental leaderboard of who's worth asking for a review. It's built from the same four columns every time: how much you review, how fast you turn it around, how rarely your review asks for changes, and how long you've been around. All four are cheap to measure. All four are, according to a paper I read this week, pointed almost exactly backwards.
The study looks at 541,621 reviewer events in a human-AI cognitive system — humans reviewing model output that had verifiable ground truth. The behavioural profile, the one your team's gut ranking is built on, identified a cohort whose corrections survived contact with reality 5.2% of the time. Genuine experts, ranked by whether their corrections actually held up, survived at 99.2%. Same pool of people. Two scoring functions. One of them is describing a completely different population than the one you think you're ranking.
I've been on both sides of that inversion. I've been the reviewer with a fast turnaround and a high approval rate, and I've shipped things I should have caught. It's a bad feeling to read a paper that explains your own worst quarter.
What the proxy profile actually measures
Take the four columns apart.
Speed selects for people who don't read carefully. There is essentially no version of careful review that is fast, unless the diff is three lines. If you rank on turnaround time, you are optimising for skimming and calling the result responsiveness.
Low revision count selects for people who approve. A reviewer who asks for changes is doing work. A reviewer who never asks for changes is either looking at unusually clean code or is not looking. Your metric can't tell those apart, so it rewards the second one.
Tenure selects for people who were around before the codebase changed underneath them. It's a proxy for context, and context rots.
Activity selects for capacity. Someone with slack in their week reviews more. That's a scheduling fact wearing judgment's coat.
All four are cheap to compute from data you already have. That's the entire reason they're popular. Not because they predict anything.
Calibration is the only signal that survives the gate
Here's the comparison that matters, and it's not close.
The proxy profile is a preference measure: it tells you who shows up and who agrees. Calibration is an accuracy measure: how often the things you flag turn out to be real, compared against how confident you were when you flagged them. Those are different quantities. One of them is available today and free. The other needs ground truth, and ground truth is expensive, which is why almost nobody computes it.
The gate is the second half of the trick, and it's the part I keep coming back to. Calibration on its own is just a better-stated opinion. What the paper does is require the flag to survive verification before it counts. A reviewer who says "this is definitely broken" and is wrong takes the same hit as one who hedges and is wrong, and that's correct — confidence is free, survival isn't.
Apply the same gate to the model and the number gets uncomfortable: 92.5% of human review effort landed on real errors. Not because the model got smarter. Because everything it emitted was filtered by whether the flagged issue held up, rather than by whether the model sounded sure.
The part I got wrong in my own stack
I've been using model self-confidence as a routing signal for months. Low confidence goes to a human, high confidence ships. That's an uncalibrated reviewer ranking with an LLM on it. The model is exactly the cohort that scores 5.2% on survival while sounding like it scores 99.2%, and I built a router that trusts the sound.
The fix isn't a better confidence threshold. LLM confidence is a token-level artefact, not a belief. The fix is a verification step that produces an independent signal — run the check, hit the endpoint in staging, replay the input against a known-good output — and then let that decide whether a human sees it.
What I'd actually build
Three things, in order of how much they hurt.
- A golden set of known defects and known non-defects. A few dozen seeded bugs in a staging environment, plus a few dozen plausible-looking non-issues. Without this, you can't compute calibration for anyone, human or model. It's the cold-start problem and it's the reason nobody does this.
- A per-reviewer survival score. Every flag gets an ID. Every flag gets resolved: real, not real, abandoned. Abandoned counts against you, because it means nobody ever checked. Score is survived flags over raised flags, weighted by how confident the reviewer was.
- Retire speed as a quality metric. Keep it as a cost input, the way you'd track token spend. Never let it into a ranking.
The objections I can't fully answer
A small golden set is gameable. If I know which twenty bugs are seeded, I can find those twenty and look brilliant. That's real, and it's why the paper needed half a million events rather than my fifty. The mitigation is churn — rotate the seeded set, keep part of it hidden, and treat calibration scores as a trend rather than a number. It's weaker than what the paper did. It's still strictly better than ranking on who replies fastest.
The other objection is social. If you tell people you're scoring their corrections by survival rate, the ones who ask for changes get nervous and the ones who rubber-stamp stop being visible. I think that's the point. The people who should be nervous are the ones I've been promoting.
I haven't deployed this yet. I've deployed the version of it that trusts model confidence, and I now think that was a mistake I can measure. The paper gives me the measurement.
Top comments (0)