Update: a reader pushed back on Gate 1 — a fresh-context LLM verifier still shares the generator's prior, so "refuting harder" only helps if the ve...
For further actions, you may consider blocking this person and/or reporting abuse
Excellent point. AI code review has huge potential, but reliability is the real challenge. Combining model feedback with deterministic checks, tests, static analysis, and clear validation gates feels like the right approach. AI should augment engineering judgment, not replace the verification processes that keep software safe and maintainable. Great breakdown of the practical limitations and solutions!
Thanks, "augment judgment, don't replace verification" is exactly the framing. What I've come around to is that the verification layer is where the leverage is, not the finding layer: LLMs are already decent at "this looks off," but "is it actually real, and did the fix stay in its lane" is what decides whether you can trust the output. Deterministic checks + tests as the gate, rather than a second LLM opinion, is where I think it all lands. Appreciate you reading it.
Great perspective. I agree that the verification layer is where AI code review becomes truly reliable. The combination of LLM reasoning with deterministic validation is a strong direction.
I’m exploring similar ideas around AI-assisted development workflows and would be interested in exchanging thoughts or potential collaboration opportunities. Would you be open to connecting on Microsoft Teams for a quick discussion?
The two-gate framing is useful because AI code review needs both semantic suspicion and mechanical proof. A model can notice the risky pattern, but tests, type checks, and policy checks need to carry the final decision.
Semantic suspicion and mechanical proof is a cleaner summary than my whole post 😄 That's exactly the division of labor: the model is good at recall (noticing the risky pattern), deterministic checks are good at precision (carrying the verdict). The failure mode of most tools is letting the recall layer also make the call. I just opened an issue to push CCA further down this path, have the verifier cite deterministic evidence, instead of reasoning to a verdict. Thanks for reading.
That division of labor is the cleanest way to explain it. Let the model widen the search space and let deterministic checks narrow the verdict. Mixing those roles is where a lot of AI review tools become confident but unreliable.
The verification step is the part I keep coming back to. I’ve seen AI reviews create more noise than value when nobody checks if the issue is real.
Making the model prove the bug before touching code changes the whole workflow.
Exactly, the shift is putting a gate between finding and fixing, so a hunch can't become a diff. Once the reviewer has to stand behind a finding before acting on it, the noise mostly never reaches your code. Thanks for reading!
The refute-biased verifier is the load-bearing gate and the asymmetry argument for it is exactly right. The trap to watch: if the same model that emitted the finding also runs the verification pass, it shares the prior that produced the false positive, so re-asking it tends to confirm its own hallucination ('yes, line 14, that does look null'). Refutation bias helps only if the verifier gets evidence the generator didn't have. So the question that decides whether Gate 1 actually holds: does the verifier get anything deterministic (resolved control-flow/dataflow to the cited line, the AST showing the guard three lines up, a failing test that reproduces the impact), or is it the same model re-reading the same diff with a sterner prompt? 'Prove the impact can occur' is strongest when 'prove' means an executable repro, not a second LLM opinion on the same text. A guard three lines up is something a reachability check settles for free and an LLM keeps re-litigating. How independent is the verifier's context from the auditor's?
Update for you: you were right, and it's shipped.
The verifier no longer rests on "another LLM couldn't refute it." Where a tool exists, the verdict is now carried mechanically: pyright settles definedness claims from real diagnostics, and a generated pytest repro, driven through the validated entry point, so guards are respected, settles crash/impact claims. A green repro is the proof; a non-repro escalates rather than silently refuting. The LLM is demoted to adjudicating only the residue, and it has to cite the facts it gathered. A CONFIRMED verdict now requires an artifact, else it's UNCERTAIN.
The honest ceiling, which you'd spot anyway: claim classification is still LLM-inferred, and for crash_impact the repro is a test an LLM wrote, nothing mechanical enforces the entry point. Semantic/business-logic claims stay LLM-adjudicated. What this does is shrink the LLM-dependent residue to the claims no tool can settle, and force even those to cite evidence.
Design: github.com/GiulioDER/cca-audit/blo...
Shipped in: github.com/GiulioDER/cca-audit/pull/6 (captured from your comment as issue #5)
Your comment is literally the reason this exists. thank you.
Fair, and this is the whole ballgame. What CCA does isn't a sterner re-read, though: the verifier is a separate agent with fresh context that re-derives the finding from source — it reads the whole file and the call graph via grep/shell, not the diff hunk the auditor flagged. That's how the guarded div-by-zero got dropped: the verifier traced the validator in a different, off-diff file the finding never used. So the evidence set differs from the generator's — it's not the same context with a colder prompt.
But you're right on the deeper point: it's still an LLM, so it still shares the prior — fresh context reduces self-confirmation, doesn't kill it. There's no deterministic reachability pass or executable repro as the gate's evidence yet, and that's exactly the hardening I want: make "prove" mean a reachability check for the guard + a generated failing test for the impact, and let the LLM adjudicate only the residue. Best comment I've had on this — thank you.
The refutation-leaning bias in Gate 1 is the detail that stands out to me. Most AI reviewers I've used treat a pattern match as a verdict, so they lean toward reporting more findings, not fewer, and a wrong fix ends up costing far more trust than a missed bug you catch on the next pass. The fix mapping idea matches what I actually want when I sit down with an agent's diff, knowing why a line changed matters more than a green test run. Curious how the verification pass scales on a large diff, does it start to cost close to a second full review?
Right a wrong fix costs more trust than a missed bug you catch next pass, and "why did this line change" beats a green test run. That's the whole bias.
Good scaling question, and the answer's better than I expected when I built it: the verification pass runs over the findings, not the diff. Dedup happens first (in the demo, 6 raw findings collapsed to ~1 root), and the gate verifies that shortlist, re-reading the code and call graph around each finding, not re-reviewing the whole change. So it scales with the number of distinct findings, not lines changed. The parallel auditors are the part that scales with diff size; the gate downstream is a fraction of that, not a second full review.
Where it does creep toward "second review" cost is a big diff that legitimately produces many independent findings and that's exactly where tiering helps: the FAST tier skips the gate entirely, and p1-only / no-fix let you bound it to the critical set. You pay the verification cost roughly in proportion to what's actually at stake.
The two-gate framing is useful because AI review needs both fresh context and deterministic evidence. A second model can reduce some bias, but it still needs receipts: diff scope, tests, commands, affected files, and explicit uncertainty where the system cannot verify the claim.