DEV Community

Nicolas Rocchia
Nicolas Rocchia

Posted on

Your AI code reviewer approved the PR. What did it leave open?

The green checkmark is a strange object. It says that a process ran and nothing blocked the merge. It does not say what the process saw, what it argued about, or what it gave up on. With AI code review this gets worse, because the review produces a lot of findings, the findings get resolved or dismissed inside a chat window, and then all of it evaporates. What survives is the checkmark.

For the past few months I have been working with a setup that takes the opposite bet. I call the method controlled disagreement. One coding assistant generates the plan or the code. A second assistant, from a different model family, attacks it under an explicitly adversarial prompt. The generator has to answer every finding, and the cycle only ends when each one is incorporated, refuted with evidence, or escalated to a human. No finding is allowed to just fade away.

The interesting part is what happens at the end of each cycle. The event gets written down in a small JSON artifact called a residue declaration. It records who generated, who attacked (family and model, with the hash of the adversarial prompt), every finding with its terminal state, and above all the residue: what the cycle could not close on its own and now rests on somebody's judgment.

A trimmed example, real structure:

{
  "schema": "residue/v0.3",
  "profile": "full",
  "event": { "gate": "diff", "base_commit": "...", "head_commit": "..." },
  "actors": {
    "generator": { "family": "anthropic", "model": "claude-code" },
    "reviewers": [{ "family": "openai", "prompt_hash": "sha256:..." }]
  },
  "findings": [
    { "id": "h1", "severity": "critical", "final_state": "incorporated", "title": "..." }
  ],
  "residue": { "items": [
    { "class": "escalation_without_decision", "finding_ref": "h2",
      "requires_human_attention": true, "description": "..." }
  ] },
  "metrics": { "counts": { "total_findings": 3, "escalated_open": 1 } }
}
Enter fullscreen mode Exit fullscreen mode

Disensor is the tool that makes this artifact enforceable. It is a Python CLI plus a GitHub Action. The validator applies rules R0 to R10 to each declaration: counts have to add up, generator and reviewer have to come from different families, verifiable refutations need material evidence, generic markers like "reviewed, all good" are rejected. The CI gate applies checks G1 to G9 to each PR: if the PR touches code it has to carry a valid declaration, the declaration has to cover the exact tree being merged (a stale one covers nothing), some declaration has to have seen the final integrated tree, and evidence is append only, so a PR cannot rewrite or delete the declarations that were already there.

Two design decisions I want to be upfront about.

First, disensor does not run any models. It does not need API keys and nothing of your code leaves the repo. It validates a JSON file that is already versioned next to your code. The orchestration of the review loop lives wherever you already work; if you use Claude Code, disensor init writes the integration for you (a CLAUDE.md section plus a skill), and any other agent gets the same instructions with disensor guide.

Second, the honest limit. The machine detects the empty field and the generic marker. It cannot detect a false declaration. If a model or a person lies in the artifact, only human sampling of merged PRs catches it. I would rather build around that boundary than pretend a tool can certify honesty.

Does the method hold up? I ran it for 52 days across 5 professional projects and wrote it up as an experience report: 91 documented review events, at least 64.9 percent of findings incorporated, 3.6 percent refuted with evidence. The paper is on Zenodo in Spanish and English: https://doi.org/10.5281/zenodo.21633495

And the repo eats its own cooking: every PR to disensor passes through the disensor gate, so the .residue/ directory contains the real declarations of its own development, including the rounds where the adversarial reviewer found holes in the schema and forced a version bump.

If you want to try it without touching your CI, this runs entirely on your machine and tells you exactly what the gate would say:

pip install disensor
disensor init --no-workflow
disensor prompt --gate diff
disensor new --gate diff --level B
disensor validate .residue/<id>.json
disensor gate --no-comment --base <sha> --head HEAD
Enter fullscreen mode Exit fullscreen mode

Site: https://disensor.dev
Repo: https://github.com/NicolasRocchia/disensor (MIT)

I am especially interested in two kinds of feedback: ways to break the gate, and whether the residue framing is useful to teams that are not mine.

Top comments (0)