A Claim Ledger for AI-Assisted OSS Reviews

AI review is cheap. Verified review is not.

The AI conversation has moved from generation to verification. Many OSS patches now pass through a model before a maintainer reads them. The output often sounds correct. Some claims are correct. Others cite lines that do not exist.

This article describes a claim ledger for AI-assisted OSS review. The ledger checks each model claim against a diff and a test log. It rejects claims without local evidence. The model stays in a reviewer role.

1. Reproduce the bug before any model sees it

Start with a failing test. A reproducible failure gives the review a ground truth.

git switch -c fix/parser-timeout
python -m pytest tests/test_parser.py::test_timeout -q

Record the exit code and the full log. The failure must be stable across two runs.

set -o pipefail
python -m pytest tests/test_parser.py::test_timeout -q 2>&1 | tee /tmp/repro.log

A model cannot reproduce a bug from a vague description. It can review a patch against a known failure. Keep the reproduction step human-owned.

2. Patch small and capture the evidence

Make the smallest change that fixes the failure. Avoid unrelated cleanup in the same patch. Review quality drops when the diff mixes concerns.

git diff --binary > /tmp/patch.diff
python -m pytest tests/test_parser.py -q 2>&1 | tee /tmp/test.log

Run the narrow test first. Then run the package suite. Save both logs. The model will see a capsule, not the whole repository.

The capsule has three files: the patch, the test log, and a short note about the expected behavior. This boundary matters. It limits context and reduces accidental disclosure.

3. Ask a free model for claims, not prose

MonkeyCode is an open-source project with free model access. Its free access fits this step as a reviewer, not an author. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes the current offer as 10 million free tokens and a free server option. Verify current terms before relying on either. Free access can change. The workflow must still work when the free tier changes.

Use a narrow prompt. Require JSON. Require file and line evidence.

You are reviewing one OSS patch. Use only the diff and test log. Return a JSON array. Each item needs claim, file, line, evidence, severity. Use an added line number. Do not give general advice. Mark uncertainty as low confidence.

The model should return claims like this.

[{"claim": "The new timeout path can return None.", "file": "src/parser.py", "line": 88, "evidence": "return None", "severity": "bug"}]

Only added lines are valid. The script ignores removed lines. A claim about an unchanged line is not a review of this patch.

4. Validate every claim with a ledger script

The script below reads the patch, the review, and the test log. It accepts a claim only when the file and line are in the diff. It also requires the evidence string to appear in the patch or the log.

import argparse
import json
from pathlib import Path


def changed_lines(patch_text):
    current = None
    new_line = 0
    out = {}
    for raw in patch_text.splitlines():
        if raw.startswith('+++ b/'):
            current = raw[len('+++ b/'):]
            out.setdefault(current, set())
        elif raw.startswith('--- a/'):
            current = None
        elif raw.startswith('@@'):
            part = raw.split('+', 1)[1]
            new_line = int(part.split(',', 1)[0])
        elif current and raw.startswith('+') and not raw.startswith('+++'):
            out[current].add(new_line)
            new_line += 1
        elif current and raw.startswith('-') and not raw.startswith('---'):
            continue
        elif current and not raw.startswith(chr(92)):
            new_line += 1
    return out


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--patch', required=True)
    ap.add_argument('--review', required=True)
    ap.add_argument('--tests', required=True)
    args = ap.parse_args()

    patch = Path(args.patch).read_text()
    tests = Path(args.tests).read_text().lower()
    claims = json.loads(Path(args.review).read_text())
    changed = changed_lines(patch)
    bad = 0

    for c in claims:
        f = c.get('file', '')
        line = c.get('line', -1)
        ev = c.get('evidence', '').lower()
        ok_file = f in changed
        ok_line = ok_file and line in changed[f]
        ok_ev = len(ev) >= 6 and (ev in patch.lower() or ev in tests)
        if ok_file and ok_line and ok_ev:
            print('ACCEPT', f + ':' + str(line), c.get('claim', ''))
        else:
            bad += 1
            print('REJECT', f + ':' + str(line), c.get('claim', ''))

    raise SystemExit(2 if bad else 0)


if __name__ == '__main__':
    main()

Run the ledger after the model returns its review.

python review_ledger.py --patch /tmp/patch.diff --review /tmp/review.json --tests /tmp/test.log
echo $?

Exit code 0 means every claim has local evidence. Exit code 2 means at least one claim failed the check. The ledger does not prove the patch is correct. It proves that the review is checkable.

5. Use a decision table before merging

Ledger output	Human action
All claims ACCEPT	Read each claim, then review design.
One claim REJECT	Discard that claim and ask for a narrower review.
Many claims REJECT	Stop the model step and inspect the capsule.
Evidence matches but risk is high	Add a test before merge.

A clean ledger is not an approval. It is a short list of claims that deserve human attention. The maintainer still owns the merge decision.

6. Know the limits and the wrong fits

The ledger checks local evidence, not semantics. A claim can cite a real line and still be wrong. Line numbers drift after a rebase. Test logs can miss race conditions and integration failures.

Do not send secrets, private code, or customer data to a free model. Do not treat the ledger as a security audit. Do not use this workflow for large refactors without a stronger review process.

Some teams should not use this approach. Skip it when the project lacks reproducible tests. Skip it when the patch changes authentication or payment code. Skip it when the reviewer cannot inspect the model output directly.

7. Place the free server carefully

The free server option can run the ledger and the test runner. Keep repository credentials off that machine. The model should see the capsule, not the server.

A free server is useful for a repeatable check. It is not a trust boundary. Run the same script locally when the result matters. Compare both outputs before merge.

A small ledger turns model review into a checklist. That is a better use of free access than another broad prompt. Try it on a documentation patch first, then on a narrow bug fix.