A Patch Provenance Ledger: Make AI-Assisted OSS Patches Auditable in 60 Seconds

A maintainer receives a working patch with green tests. They still cannot judge it. The diff shows what changed, not why it was trusted.

The fix is a small, machine-checkable record that ships with the patch. Call it a patch provenance ledger. It answers four questions in one file: which environment, which failure, which evidence, which tooling.

This article builds that ledger by hand, validates it in CI, and explains where free model access fits without pretending the ledger proves correctness.

The core claim

Review effort scales with missing context, not with diff size. A ten-line patch with no provenance costs more review time than a sixty-line patch with a ledger. The ledger converts an unbounded trust problem into a bounded schema check.

What the ledger must answer

1. Environment. Commit SHA, language runtime, OS, dependency lockfile hash.
2. Red state. The command that failed before any code change, plus its exit code.
3. Green state. The same command after the change, same environment, new exit code.
4. Ownership of hunks. Every changed file maps to a specific failure or assertion.
5. Tooling. Whether a model produced any part of the diff, and if so which parts.

Item five is the one most AI-assisted patches omit. It is also the one maintainers care about most.

Step 1 — Capture the environment fingerprint

Run these commands on a clean checkout and save the output. The commands below are illustrative; run them yourself and keep the real values.

git rev-parse HEAD
git status --porcelain          # must be empty before you start
python3 --version
uname -srm
sha256sum requirements.lock 2>/dev/null || sha256sum package-lock.json

An empty git status matters. A patch produced on a dirty tree cannot be reproduced by anyone else.

Step 2 — Record the red run before touching code

Run the failing test on the untouched commit. Capture the command, the exit code, and the last failing assertion. Do not edit source yet.

set +e
pytest tests/test_parser.py::test_empty_chunk -q 2>&1 | tee /tmp/red.log
echo "exit=$?" >> /tmp/red.log

A red run that takes more than one attempt is a warning sign. Flaky reproduction poisons every later claim in the ledger.

Step 3 — Record the green run with the identical command

Apply the patch, then rerun the exact same command string. Byte-identical commands are the whole point. A passing run under a different flag set proves little.

pytest tests/test_parser.py::test_empty_chunk -q 2>&1 | tee /tmp/green.log
echo "exit=$?" >> /tmp/green.log

If the green run needs extra environment variables, list them in the ledger. Hidden prerequisites are the most common reason a patch fails on a maintainer's machine.

Step 4 — Write the ledger file

Commit the ledger next to the diff, for example as .patch-ledger.json. The schema below is illustrative and intentionally small.

{
  "repo": "example/parser",
  "commit": "<40-char sha>",
  "env": {
    "runtime": "python3.12",
    "os": "<uname output>",
    "lockfile_sha256": "<sha256 of requirements.lock>"
  },
  "red": {
    "command": "pytest tests/test_parser.py::test_empty_chunk -q",
    "exit": 1,
    "assertion": "IndexError: list index out of range"
  },
  "green": {
    "command": "pytest tests/test_parser.py::test_empty_chunk -q",
    "exit": 0
  },
  "hunks": [
    {"file": "parser/chunk.py", "covers": "empty input path"},
    {"file": "tests/test_parser.py", "covers": "regression test"}
  ],
  "tooling": {
    "model_assisted": true,
    "scope": "test name suggestion, ledger draft",
    "human_verified": ["patch", "test assertions", "green run"]
  }
}

The human_verified list is not decoration. It tells the reviewer which artifacts a person actually read.

Step 5 — Validate the ledger in CI

The ledger is only useful if something checks it. The validator below uses the Python standard library only. It is illustrative and has not been executed as part of this article; run it locally and adjust the rules to your repo.

#!/usr/bin/env python3
"""validate_ledger.py - illustrative, unexecuted."""
import json
import subprocess
import sys

REQUIRED = ("repo", "commit", "env", "red", "green", "hunks", "tooling")


def changed_files():
    out = subprocess.run(
        ["git", "diff", "--name-only", "HEAD~1"],
        capture_output=True, text=True, check=True,
    ).stdout.split()
    return {p for p in out if not p.endswith("ledger.json")}


def main(path):
    with open(path, encoding="utf-8") as fh:
        ledger = json.load(fh)

    missing = [k for k in REQUIRED if k not in ledger]
    if missing:
        print(f"FAIL missing keys: {missing}")
        return 1

    if ledger["red"]["exit"] == 0:
        print("FAIL red run exited 0; nothing was broken")
        return 1
    if ledger["green"]["exit"] != 0:
        print("FAIL green run did not exit 0")
        return 1
    if ledger["red"]["command"] != ledger["green"]["command"]:
        print("FAIL red and green commands differ")
        return 1

    covered = {h["file"] for h in ledger["hunks"]}
    uncovered = changed_files() - covered
    if uncovered:
        print(f"FAIL hunks without provenance: {sorted(uncovered)}")
        return 1

    print("OK ledger consistent")
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Wire it into a workflow step:

- name: Validate patch ledger
  run: python3 scripts/validate_ledger.py .patch-ledger.json

An honest way to check the validator works: delete the green block from a copy of the ledger. The script must exit non-zero. If it exits zero, the check is too weak.

Where free model access fits

MonkeyCode's free model access and free server option are relevant to three narrow jobs here. All three are low-risk text work, not patch authorship.

1. Test naming. Feed the red log and the changed function signature, ask for three candidate test names. Cheap, verifiable output.
2. Ledger drafting. Convert two log files into the JSON skeleton. A human then corrects the exit codes and the covers text.
3. Assertion second-read. Ask for edge cases the new assertion misses, such as empty input, non-UTF-8 bytes, or a single-element list.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free server matters mostly for job one and job three, which are short and bursty. Long test suites still belong on your own hardware or CI runner. Current token allowances, model names, and server specifications are set by the project and change over time. Confirm them in the official documentation before planning a workflow around them; this article deliberately quotes no figures.

Failure modes the ledger will not catch

• A test that asserts nothing. The ledger records exit codes, not assertion strength. A red-to-green flip on an empty test body passes validation.
• Overfitted patches. Green on one input does not imply green on the class of inputs. The ledger cannot replace a second test case.
• Stale fingerprints. A lockfile hash captured before a dependency bump silently invalidates the environment claim.
• Self-reported tooling. Nothing stops a contributor from writing model_assisted: false. The ledger raises the cost of hiding it, nothing more.

Mutation testing closes the first gap. It is also slow, so treat it as a spot check on the changed test file rather than a repo-wide gate.

Who should not use this

• Single-author repositories. The ledger duplicates information already in your head.
• Trivial documentation patches. Provenance overhead exceeds review value.
• Repos with heavy flakiness. Fix the flake first; the ledger will otherwise record noise as evidence.
• Security-sensitive patches. Public ledgers can leak reproduction details. Use a private tracker and link only a summary.

Start with one real patch. Write the ledger by hand, run the validator, and note how long a reviewer spends on the diff. If the answer drops, keep the practice for non-trivial changes and drop it everywhere else.