Score Agent Patches Only on Assertions They Did Not Write

A green check on an agent pull request is a mixed number. Frozen checks, new tautologies, deleted failures, and silenced flakes share one status bit. Score the patch only on assertions the diff did not author, and treat deletions as a penalty rather than a cleanup.

CI still answers a useful question: did this tree run? It does not answer the review question: did existing evidence still hold after the agent edited production code? Those two questions diverge as soon as the model can add tests, rewrite expect() calls, or drop a case that used to fail.

This article is a scoring workflow, not a philosophy note. The artifact is a small provenance scorer. It classifies assertions in the diff, runs a frozen corpus off the agent’s writable tree, leaves flakes unscored instead of pinning them, and emits a merge score a human can reject.

The mixed number

Most agent gates still publish three aggregates: tests run, tests passed, tests failed. The first two inflate when the patch introduces assertions that restate the new code. The third shrinks when the patch deletes a fixture that disagreed with the new code.

None of those aggregates record provenance. An assertion that existed on the target branch before the agent started is not the same object as an assertion the agent added in the same commit. A property that failed last week and vanished from the tree is not a fix. It is a missing witness.

You do not need a research benchmark to see the distortion. One added assert result == result moves the pass count. One deleted parameterized case moves the fail count. Both leave the production diff looking cleaner than the evidence is.

What counts as independent evidence

Independent evidence has three properties. It existed before the patch. The agent cannot write the expected value in the same change. The run happens on a tree the agent does not control.

That definition excludes several things teams still treat as proof. Snapshot files regenerated in the patch do not count. Golden JSON the agent rewrote does not count. “Unit tests” that only cover functions the agent introduced do not count toward the merge score, even if they are well written. They can still ship. They just do not vote.

Flakes do not vote either. A test that failed for non-deterministic reasons on the last independent run is unscored until it produces a stable result off-tree. Unscored is not skipped forever and it is not frozen in source. It is withheld from the numerator and the denominator.

Four-step scoring workflow

The pipeline below is labeled as a proposed harness. It does not claim production metrics. Wire it to your real test runner and fixture layout before you enforce it.

1. Classify every assertion in the diff

Parse the pull request against the merge base. Walk only test paths. Tag each added or removed assertion-like line. Keep the tags boring: preexisting, agent_added, agent_removed, human_added.

If your process cannot tell human edits from agent edits, treat the whole PR as agent-authored for scoring. That is conservative and reviewable. Guessing authorship from commit messages is not.

# proposed commands; adjust paths to your repo
git fetch origin main
BASE=$(git merge-base HEAD origin/main)
git diff -U0 "$BASE" -- tests test spec > /tmp/test.diff
python3 score_agent_pr.py classify /tmp/test.diff > /tmp/provenance.json

A minimal classifier can start with syntax, not AST heroics. Lines that match assert, self.assert, expect(, should(, or @given are candidates. Comments and string literals need a second pass so a docstring that mentions assert does not become evidence.

2. Run the frozen corpus off-tree

Copy the characterization corpus and the scorer into a directory the agent patch cannot write. Hash that directory. Run only that corpus against the patched application code. Do not collect tests that arrived in the diff.

# score_agent_pr.py — proposed harness, not a measured benchmark
from __future__ import annotations

import hashlib, json, re, subprocess, sys
from pathlib import Path

ASSERT_RE = re.compile(
    r"^\+.*\b(assert |self\.assert|expect\(|should\(|@given)\b"
)
REMOVED_RE = re.compile(
    r"^\-.*\b(assert |self\.assert|expect\(|should\(|@given)\b"
)

def sha256_tree(root: Path) -> str:
    h = hashlib.sha256()
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        rel = path.relative_to(root).as_posix().encode()
        h.update(rel + b"\0")
        h.update(path.read_bytes())
    return h.hexdigest()

def classify_diff(diff_text: str) -> dict:
    added = [ln for ln in diff_text.splitlines() if ASSERT_RE.search(ln)]
    removed = [ln for ln in diff_text.splitlines() if REMOVED_RE.search(ln)]
    return {
        "agent_added_assertions": len(added),
        "agent_removed_assertions": len(removed),
        "added_samples": added[:20],
        "removed_samples": removed[:20],
    }

def score(prov: dict, independent: dict, flake: dict) -> dict:
    total = independent["ran"] - flake["unscored"]
    passed = independent["passed"] - flake["unscored_passed"]
    if total <= 0:
        return {"merge": "reject", "reason": "no_independent_evidence", **prov, **independent}
    pass_rate = passed / total
    deletion_penalty = min(0.5, 0.05 * prov["agent_removed_assertions"])
    added_penalty = min(0.2, 0.01 * prov["agent_added_assertions"])
    value = pass_rate - deletion_penalty - added_penalty
    decision = "review" if value >= 0.95 and prov["agent_removed_assertions"] == 0 else "reject"
    if value >= 0.99 and prov["agent_removed_assertions"] == 0 and added_penalty == 0:
        decision = "pass_score"  # still needs human review of the production diff
    return {
        "merge_score": round(value, 4),
        "independent_pass_rate": round(pass_rate, 4),
        "deletion_penalty": deletion_penalty,
        "added_penalty": added_penalty,
        "decision": decision,
        **prov,
        **independent,
        **flake,
    }

The off-tree copy is the point. If the scorer reads fixtures from the same checkout the agent just mutated, characterization data is not frozen. Hash mismatch is a reject, not a warning.

This is the only place a hosted runner matters. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already generate candidate patches with free model access, run this scorer on a free server option so the frozen corpus and the flake ledger live outside the agent’s workspace. The scoring rules are the product. The host is just isolation.

python3 score_agent_pr.py hash ./oracle_corpus
python3 score_agent_pr.py run --app ./patched_src --corpus ./oracle_corpus --out /tmp/independent.json

Keep the corpus small enough to finish in a normal review window. Prefer hermetic fixtures with byte-stable inputs. Network calls, clocks, and unordered maps belong behind fakes or they will contaminate the independent lane the same way flakes do.

3. Leave flakes unscored, not frozen

Do not pin a flaky test to a last-known-green hash and forget it. That turns intermittency into a permanent hole in the independent lane.

Record the last N off-tree results per test id. If the results disagree, mark the test unscored. It stays in the suite. It does not increment passed or ran for the merge score. After a streak of agreeing independent runs, it re-enters the denominator automatically.

{
  "tests/invoice_roundtrip.py::test_tax_table": {
    "results": ["pass", "fail", "pass"],
    "state": "unscored",
    "reason": "disagreement_on_independent_runner"
  }
}

The ledger belongs next to the corpus, not in the agent branch. If the patch edits the ledger, reject. Agents are good at writing “this is stable now” comments. That is not a stability measurement.

4. Apply a deletion penalty

Removed assertions are the cheapest way to make a failing independent lane disappear. The scorer above charges a linear penalty and hard-rejects when any assertion-like line is deleted, unless a human relabels the removal.

Relabeling should be explicit and boring:

# HUMAN_ORACLE_REMOVAL: tests/tax.py::test_legacy_bracket
# reason: product no longer ships 2019 brackets; see issue 1842

Without that marker, a deletion is scored as evidence destruction. With the marker, the line still does not count as a pass. It only avoids the automatic reject so a reviewer can inspect the product change.

Putting the JSON together

A review comment needs one object, not a dashboard. Merge the three files and stop.

python3 score_agent_pr.py merge \
  /tmp/provenance.json /tmp/independent.json /tmp/flake_ledger.json

Example output, illustrative only:

{
  "merge_score": 0.91,
  "independent_pass_rate": 1.0,
  "deletion_penalty": 0.05,
  "added_penalty": 0.04,
  "agent_added_assertions": 4,
  "agent_removed_assertions": 1,
  "decision": "reject",
  "reason": "assertion_removed_without_human_marker"
}

Read the decision in order. No independent tests means reject. Any unmarked deletion means reject. A pass rate of 1.0 with a pile of agent-added assertions is still not a pass; the added-penalty exists to keep that case visible. pass_score only means the independent lane did not regress. The production diff still needs a human.

What this catches that “all tests passed” does not

The scorer is built for a short list of failure modes that show up in agent diffs.

1. The patch adds tests that call the new helper and assert the helper’s own return value. Provenance tags those lines agent_added. They do not lift the score.
2. The patch deletes a parametrized row that encoded a business rule. The deletion penalty fires even if the remaining rows pass.
3. The patch “fixes” a flake by widening a timestamp comparison. If the test was unscored, the widening does not buy a pass in the independent lane until the off-tree streak is clean.
4. The patch edits expected JSON in the same commit as the serializer. The corpus hash changes, or the expected file is classified as agent-authored. Either way the independent lane does not accept it as confirmation.

None of those modes require a large model eval. They require authorship, hashes, and a runner the agent cannot reach.

Limitations

The classifier is a regex over unified diffs. It will miss helper-based assertions (self.check_invoice(x)), generated tests, and snapshot frameworks that store expected values in binary. If your suite hides oracles in helpers, inventory those helpers first or the provenance lane is theater.

The score is not a quality rating of the production code. A patch can earn pass_score and still be a bad design. The harness only answers whether preexisting evidence still holds.

Off-tree isolation is only as strong as your permissions. A runner that mounts the agent workspace writable, or a corpus stored in the same Git branch the agent commits to, collapses the model back into mixed-number CI.

Free model access and a free server option do not define throughput, model identity, or how long a job may run. Size the corpus to the review budget you already have. Do not treat a hosted free tier as a load-test farm.

Who should not use this

Skip the merge score if humans already author every test and agents cannot touch tests/. You already have provenance.

Skip it if the product is a UI with no hermetic fixtures. Unscored flakes will swallow the denominator and every PR will reject.

Skip it if snapshot regeneration is the release process, as in some compiler or schema dumps, unless you split dumps the agent may rewrite from dumps a human signs. Mixing them recreates the mixed number.

Do not use the score as an auto-merge gate on public repos where anyone can open a PR that also edits the corpus. The independent lane has to be write-gated to maintainers.

Close

Keep CI for “did the tree run.” Keep the provenance scorer for “did evidence the agent did not write still hold.” If you need a place to run that second question away from the agent’s checkout, isolate the corpus and ledger first; a free server is sufficient when the rules above are the gate, not the vendor name.