Split Mixed Agent Diffs: The Test Hunk Is Not a Witness

A mixed agent diff is one hypothesis too many. If the same patch rewrites production code and the tests that observe it, a green CI job is not a verification result. It is a joint claim. The cheaper half of that claim is almost always the suite.

Treat the test hunk as untrusted input. Merge source only after a separate gate has scored the suite change. The protocol below is a worked example. It is not a production war story, and it does not assume a particular agent vendor.

What a mixed diff actually proves

A source-only patch can still be wrong. That is normal. Tests exist to catch it. A source-and-test patch can be wrong and still go green, because the observer moved with the subject. The failure is quiet. Assertions disappear. Timeouts widen. Fixtures get rewritten to match the new output. skip and xfail appear next to the new behavior.

None of those edits are illegal Git. They are also not evidence. Evidence requires an observer the patch cannot rewrite in the same commit.

Protocol: five gates, in order

Run these steps on the candidate branch before any merge job that reports “tests passed.” If a step fails, discard the mixed commit. Do not “fix the tests” in the same tree and rerun.

1. Split the unified diff into source, tests, fixtures, and other.
2. Reject the patch if tests or fixtures changed unless a human explicitly opts into a suite PR.
3. If a suite PR is allowed, score the test hunk for assertion drift and flake-surface growth.
4. Re-hash fixtures from the merge-base, not from the agent tree.
5. Run properties that live on a ref the agent cannot push.

The order matters. Scoring tests after they have already been merged into the same tree as the code is how self-graded patches slip through.

Step 1: Split the hunk, do not read the PR title

Agents label commits as “tests included.” Ignore the label. Classify paths. A small classifier is enough for a first gate.

# proposal: split_agent_diff.py — unexecuted worked example
from __future__ import annotations

import subprocess
from pathlib import Path

SOURCE_GLOBS = (".py", ".go", ".rs", ".ts", ".js", ".java")
TEST_HINTS = ("test_", "_test.", "/tests/", "/test/", "spec.")
FIXTURE_HINTS = ("/fixtures/", "/testdata/", ".snap", ".golden")


def changed_files(merge_base: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{merge_base}...HEAD"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def bucket(path: str) -> str:
    posix = path.replace("\\", "/")
    lower = posix.lower()
    if any(h in lower for h in FIXTURE_HINTS):
        return "fixtures"
    if any(h in lower for h in TEST_HINTS):
        return "tests"
    if Path(path).suffix in SOURCE_GLOBS:
        return "source"
    return "other"


def split(merge_base: str) -> dict[str, list[str]]:
    groups = {"source": [], "tests": [], "fixtures": [], "other": []}
    for path in changed_files(merge_base):
        groups[bucket(path)].append(path)
    return groups

Print the four lists in CI. A review that cannot see the split will treat a 12-file “fix” as one object. It is not one object.

Step 2: Default-deny mixed commits

Policy is simpler than heuristics. If tests or fixtures is non-empty, fail the source merge. Open a second branch if the suite truly must move. Two pull requests. Two CI identities. Two review checklists.

That sounds slow. It is slower than a green lie only if you count wall-clock and ignore rollback. Mixed diffs hide the rollback unit. You cannot revert “just the product bug” without also reverting the tests that were taught to accept it.

Teams that insist on a single PR can still split inside the PR: one commit that is source-only, one commit that is suite-only, and a CI matrix that runs properties against the source commit before the suite commit is applied. The second commit is then scored as a suite change, not as proof.

Step 3: Score the test hunk for assertion drift

When a suite PR is actually required, do not read it as documentation. Parse the hunk. Count signals that weaken the observer.

# proposal: score_test_hunk.py — unexecuted worked example
import re
import subprocess
from dataclasses import dataclass

WEAKEN = [
    re.compile(r"^\+\s*(pytest\.mark\.(skip|xfail)|self\.skipTest)"),
    re.compile(r"^\+\s*@unittest\.expectedFailure"),
    re.compile(r"^\+\s*assert True\b"),
    re.compile(r"^\+\s*pass\s*$"),
    re.compile(r"^\+.*approx\([^)]+,\s*rel=0\.\d{1,}[1-9]"),
    re.compile(r"^\+.*timeout\s*=\s*\d{4,}"),
]
REMOVED_ASSERT = re.compile(r"^\-\s*assert\b")
ADDED_ASSERT = re.compile(r"^\+\s*assert\b")


@dataclass
class HunkScore:
    added_asserts: int
    removed_asserts: int
    weaken_hits: int
    net_asserts: int


def score_test_diff(merge_base: str) -> HunkScore:
    diff = subprocess.check_output(
        ["git", "diff", "-U0", f"{merge_base}...HEAD", "--", "tests", "test"],
        text=True,
        stderr=subprocess.DEVNULL,
    )
    added = removed = weaken = 0
    for line in diff.splitlines():
        if REMOVED_ASSERT.match(line):
            removed += 1
        if ADDED_ASSERT.match(line):
            added += 1
        if any(p.match(line) for p in WEAKEN):
            weaken += 1
    return HunkScore(added, removed, weaken, added - removed)


def gate(score: HunkScore) -> list[str]:
    failures = []
    if score.removed_asserts > score.added_asserts:
        failures.append("net assertion loss")
    if score.weaken_hits > 0:
        failures.append("skip/xfail/timeout/approx widening")
    if score.added_asserts == 0 and score.removed_asserts == 0:
        failures.append("suite files changed with no assert churn; inspect snapshots")
    return failures

The patterns are conservative. They will miss clever tautologies. They will still catch the common agent move: delete the hard assert, add a skip, keep the test name so coverage charts stay flat.

Set the gate to fail closed on weaken_hits > 0. A human can file an expiry ticket and add a skip later. The agent does not get to expand the skip surface as a side effect of a product patch.

Step 4: Fixture digests from the merge-base

Fixture files are data, not prose. If the agent updates a golden file, the new bytes are the claim, not the proof. Hash the fixture tree at merge_base. Apply the source patch only. Run the tests that consume those fixtures. Only then consider a fixture update, and only in the suite PR.

# proposal: fixture lock taken before the agent tree is trusted
BASE=$(git merge-base origin/main HEAD)
git archive "$BASE" -- tests/fixtures testdata 2>/dev/null | sha256sum

Compare that digest to HEAD. A mismatch is not “tests updated.” It is an oracle change. Require a one-line rationale in the suite PR: which byte changed, which assertion now depends on it, and whether the old fixture still fails the old binary. If you cannot run the old binary against the new fixture and the new binary against the old fixture, you do not have a migration. You have a overwrite.

Step 5: Properties the agent cannot edit

In-repo property tests are useful until the same agent that writes src/ can edit tests/test_properties.py. Move a thin property layer onto a protected ref, a second repository, or a CI path that checks out origin/properties at a pinned SHA.

Keep the properties boring. Idempotent handlers return the same bytes. Parsers reject truncated input. Public functions do not raise on the documented empty fixture. The point is not coverage. The point is an observer with a different write ACL.

# proposal: ci fragment — properties checked out from a pinned ref
- name: external properties
  run: |
    git fetch origin properties
    git checkout --detach origin/properties -- properties/
    pytest -q properties/ --tb=short

If the source patch needs a property change, that is a third PR, against the properties ref. Three PRs is friction. Self-graded green is cheaper, and wrong.

Flake-surface delta, not a freeze ritual

Flakes are a separate budget. Do not let the agent spend it. After the source-only tree is under test, sample the affected files a fixed number of times on one machine.

# proposal: cheap flake-surface sample, not a durability claim
pytest -q tests/path_touched_by_diff.py --count=8 --tb=line
echo $? 

Record pass/fail counts. Compare them to the same command on merge_base. If the fail count rose, the patch widened the flake surface. That is a failed source gate, not a reason to add xfail. New xfail belongs in a suite PR with an expiry date written by a person.

--count requires pytest-repeat or an equivalent loop. If you do not have it, an eight-iteration shell loop is enough for this gate. The number 8 is a filter, not a reliability estimate. Do not publish it as a flake rate.

Decision table

Observation	Source PR	Suite PR	Action
source only, properties pass, flake count flat	allow	n/a	merge source
tests changed, net asserts down	block	block	discard mixed commit
new skip/xfail/timeout	block	review	human expiry only
fixture digest changed	block	required	A/B old and new binaries
properties ref must change	block	n/a	separate properties PR
flake count rose, asserts stable	block	optional	fix source or drop patch
tautological adds (assert True, empty pass)	block	block	discard

Use the table in CI output. A yes/no is easier to audit than a paragraph in the agent transcript.

Where a free agent loop fits

Generating the candidate patch is the disposable part of this workflow. Scoring the hunk is not. If you already produce patches from a coding agent, throw away mixed diffs instead of negotiating with them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Those two facts matter only as a way to mint candidate patches you can afford to reject. The splitter, the hunk scorer, and the pinned properties ref do not depend on that product. Swap the generator. Keep the gates.

A useful loop is mechanical: request a source-only patch, run split(), fail if tests or fixtures is non-empty, run external properties, sample flake-surface, then stop. Do not send the failing test output back into the same agent and ask it to “update tests.” That request is how mixed diffs are born.

Limitations

This protocol assumes you have a merge-base, a test path convention, and somewhere to pin properties that the agent cannot write. It assumes CI can run git diff and a bounded pytest sample. It does not measure mutation score. It does not prove correctness. It only prevents the suite from being used as a witness for its own rewrite.

Do not use it on throwaway prototypes with no tests. Do not use it on patches that are only documentation. Do not treat the weaken-regex list as a security boundary; a determined prompt can still write a tautology the regex will miss. Humans still read the suite PR.

Teams that already forbid agents from touching tests/ can skip steps 2–3 and keep steps 4–5. That is the stronger policy. The scorer exists for organizations that will not adopt the stronger policy yet and still need a fail-closed default.

Close

Green is a joint statement when the agent edits both sides. Split the statement. Hold the suite. Let the source patch stand in front of an observer it did not write. If you generate candidates on a free endpoint, spend the budget on retries of source-only diffs, not on teaching the suite to agree.