An Agent Patch Is a Hypothesis. Test It Like One.

An agent patch is a hypothesis about your codebase. It makes three claims: the invariants still hold, the output changes are intended, and the new failures are not flukes. CI should test all three claims before a human opens the diff.

The gate I keep coming back to is a three-checkpoint contract. Property delta. Fixture delta. Flake freeze. Each checkpoint answers a question a reviewer would ask anyway; the goal is to make the answers machine-checkable so the review budget goes where it matters.

Patch generation is getting cheaper, which moves the constraint. Give a patch generator free model access and a free server option — MonkeyCode is the concrete case this article is written around — and the cost per patch attempt drops toward zero. Review time does not drop with it. If reviewing one patch costs more than generating ten, the agent wins by attrition.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The contract below is model-agnostic. It does not care whether the patch came from a free endpoint, a local model, or a teammate, and treating the free tier as an availability claim rather than a benchmark is enough. Nothing here depends on a model name, a quota, or an uptime number.

Checkpoint 1: property delta

A property test describes an invariant. An LRU cache never exceeds its capacity. A token bucket never goes negative. A serializer round-trips. Checkpoint 1 runs the property suite on the merge-base, runs it again on the patch, and diffs the results.

Rule: any property that passed on baseline and fails on HEAD blocks the merge. The patch did not "change behavior" — it broke a standing contract with the rest of the code.

from hypothesis import given, strategies as st

@given(
    st.integers(min_value=2, max_value=64),
    st.lists(
        st.tuples(st.sampled_from(["get", "put"]), st.text(min_size=1)),
        min_size=50,
        max_size=300,
    ),
)
def test_lru_size_never_exceeds_capacity(capacity, ops):
    cache = LRUCache(capacity)
    for action, key in ops:
        if action == "put":
            cache.put(key, len(key))
        else:
            cache.get(key)
        assert len(cache) <= capacity

Both runs emit JUnit XML. A short parser compares them:

import sys
import xml.etree.ElementTree as ET

def failures(path):
    root = ET.parse(path).getroot()
    return {
        case.get("name"): int(case.get("failures", "0"))
        for case in root.iter("testcase")
    }

baseline, head = failures("baseline.xml"), failures("head.xml")
broken = [
    name for name, fails in baseline.items()
    if fails == 0 and head.get(name, 1) != 0
]
if broken:
    print("contract violation:", broken)
    sys.exit(1)

One rule is easy to miss. If the agent adds a new property, that property should fail on baseline code. A property that passes on both old and new code proves nothing; it is a test that never learned to say no.

Checkpoint 2: fixture delta

Property tests are generative. Fixtures are evidence. A fixture is a fixed input with a locked output, and it catches what properties rarely do: deliberate shifts in format, ordering, or rounding that still respect the invariants.

Rule: every changed fixture must be named in FIXTURES.md, in the same commit that touches it, with a one-line justification. The checkpoint lists the changed files and greps the manifest.

CHANGED=$(git diff --name-only "$(git merge-base HEAD origin/main)" HEAD -- tests/fixtures/)
for fixture in $CHANGED; do
  grep -qF "$fixture" FIXTURES.md || { echo "unannounced fixture: $fixture"; exit 2; }
done

An agent that reformats a golden file to match its own style has not fixed the test; it has moved the finish line. The manifest forces intent to be stated in the same commit, where a reviewer can actually see it.

Checkpoint 3: flake freeze

A flaky test makes every other verdict radioactive. One nondeterministic failure in the property suite poisons the delta; one in fixtures hides real change. The freeze is simple: a test that fails even once during a gate run is quarantined, and a non-empty quarantine blocks the merge. The agent cannot retry into a green run, because the retry loop belongs to the human, and this loop is closed.

rm -f .quarantine/current
for run in 1 2 3; do
  if ! pytest tests/flaky -q; then
    echo "failure on run $run" >> .quarantine/current
  fi
done
[ -s .quarantine/current ] && { echo "FROZEN: quarantine must stay empty for 3 runs"; exit 3; }

Three consecutive clean runs empty the quarantine. The rule sounds harsh. It is the only way to keep the first two checkpoints honest.

The contract, assembled

The full gate is a twenty-line script plus the JUnit diff. CI runs it on every patch the agent proposes:

#!/usr/bin/env bash
set -euo pipefail
BASE=origin/main
MERGE_BASE=$(git merge-base HEAD "$BASE")
WORK=/tmp/contract-base
trap 'git worktree remove --force "$WORK" 2>/dev/null' EXIT

# Checkpoint 1: property delta
git worktree add -f "$WORK" "$MERGE_BASE"
(cd "$WORK" && pytest tests/property -q --junitxml=baseline.xml)
pytest tests/property -q --junitxml=head.xml
python compare_junit.py "$WORK/baseline.xml" head.xml || exit 1

# Checkpoint 2: fixture delta
for fixture in $(git diff --name-only "$MERGE_BASE" HEAD -- tests/fixtures/); do
  grep -qF "$fixture" FIXTURES.md || { echo "unannounced fixture: $fixture"; exit 2; }
done

# Checkpoint 3: flake freeze
rm -f .quarantine/current
for run in 1 2 3; do
  if ! pytest tests/flaky -q; then
    echo "failure on run $run" >> .quarantine/current
  fi
done
[ -s .quarantine/current ] && { echo "FROZEN"; exit 3; }

echo "contract PASS"

Reading the verdict

Exit	Meaning	Action
0	contract PASS	merge candidate
1	invariant broken	reject; return the failing property
2	behavior changed silently	reject; require a manifest line
3	suite unstable	freeze; wait for three clean runs

Each failure mode tells the agent what to do next. That is the property of a good gate: it rejects, and it explains.

Who should not use this

The contract inherits the quality of your properties. A suite of tautological tests produces a gate that says yes to everything, so the real work is writing invariants that can fail. A team starting from zero does not need a contract yet; it needs one or two properties that matter.

Golden fixtures rot in UI-heavy and integration-heavy codebases, where output changes with every dependency bump. There, Checkpoint 2 generates noise instead of signal, and FIXTURES.md becomes a lie by week two.

The freeze makes a stable suite a precondition. A codebase with dozens of flaky tests stays frozen forever, and the team disables the gate out of frustration. Stabilize the suite before installing the freeze.

Finally, this contract does not approve security-sensitive changes. A patch that touches authentication or crypto still needs a human who understands both the patch and the threat model. No generator, free or paid, replaces that.

The bottom line

Every developer is now a reviewer, and the pipeline that feeds reviewers is what needs testing. The three-checkpoint contract is that test: it checks the patch, and it checks the suite that checks the patch.

Start with Checkpoint 1 alone. One invariant, one diff, one week of patches — the rest of the contract will suggest itself.

Comments

[Kane Lim]

私が特に注目したのは、エージェントが生成したパッチを信頼できる変更ではなく仮説として扱う点です。

3つのチェックポイントは、不変条件の破綻、意図的な出力変更の未文書化、不安定なテストによるシグナルの汚染といった、異なる障害モードを区別できるため、理にかなっています。特に、新しく追加されたプロパティはベースラインに対して失敗するべきであるというルールは素晴らしいと思います。そうでなければ、防止しようとしている回帰について、実際にはほとんど何も証明できないからです。

フィクスチャマニフェストも良いアイデアです。変更理由の記録がない場合、ゴールデンファイルは「CIが合格するまで期待される出力を更新する」という状態に陥りやすくなります。

私が興味を持っているのは、エージェントが多くの無関係な領域に影響を与える大規模なパッチを提案した場合、この契約がどのように動作するかということです。その場合、3つのチェックポイントすべてが合格したとしても、デルタ自体の推論が困難になる可能性があります。

しかし、全体的な方向性は正しいです。パッチ生成が低コストになれば、真のエンジニアリング上のボトルネックは、信頼できる検証とレビューへと移行するでしょう。