Diff the Behavior, Not the Code: A Differential Gate for Agent Patches

A test suite that passes after an agent patch proves one thing: the patch satisfies the tests you wrote. It says nothing about the behavior users already depend on. The cheapest reliable oracle is differential — run the old code and the new code on the same inputs, compare outputs. This article shows a gate built on that idea, plus a change manifest for the diffs you actually want.

Why "the agent's tests pass" is the wrong signal

Agents patch toward the tests in context. That is the failure mode.

Consider the common case: an agent "fixes" a slow lookup by adding a cache. Every existing test passes, because each one calls the lookup once. State is invisible. The patch changes the answer for the second call on the same key, and no test asks about the second call.

The pattern deserves a name: unit tests assert single-call correctness. Agent patches introduce caches, retries, ordering, and shared state. Differential testing catches those changes because it compares call sequences, not single calls.

The equivalence property

An equivalence property holds for every implementation you feed it. For a patch, write two.

Call-level equivalence:

from hypothesis import given, strategies as st

payload = st.text(min_size=0, max_size=2048)

@given(payload)
def test_single_call_matches(p):
    assert old_impl(p) == new_impl(p)

Sequence-level equivalence:

@given(st.lists(payload, min_size=1, max_size=20))
def test_call_sequences_match(seq):
    old_out = [old_impl(s) for s in seq]
    new_out = [new_impl(s) for s in seq]
    assert old_out == new_out

The second property catches caches. It is also the property agent-generated tests never write, because the agent generates tests from the same spec it read. In practice old_impl and new_impl are thin wrappers around subprocess calls to the two binaries. For a C++ codebase, keep the oracle as a binary — that is the rest of this guide.

Step 1: Freeze the baseline

Build the current code and record what it does for every fixture. The recorded outputs are the truth the gate protects; hash them so no one edits them later.

git checkout main
make release
mkdir -p baseline
for f in ./corpus/*.json; do
  id=$(basename "$f" .json)
  ./bin/service --read "$f" > "baseline/$id.out"
done
sha256sum baseline/*.out > baseline.sha

Keep baseline/ and baseline.sha out of the agent's write path. If the agent can touch the truth, the gate proves nothing.

Step 2: Generate a hostile corpus

A corpus is only as good as its edges. Generate inputs that include the cases agents love to break: empty input, maximum length, punctuation, repeated keys.

# corpus_gen.py
import json
import random
import string
from pathlib import Path

random.seed(4812)
out = Path("corpus")
out.mkdir(exist_ok=True)

fixtures = []
for i in range(2000):
    n = random.choice([0, 1, 8, 128, 2048])
    s = "".join(random.choices(string.ascii_letters + string.punctuation, k=n))
    fixtures.append({"id": i, "payload": s})

# repeated keys: the case cache bugs hide in
for i in range(2000, 2050):
    fixtures.append({"id": i, "payload": "AAA", "repeat": 2})

for fx in fixtures:
    (out / f"{fx['id']}.json").write_text(json.dumps(fx))

The seed makes the corpus reproducible. The repeated-key block is the part you must not skip; it is the cheapest way to force a cache or state bug into the open.

Step 3: Run the gate

The gate replays every fixture against the candidate binary and compares byte-for-byte with the frozen baseline. No test runner, no assertions beyond equality.

#!/usr/bin/env bash
# diffgate.sh -- replay a frozen corpus against a candidate binary
sha256sum -c baseline.sha || { echo "baseline modified"; exit 1; }
count=0
for f in ./corpus/*.json; do
  id=$(basename "$f" .json)
  if ! cmp -s <(./bin/new --read "$f") "baseline/$id.out"; then
    echo "BEHAVIOR DIFF: $f"
    diff "baseline/$id.out" <(./bin/new --read "$f")
    exit 1
  fi
  count=$((count + 1))
done
echo "no behavioral diffs over $count fixtures"

The sha256sum -c line is the lock. Even if the agent edits a baseline output, the gate fails before it reads a single fixture.

Step 4: Triage intentional diffs with a change manifest

Some diffs are intentional. The question is who decides. Reviewing a large code diff is slow; reviewing a list of behavior diffs is fast.

{
  "allowed_diffs": [
    {"fixture": "corpus/177.json", "reason": "cache added intentionally, ticket #4812"}
  ]
}

The rule: a diff without a manifest entry fails the gate. The manifest is the contract. Every entry is a behavior decision you made, and the list is short enough to read in one sitting. If one "fix" changes twenty inputs, the manifest says so before the patch merges.

Why this gate sidesteps flaky tests

A flaky test fails for reasons unrelated to the code under test: timing, ports, ordering, shared state. The differential gate removes the runner from the loop. One process, one input, one output, compared with cmp. For deterministic code, the gate is deterministic — flakiness has nothing to attach to.

This is not an argument against freezing flaky tests. The freeze keeps CI honest; the gate keeps behavior stable. They answer different questions and both belong in the review pipeline.

Running the gate inside a free agent loop

The gate only earns its keep if it runs on every patch. I used MonkeyCode's free model tier for patch generation and its free server option to run the verification loop without my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The loop I used:

1. The agent proposes a patch.
2. The loop builds the candidate binary.
3. The loop replays the frozen corpus against the candidate.
4. Any diff outside the manifest rejects the patch.
5. Only after the gate passes does the full suite run.

That ordering matters. The gate answers "did behavior change" cheaply on a small corpus; the full suite answers "does the spec still hold." Agents should not gate on the second question first. I did not benchmark the free tier's quotas or throughput, and I will not repeat numbers I cannot verify; availability claims change, the gate script does not. Run the gate on your own machine with your own corpus before you decide what the agent loop is worth.

Limitations

Differential testing is a consistency oracle, not a correctness oracle.

It breaks on nondeterministic output: timestamps, random seeds, hash ordering. Seed what you can; mask the fields you do not ship.

Floating point needs a policy. Bit-exact comparison rejects harmless reordering; tolerance comparison accepts real drift. Decide which one matches your product.

New features have no old counterpart. The gate covers only the overlap between old and new behavior; the new surface still needs property and contract tests.

The honest case: when you are intentionally fixing a bug, the gate flags every affected input. The manifest forces you to admit the change. That is the feature, not the bug.

Who should not use this

Greenfield code has no baseline to diff against.

Deep UI code cannot be isolated to stdin and stdout.

Domains with nondeterministic semantics — distributed consensus, real-time systems — rarely have a well-defined notion of process-level output equality.

If none of those apply, the differential gate is the cheapest part of agent-patch review. It does not prove the patch correct. It proves the patch did not change what you already ship — a weaker claim and a necessary one.

If you build a gate like this, publish the manifests you approve. Those diff lists document what your product actually promises, and the community needs more real data on what agent patches silently change.