Test the Behavior Delta After an Agent Patch, Not the Whole Suite

A green full-suite run is a weak signal after an agent patch. Suites miss the edited helpers more often than they catch them. The evidence that matters is the behavior delta on a probe catalog taken from the diff, checked with metamorphic relations, with flakes moved into a dated quarantine ledger instead of a silent skip.

This write-up is a method, not a war story. It does not assume a particular model, quota, or machine. The code is a worked example. Swap the subject-under-test for your own module before you trust any output.

The failure mode

Agent patches concentrate edits in a few functions. A large suite can stay green while those functions are wrong. Coverage percentages do not fix that. They report lines executed in some prior run, not paths the new diff actually changed.

Flakes make the signal worse. Teams skip the noisy case. The skip then outlives the bug. A quarantine row with an expiry date is a different object from a skip. It is a scheduled decision, not a deletion.

Exact expected values are the third trap. If the same pull request can edit the implementation and the remembered string, the gate is circular. Related inputs with a related-output rule avoid that circle. The rule lives in a file the agent does not own.

Four files, one report

Keep four files next to the code. Generate a fifth at run time.

1. probes.json — inputs that exercise the blast-radius slice.
2. relations.py — metamorphic checks. No expected golden values for the patch itself.
3. quarantine.json — flaky probe ids, reason codes, and ISO-8601 expiry dates.
4. slice.txt — symbols extracted from the diff.
5. delta_report.json — written by the harness: pass, fail, quarantine-hit, expired-quarantine.

The report is the only artifact the gate should read. Humans read the other four. If a chat transcript holds the only copy of a relation, the method has already failed.

Step 1 — Slice the diff

Parse the unified diff. Collect added and removed function names. Map each name to a module. That list is the blast-radius slice. Everything else is out of scope for this gate.

git diff --unified=0 HEAD~1 -- '*.py' > /tmp/agent.patch
python3 extract_slice.py /tmp/agent.patch > slice.txt
cat slice.txt

A tiny extractor is enough for a first cut. It will miss dynamic dispatch. Document that miss in the report rather than pretending the regex is complete.

# extract_slice.py — worked example, not a production parser
import re
import sys
from pathlib import Path

name_pat = re.compile(r'^[+-]\s*def\s+(\w+)\s*\(', re.M)

text = Path(sys.argv[1]).read_text(encoding='utf-8')
names = sorted(set(name_pat.findall(text)))
for n in names:
    print(n)

If the patch is a one-line constant change inside a huge function, add that function to slice.txt by hand. Short names only. Empty slice means this gate is a no-op, not a free pass on the rest of CI.

Step 2 — Build a probe catalog for the slice

A probe is an input, not an expected output. Store values the function already accepts. Keep the catalog in git so two runs can be compared without a chat log.

{
  "module": "cachekey",
  "function": "canonical_key",
  "probes": [
    {"id": "p01", "params": {"q": "coffee", "sort": "price"}},
    {"id": "p02", "params": {"sort": "price", "q": "coffee"}},
    {"id": "p03", "params": {"q": "coffee%20mug", "sort": "price"}},
    {"id": "p04", "params": {"q": "coffee mug", "sort": "price"}}
  ]
}

p01 and p02 differ only by insertion order. p03 and p04 differ by encoding of a space. Those pairs are the metamorphic fuel. They are not assertions that the key equals a remembered string.

If a coding model is in the loop, use it only to propose extra probes from the diff. Do not let it write the relations that judge those probes. Judgment stays in relations.py, which a human reviews.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Those two facts are the only product claims used here. The harness runs on any Python 3 environment. The free server is one place to park the run when a laptop is the wrong machine. The free model is one way to draft additional probes. Neither replaces the catalog or the relations file.

Step 3 — Encode relations, not answers

Metamorphic testing asks whether two related inputs produce related outputs. It does not ask whether output equals a fixture the agent could have edited.

# relations.py — worked example
from cachekey import canonical_key


def same_map_same_key(a, b):
    """Permutation of equal dicts must not change the key."""
    return canonical_key(a) == canonical_key(b)


def encoding_equivalence(raw, encoded):
    """A space and %20 in a query value must collapse to one key."""
    return canonical_key(raw) == canonical_key(encoded)


RELATIONS = [
    ("order-invariant", same_map_same_key, "p01", "p02"),
    ("space-encoding", encoding_equivalence, "p04", "p03"),
]

If both sides are wrong in the same way, the relation still holds. That is a real limitation. Pair at least one relation with an extreme probe: empty dict, a very large dict, or duplicate keys if the parser allows them. Extremes are not oracles. They are cheap attempts to make two wrongs diverge.

Every new probe needs a relation, or it is dead weight. A catalog that grows without pairs becomes a corpus nobody reads. When a model suggests ten probes, accept two that complete a pair. Reject the rest. A one-line PR comment is enough: dropped p11, no relation. Future agents will try to reintroduce it. The comment is the trail.

Step 4 — Quarantine with a calendar date

A skip is forever until someone notices. A quarantine row dies on a date. On 2026-09-03, a 14-day window lands on 2026-09-17. Put that date in the file. The harness must fail the gate if today is after expiry and the row is still present.

{
  "items": [
    {
      "probe_id": "p03",
      "reason": "percent-encoding depends on upstream urllib version",
      "first_seen": "2026-08-20",
      "expires": "2026-09-17",
      "ticket": "QA-4412"
    }
  ]
}

Rules the harness enforces:

1. A quarantined probe does not fail the gate before expires.
2. A quarantined probe that is stable for two consecutive runs should be proposed for deletion, not kept.
3. An expired row is a failure, even if the probe passed this time. The failure is process, not product. Someone forgot the ticket.

Do not encode “retry three times and ignore.” Retries hide timing bugs. The ledger is the place for known noise. The gate is the place for unknown noise.

Step 5 — Run, then write the delta report

# probe_harness.py — worked example
from __future__ import annotations

import json
import sys
from datetime import date
from pathlib import Path

from relations import RELATIONS


def load(p):
    return json.loads(Path(p).read_text(encoding="utf-8"))


def today():
    return date.today().isoformat()


def main(catalog_path, quarantine_path, report_path):
    catalog = load(catalog_path)
    probes = {p["id"]: p["params"] for p in catalog["probes"]}
    qfile = load(quarantine_path)
    qmap = {i["probe_id"]: i for i in qfile["items"]}
    results = []
    now = today()

    for name, fn, left_id, right_id in RELATIONS:
        row = qmap.get(left_id) or qmap.get(right_id)
        if row and row["expires"] < now:
            results.append({
                "relation": name,
                "status": "expired-quarantine",
                "ticket": row.get("ticket"),
            })
            continue
        if row and row["expires"] >= now:
            results.append({
                "relation": name,
                "status": "quarantined",
                "expires": row["expires"],
            })
            continue
        ok = fn(probes[left_id], probes[right_id])
        results.append({
            "relation": name,
            "status": "pass" if ok else "fail",
            "left": left_id,
            "right": right_id,
        })

    failed = [r for r in results if r["status"] in ("fail", "expired-quarantine")]
    report = {
        "date": now,
        "slice_function": catalog["function"],
        "results": results,
        "gate": "red" if failed else "green",
    }
    Path(report_path).write_text(json.dumps(report, indent=2), encoding="utf-8")
    print(json.dumps(report, indent=2))
    sys.exit(1 if failed else 0)


if __name__ == "__main__":
    main(*sys.argv[1:4])

Run it:

python3 probe_harness.py probes.json quarantine.json delta_report.json
echo $?

Exit code 1 is the gate. Do not parse logs. Parse delta_report.json.

A subject-under-test for the example:

# cachekey.py — illustrative module an agent might patch
from urllib.parse import quote_plus


def canonical_key(params: dict) -> str:
    items = sorted((str(k), str(v)) for k, v in params.items())
    return "&".join(f"{quote_plus(k)}={quote_plus(v)}" for k, v in items)

If an agent “simplifies” this by dropping sorted, p01 and p02 diverge. The relation fails. The full suite may still be green. That split is the point of the slice.

Decision table

Situation	Run	Gate reads
Diff touches canonical_key	Probe catalog + relations	delta_report.json
Diff touches only comments	Empty slice, skip harness	No report, CI continues
Probe flaps, ticket open	Quarantine row with expiry	quarantined is not fail
Expiry date in the past	Harness still loads the row	expired-quarantine is fail
Agent rewrites tests in the same PR	Reject the PR for this gate	Relations file must be human-owned

The last row is the one teams skip. If the same patch edits relations.py and cachekey.py, the gate is compromised. Split the PR. Or require a second reviewer on relations.py.

Wire the check as a required status, not a comment bot:

# ci-behavior-delta.sh — worked example
set -euo pipefail
python3 extract_slice.py /tmp/agent.patch > slice.txt
if [ ! -s slice.txt ]; then
  echo "empty slice; behavior-delta gate skipped"
  exit 0
fi
python3 probe_harness.py probes.json quarantine.json delta_report.json

What this does not do

It does not measure model quality. It does not claim a pass rate. It does not replace load tests, security tests, or typed contracts. Metamorphic relations can hold for a systematically wrong implementation. Empty slices from a weak regex will silently under-test. Quarantine rows without ticket hygiene become skips with extra JSON.

Hosting the process on another machine does not change those limits. Drafting probes with a model does not change them either. The catalog and the relations file remain the source of evidence.

Who should not use this

Do not use this as the sole gate for cryptography, payments that require exact rounding proofs, medical device firmware, or anything where “two outputs related” is weaker than “this output is the one the standard names.” Do not use it if your language has no reliable diff-to-symbol path and nobody will maintain slice.txt by hand. Do not use it to justify deleting the existing suite. The suite still catches regressions outside the slice. This gate only densifies evidence inside it.

Teams that cannot keep a calendar for expiry dates should not start a ledger. An expired row that nobody triages is worse than a known skip, because it pretends to be a process.

Keep the files on the default branch

Keep probes.json, relations.py, and quarantine.json on the default branch. Run the harness on pull requests that touch the slice. Store delta_report.json as a CI artifact. Review relations.py like production code.

If you try this, put the first expiry date on a calendar you actually look at. The method fails the day the ledger becomes a junk drawer.