Catch Silent Behavior Drift in Agent Patches with Golden Snapshots

Last Tuesday, an agent patch turned every 404 into a 403. All tests passed. The patch changed the error handler after the assertions ran. That is not a test failure. That is a blind spot.

Most test suites only assert what you remembered. They cannot see the shadows. Agent patches love shadows. They move code. They rename symbols. They reorder branches. The behavior stays the same until it does not.

I use behavioral snapshots to catch that drift. The idea is painfully boring. Record the output of fixed probes. Run the same probes after the patch. Compare the fingerprints. If they differ, the patch changed something unintended. You get a diff. No guesses. No flaky tests.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The infrastructure is intentionally cheap. I run the harness on MonkeyCode's free server. I use the free model token pool of 10 million tokens to generate probe inputs. That pool pays for edge cases. No GPU is needed. No paid CI runner.

The core harness is a single Python script. It reads fixture files. It executes your function in a subprocess. It hashes the JSON output. Here is the code.

import json
import hashlib
import subprocess
import sys

def run_probe(func_file, input_data):
    proc = subprocess.run(
        [sys.executable, func_file],
        input=json.dumps(input_data),
        capture_output=True,
        text=True
    )
    if proc.returncode != 0:
        return {"crashed": proc.stderr[-200:]}
    return json.loads(proc.stdout)

def snapshot(func_file, fixtures):
    out = {}
    for name, data in fixtures.items():
        result = run_probe(func_file, data)
        raw = json.dumps(result, sort_keys=True)
        out[name] = hashlib.sha256(raw.encode()).hexdigest()
    return out

That is the whole engine. You might want to filter volatile keys first. Add a cleaner function.

def scrub(obj, volatile):
    if isinstance(obj, dict):
        return {k: scrub(v, volatile) for k, v in obj.items() if k not in volatile}
    if isinstance(obj, list):
        return [scrub(i, volatile) for i in obj]
    return obj

Run snapshot against the base commit. Store the JSON file. Check out the agent patch. Run it again. Diff the two files. The exit code is nonzero when anything moves.

Let me show you a concrete wrapper. Suppose your code has a payment function.

# probe_payment.py
import sys, json
from payments import process

def main():
    payload = json.load(sys.stdin)
    result = process(payload["amount"], payload["currency"])
    print(json.dumps({
        "status": result.status,
        "code": result.code
    }))

if __name__ == "__main__":
    main()

Your fixtures file contains extreme values.

{
  "zero": {"amount": 0, "currency": "USD"},
  "negative": {"amount": -5, "currency": "USD"},
  "huge": {"amount": 10000000, "currency": "EUR"}
}

The model-generated probes follow a simple prompt. I ask it to list five inputs that would break the function. The free token pool covers that request. I paste the JSON into the fixture file. The model suggests weird values I would never pick. I do not run it at test time. That would be non-deterministic. I run it once, offline, and commit the fixtures.

Now add the cron job. On the free server, I set up a ten-minute loop.

*/10 * * * * /opt/snapshot/run.sh

The shell script pulls the candidate branch, rebuilds the fingerprint, and compares. If the diff is not empty, it logs the failure and exits with code 1.

#!/bin/bash
cd /opt/snapshot
git fetch origin agent-patch
git checkout agent-patch
python build_snapshot.py > candidate.json
diff baseline.json candidate.json || exit 1

That is it. No test framework. No dependencies beyond Python and git.

Why does this work? Because agent patches usually preserve the happy path. They break the quiet corners. The happy path has assertions. The quiet corners have nothing. Snapshots give those corners a witness.

The 404-to-403 case would have produced a different status code. The fingerprint would change. The merge would stop. A human would look at the diff. The bug would never ship.

There are limits. Deterministic code only. If your function returns the current time or a random UUID, the snapshot will say \"everything changed.\" You need to scrub those fields. The scrub function above works for most cases.

Do not use this for exploratory prototypes. Do not use it for frontend components with heavy state. The probe costs will eat your token pool. The signal-to-noise ratio will be terrible.

Also, this is not property-based testing. Property tests assert invariants. Snapshots assert continuity. You want both. The earlier article on agent patches covered property checks. This one covers the missing layer.

The free tier from MonkeyCode makes experimentation cheap. I can spin up a new probe set without thinking about cost. The 10-million-token allowance is enough for a month of edge-case generation. The free server keeps the loop running.

I am going to keep this harness in my stack. Next I want the model to generate probes automatically. I will ask for pathological inputs: empty strings, null bytes, Unicode snowmen. Then the snapshot grows with the codebase.

Try it on your next agent patch. Build one probe. Add two fixtures. Run it before and after. See what your model changed underneath. The diff will tell you a story.

That is the whole trick. No magic. Just a fingerprint of behavior, taken twice.