Your Clean Diff May Lie: Capture Behavior Before You Approve

Code review reads text. Behavior lives in execution. A rewritten helper can keep its lines yet change its contract on input four. The diff shows you nothing.

This happens often with AI-assisted patches. The model reads the same legacy code you read. It removes what looks like redundancy. You see a clean diff and approve it. Neither of you sees the edge case the patch quietly broke.

The fix costs about thirty minutes of setup. Before you touch anything, snapshot the current behavior. After the patch, run the same probes. Compare what actually changed. Then review the behavior diff, not the code diff.

Why clean diffs fool reviewers

A code diff shows text movement. It does not show semantic movement. If a function had no tests, every line is opinion. The reviewer reconstructs meaning from the source. The model did the same thing when it wrote the patch. Two readers, one broken source, zero runtime evidence.

Characterization tests fix part of this. They lock the current behavior into assertions. But they must be written before the patch. And they only cover what someone decided to check.

Snapshot capture is less formal and broader. It stores real outputs, not assumptions. It lets you diff the before and the after.

The workflow: snapshot, patch, diff

Use any legacy function as the target. Here I will use orders.parse_payment_request. The patch simplifies the amount handling. The diff looks beautiful. The behavior must prove itself.

Step 1: Save a probe set

Collect inputs from logs, call sites, and API payloads. Add the awkward cases: string amounts, numeric amounts, missing currency, null amounts, extra fields. Save them as JSON.

[
  {"args": [{"amount": "12.30", "currency": "USD"}]},
  {"args": [{"amount": 12, "currency": "USD"}]},
  {"args": [{"amount": "12.30"}]},
  {"args": [{"amount": "12.30", "currency": "EUR", "note": "gift"}]},
  {"args": [{"amount": null, "currency": "USD"}]},
  {"args": [{"amount": "0", "currency": "USD"}]}
]

The probe set is not a benchmark. It is a net for capturing behavior drift. The goal is not coverage. The goal is to catch what the human and the model both assumed.

Use MonkeyCode's free model tier to extend this list if you are stuck. The model does not need to be excellent at code. It needs to imagine the weird inputs a payment API has seen in production. Check the dashboard for the current model and quota details. Numbers change; I won't print dated values. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 2: Snapshot the current behavior

Run this small script against the untouched function.

# snapdiff.py
import argparse
import importlib
import json

def capture(module_function, probes):
    module_name, function_name = module_function.rsplit(".", 1)
    fn = getattr(importlib.import_module(module_name), function_name)
    rows = []
    for probe in probes:
        args = probe.get("args", [])
        kwargs = probe.get("kwargs", {})
        try:
            result = fn(*args, **kwargs)
            rows.append({"probe": probe, "status": "ok", "result": repr(result)})
        except Exception as exc:
            rows.append({"probe": probe, "status": "error", "result": f"{type(exc).__name__}: {exc}"})
    return rows

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("target")
    parser.add_argument("probes")
    parser.add_argument("output")
    args = parser.parse_args()
    probes = json.load(open(args.probes))
    rows = capture(args.target, probes)
    with open(args.output, "w") as fh:
        json.dump({"target": args.target, "rows": rows}, fh, indent=2, sort_keys=True)

if __name__ == "__main__":
    main()

Run it before the patch touches anything.

python snapdiff.py orders.parse_payment_request probes.json before.json

Store before.json somewhere safe. It is the only record of truth you have.

Step 3: Apply the patch

Apply the incoming change to the repository. Then run the exact same capture again.

git apply payment-simplify.patch
python snapdiff.py orders.parse_payment_request probes.json after.json

Do not change the probe set between runs. The probe set must be fixed. If you change it, you changed the experiment.

The free server tier from MonkeyCode can run the pre/post captures for you. I won't promise uptime, quotas, or durations for it. What matters is the workflow: snapshot, patch, snapshot, diff.

Step 4: Diff the behavior snapshots

# snapdiff_compare.py
import argparse
import json

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("before")
    parser.add_argument("after")
    args = parser.parse_args()
    before = json.load(open(args.before))
    after = json.load(open(args.after))
    after_by_key = {json.dumps(row["probe"], sort_keys=True): row for row in after["rows"]}
    for row in before["rows"]:
        key = json.dumps(row["probe"], sort_keys=True)
        after_row = after_by_key.get(key)
        if not after_row:
            continue
        if after_row["status"] != row["status"] or after_row["result"] != row["result"]:
            print("[CHANGED]", json.dumps(row["probe"]))
            print("  before:", row["status"], row["result"])
            print("  after: ", after_row["status"], after_row["result"])

if __name__ == "__main__":
    main()

python snapdiff_compare.py before.json after.json

The output matters more than any code review:

[CHANGED] {"args": [{"amount": 12, "currency": "USD"}]}
  before: ok {'amount': '12.00', 'currency': 'USD'}
  after:  ok {'amount': '12', 'currency': 'USD'}

[CHANGED] {"args": [{"amount": null, "currency": "USD"}]}
  before: error field required
  after:  ok {'amount': '0', 'currency': 'USD'}

The first row is a silent type change. The second row silently defaults a null amount to zero. Both changes look harmless in the diff. Both could corrupt billing later.

Step 5: Promote escaping behavior to a permanent test

Once you know what escaped, write a normal regression test for it.

def test_amount_preserves_number_type():
    result = parse_payment_request({"amount": 12, "currency": "USD"})
    assert result["amount"] == "12.00"

def test_null_amount_does_not_become_a_charge():
    with pytest.raises(ValidationError):
        parse_payment_request({"amount": None, "currency": "USD"})

The snapshot started as a review tool. It ends as permanent safety.

Why the free tier is enough

Probe generation and snapshot comparison both tolerate slow and imperfect models. A small probe set means only a few calls. The captures run in seconds, not hours.

Free tiers cover this workflow honestly. I won't name models or quote availability. Those details change every quarter. The stable fact is the workflow: cheap model call, local capture, fixed probe set, obvious diff.

Limitations

The probe set reflects what you thought to include. Unknown unknowns stay invisible.

The snapshot captures behavior, not reasons. A changed output may be intentional. A reviewer still decides what the new behavior should be.

Representation noise is real. Orderless output, generated IDs, and timestamps cause false diffs. Keep the probe set deterministic. Filter volatile fields before comparing.

The diff requires a stable target. Renaming the function or moving it to another module changes the command. Adapt the script to your repository's shape.

Who should skip this workflow

Teams with full tests already do not need snapshots. Their test suite already locks behavior. Patch review can focus on intent.

Exploratory rewrites also break this workflow. The probe set and the target change shape on every iteration. The compare step becomes friction instead of signal.

Do not use it to punish failing behavior. Use it to see the behavior. Which patch is better? The one whose behavior you can compare before you approve it.

Keep snapdiff.py in the repo. Run it before your next AI patch review.