Freeze Return Values, Then Budget the Diff: Two Gates for a Refactor You Can Actually Review

A recurring failure mode in inherited codebases looks like this: a module has no tests, an AI agent produces a 900-line "cleanup" diff in one pass, and the reviewer approves it because the code reads better. Two weeks later a downstream report is off by a few cents and nobody can tell which of the 900 lines did it. The refactor was not wrong in style, but it was unreviewable in size.

Characterization tests are the standard answer to the first half of that problem, and they work. The gap is the second half: tests pin behavior, but nothing pins the size of the change that claims to preserve it. This article describes two small gates that close that gap — freeze the return values first, then spend a hard diff budget on every commit — and shows where free infrastructure fits into the loop.

Why a green characterization suite is not enough

A characterization suite tells you whether behavior moved. It does not tell you how much surface you touched to get there, so a failing assertion can arrive attached to 60 unrelated edits. Reviewers then face a diff that mixes a genuine behavior change with cosmetic churn, and the cheapest resolution is usually to trust the author.

Two independent constraints fix this, and they are cheap to build:

1. Contract freeze — every public function you plan to touch gets a recorded input/output corpus committed before the first edit.
2. Diff budget — every commit is measured in changed lines outside the freeze artifacts, and the commit fails if it exceeds a number you chose in advance.

Gate 1: freeze the outputs, not the lines

Line-level coverage is a proxy. A frozen return value is evidence. Pick the exported functions on the path you intend to change, then record what they return for a corpus you can defend.

Step 1 — write a small corpus by hand

Start with 8–15 cases per function: the empty input, the single element, the duplicate key, the value that used to break, and the one a customer reported. Keep it in a plain JSON file so a reviewer can read the intent without running anything.

{
  "billing.legacy_invoice": [
    {"case_id": "empty-cart", "args": [[]], "kwargs": {"currency": "USD"}},
    {"case_id": "one-item", "args": [[{"sku": "A1", "qty": 1, "cents": 1299}]], "kwargs": {"currency": "USD"}},
    {"case_id": "dup-sku", "args": [[{"sku": "A1", "qty": 1, "cents": 1299}, {"sku": "A1", "qty": 2, "cents": 1299}]], "kwargs": {"currency": "USD"}}
  ]
}

Step 2 — record the current behavior

The script below runs on the unmodified code, normalizes outputs so they stay comparable, and writes one contract file per function. Note the explicit normalize hook: floats are rounded, dict keys sorted, datetimes stringified. That hook is where you decide what counts as a behavior change.

# tools/freeze_contracts.py
import hashlib, importlib, json, pathlib

ROOT = pathlib.Path(__file__).resolve().parents[1]
OUT = ROOT / "tests" / "contracts"

def normalize(value):
    """Make outputs comparable across a refactor. Extend per repo."""
    if isinstance(value, float):
        return round(value, 6)
    if isinstance(value, dict):
        return {k: normalize(v) for k, v in sorted(value.items())}
    if isinstance(value, (list, tuple)):
        return [normalize(v) for v in value]
    if hasattr(value, "isoformat"):
        return value.isoformat()
    return value

def freeze(module_name, func_name, cases):
    fn = getattr(importlib.import_module(module_name), func_name)
    records = []
    for case in cases:
        args, kwargs = case.get("args", []), case.get("kwargs", {})
        out = normalize(fn(*args, **kwargs))
        records.append({
            "case_id": case["case_id"],
            "args": args,
            "kwargs": kwargs,
            "out_sha": hashlib.sha256(
                json.dumps(out, sort_keys=True, default=str).encode()
            ).hexdigest()[:12],
            "out": out,
        })
    OUT.mkdir(parents=True, exist_ok=True)
    target = OUT / f"{module_name}.{func_name}.json"
    target.write_text(json.dumps(
        {"module": module_name, "function": func_name, "records": records},
        indent=2, sort_keys=True,
    ))
    print(f"froze {len(records)} cases -> {target.relative_to(ROOT)}")

if __name__ == "__main__":
    corpus = json.loads((ROOT / "corpus.json").read_text())
    for key, cases in corpus.items():
        module_name, func_name = key.rsplit(".", 1)
        freeze(module_name, func_name, cases)

Step 3 — replay the contract as a test

The replay test is deliberately dumb: no fixtures, no mocks, no setup graph. It re-executes each recorded call and compares the normalized output, so a failure names the function and the case id rather than a line number.

# tests/test_contracts.py
import importlib, json, pathlib
import pytest
from tools.freeze_contracts import normalize

FROZEN = sorted(pathlib.Path("tests/contracts").glob("*.json"))

@pytest.mark.parametrize("path", FROZEN, ids=lambda p: p.stem)
def test_contract(path):
    contract = json.loads(path.read_text())
    fn = getattr(importlib.import_module(contract["module"]), contract["function"])
    for rec in contract["records"]:
        got = normalize(fn(*rec["args"], **rec["kwargs"]))
        expected = normalize(rec["out"])
        assert json.dumps(got, sort_keys=True, default=str) == \
               json.dumps(expected, sort_keys=True, default=str), rec["case_id"]

Commit corpus.json, tests/contracts/, and the replay test before you touch the module. That commit is the baseline; everything after it is measured against a recorded past.

Gate 2: spend a diff budget

A budget turns "smallest safe change" from an intention into a failing build. Pick a number you can review in one sitting — 40 changed lines outside the freeze artifacts is a reasonable default for a first pass — and count additions plus deletions per file.

# tools/diff_budget.py
import fnmatch, subprocess, sys

BASE = sys.argv[2] if len(sys.argv) > 2 else "origin/main"
BUDGET = int(sys.argv[1]) if len(sys.argv) > 1 else 40
FREE = ["tests/contracts/*", "tests/test_contracts.py", "corpus.json"]

def rows():
    out = subprocess.run(
        ["git", "diff", "--numstat", f"{BASE}...HEAD"],
        capture_output=True, text=True, check=True,
    ).stdout
    for line in out.splitlines():
        add, dele, path = line.split("\t")
        yield int(add or 0), int(dele or 0), path

def main():
    spent = 0
    for add, dele, path in rows():
        if any(fnmatch.fnmatch(path, p) for p in FREE):
            continue
        spent += add + dele
        print(f"{path}: +{add} -{dele}")
    print(f"diff budget: {spent}/{BUDGET}")
    if spent > BUDGET:
        print("FAIL: shave the change or split the commit", file=sys.stderr)
        sys.exit(1)

main()

The freeze paths are excluded on purpose, because contract files are allowed to be large. Run it locally before pushing, and once in CI on the pull request:

python tools/freeze_contracts.py
pytest tests/test_contracts.py -q
git add corpus.json tests/contracts tests/test_contracts.py
git commit -m "test: freeze legacy_invoice contract (3 cases)"

# each subsequent refactor commit
python tools/diff_budget.py 40 && pytest tests/test_contracts.py -q

When freezing actually pays off

Repo shape	Freeze first?	Diff budget?	Why
Legacy module, no tests, pure functions	Yes	Yes	Cheap to record, high review value
Module with heavy I/O and DB writes	Partially	Yes	Freeze only the transport-free layer
Nondeterministic output (timestamps, UUIDs)	After normalizing	Yes	Budget alone still caps blast radius
Already well-tested service	Rarely	Sometimes	Existing suite already provides the pin

Where free model access and free server fit

The loop above has two resource-hungry steps: expanding the corpus, and replaying it against a module that is slow to import. Both are good candidates for a free tier, if you keep a human on the gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Concretely, the free model access is useful for the tedious part of Gate 1 — proposing corpus cases from a function signature and a handful of real call sites. Treat every proposal as a draft: read it, delete the duplicates, and keep only cases whose expected output you would defend in review. The free server option covers the replay side, so a suite that takes twenty minutes to import and run does not block your laptop while you make the next edit.

A few honest boundaries for that setup:

• Do not paste secrets, credentials, or code your organization has not cleared for third-party processing.
• Free capacity is shared and can be slower or interrupted, so keep the budget check and the freeze replay running locally as the authoritative gate.
• A model's suggested corpus case is a hypothesis about behavior, not evidence of it. Only the recorded out_sha from your own run is evidence.

If you want to try the freeze-then-budget loop without provisioning anything first, running the two commands above on a free server session is the shortest path in.

Failure modes to expect

Non-serializable returns. If a function returns a database handle or a lambda, json.dumps will produce a stringified address that changes every run. Freeze a projection of the object instead, or skip that function and rely on the budget alone.

Over-normalized outputs. Rounding floats to six places and sorting keys hides real changes. Decide the tolerance explicitly and write it in a comment next to normalize, so the next person does not silently loosen it.

Budget gaming. A motivated author can split one large change into ten commits that each pass. The gate is a speed bump, not a proof; review the whole branch diff as well as the commit sequence.

Contract drift. If the frozen outputs were recorded against a branch that already diverged from main, the replay fails for the wrong reason. Freeze against the exact commit you will refactor from, and record that SHA in the same commit message.

Who should skip this

Skip the freeze half if your module's outputs are inherently nondeterministic and expensive to normalize, or if a real test suite already covers the path — you would be duplicating coverage. Skip the budget half if your team reviews by feature scope rather than by commit, because the number will feel arbitrary and get disabled within a month.

Skip both if the change is genuinely a rewrite. A rewrite has no "smallest safe" version, and pretending otherwise produces a freeze corpus that documents behavior nobody wants to preserve. Say it is a rewrite, and plan the migration separately.

The two gates are small on purpose: one JSON file per function, one forty-line counter, one exclusion list. Their value is not cleverness but the fact that they make an unreviewable diff fail before a reviewer has to be the one to say no.