Golden Master First: Let a Free Model Draft the Probes, Then Refactor Small

A messy repo is a behavior contract with no signature. Nobody knows what the code actually does. Refactoring without locking current behavior is gambling.

The fix is a golden master. Hash the outputs. Store the digests. Re-run after every change.

A free model can draft the edge-case probes that make the golden master useful. Then you refactor in small, budgeted diffs. This article shows the full loop.

Why characterization tests come first

Characterization tests don't assert what the code should do. They record what it does. That distinction matters in legacy code.

The current behavior is the de facto contract. Integrations depend on it, even when it looks wrong.

A golden master is the cheapest characterization harness. Feed the program known inputs. Hash stdout, stderr, and exit code. Store the digests. After a refactor, re-run and compare. Any digest change means observable behavior changed.

The weak point is input selection. Hand-written cases miss the weird edges. Empty files. Unicode. Duplicate keys. Negative numbers. A free model can propose these cases in seconds.

The workflow: five steps

Step 1 — Snapshot current behavior

Start with a small harness. It records a digest for every case.

# golden_master.py
import hashlib
import json
import subprocess
import sys
from pathlib import Path

SNAPSHOT_DIR = Path(".golden")
RUN_CMD = ["python", "legacy_app.py"]

def load_cases():
    return json.loads(Path("cases.json").read_text())

def digest_of(out):
    payload = (out.stdout + out.stderr).encode()
    return hashlib.sha256(payload).hexdigest()

def snapshot():
    SNAPSHOT_DIR.mkdir(exist_ok=True)
    for name, args in load_cases().items():
        out = subprocess.run(RUN_CMD + args, capture_output=True, text=True)
        data = {
            "args": args,
            "stdout": out.stdout,
            "stderr": out.stderr,
            "exit_code": out.returncode,
            "digest": digest_of(out),
        }
        (SNAPSHOT_DIR / f"{name}.json").write_text(json.dumps(data, indent=2))

if __name__ == "__main__":
    snapshot()

Run it once. Now the current behavior is on disk.

Step 2 — Draft probes with a free model

Hand-writing twenty edge cases is boring. A free model endpoint makes it fast. Send a prompt that asks for boundary cases only.

You are testing a legacy CLI. Propose 12 input cases that
exercise boundaries: empty input, missing files, unicode,
duplicate keys, negative numbers, large payloads, malformed JSON.
Return JSON only: [{"name": "...", "args": ["..."]}]
Never suggest modifying the program.

MonkeyCode's free model access gives you a usable endpoint for this. No paid plan needed. The prompt stays small, so token cost stays near zero.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 3 — Run the probes, keep the winners

Not every model-drafted probe is useful. Run them all against the current code. Keep the ones that exit cleanly.

def probe(model_cases):
    kept = []
    for case in model_cases:
        out = subprocess.run(RUN_CMD + case["args"], capture_output=True, text=True)
        if out.returncode == 0:
            kept.append(case)
        else:
            print(f"expected-failure: {case['name']}")
    return kept

A crashing probe is still information. It documents a known broken edge. But it cannot gate a refactor until someone fixes it.

Merge the kept probes into cases.json. Re-run the snapshot. Now the golden master covers the model-drafted edges too.

Step 4 — Refactor with a diff budget

The smallest safe change is one behavior-preserving transformation. Enforce the size mechanically. A diff budget stops the "while I'm in here" trap.

def check_diff_budget(max_lines=50):
    stat = subprocess.run(
        ["git", "diff", "--numstat"],
        capture_output=True, text=True,
    )
    total = 0
    for line in stat.stdout.splitlines():
        added, removed, _ = line.split("\t")
        total += int(added) + int(removed)
    if total > max_lines:
        sys.exit(f"Diff too large: {total} > {max_lines}")
    print(f"Diff within budget: {total} lines")

Fifty lines is a starting point. Adjust for your repo. The point is the number exists and the check runs.

Step 5 — Verify after every change

Append this to golden_master.py. All digests must match.

def verify():
    failures = []
    for snap in SNAPSHOT_DIR.glob("*.json"):
        data = json.loads(snap.read_text())
        out = subprocess.run(RUN_CMD + data["args"], capture_output=True, text=True)
        if digest_of(out) != data["digest"]:
            failures.append(snap.stem)
    if failures:
        sys.exit(f"BEHAVIOR CHANGED: {failures}")
    print("Golden master intact.")

if __name__ == "__main__":
    verify()

One changed digest means the refactor changed observable behavior. Revert or split the change. Do not argue with the harness.

Where the free server fits

The verify step is only useful if it runs often. A free server option gives it a permanent home. Point a cron job or a webhook at python golden_master.py verify after every push.

A behavior change surfaces in minutes. Not at release time. MonkeyCode's free server option covers this without a compute bill.

I did not benchmark latency or limits here. Verify those yourself for your workload.

Interpreting probe results

Probe result	Meaning	Action
Passes before and after	Characterized behavior	Keep as a regression test
Passes before, fails after	Refactor changed behavior	Revert or split the diff
Fails before and after	Known broken edge	Document as expected failure
Crashes before	Undefined behavior	Do not use as a gate

This table is the decision loop. It removes guesswork from the refactor.

Limitations

Golden masters capture outputs, not internal state. Side effects like file writes and network calls need separate handling.

Model-drafted probes are only as good as the prompt. Vague prompts produce shallow cases. Iterate on the prompt like you would on a test.

A passing probe does not prove correctness. It proves current behavior. The golden master is a safety net, not a spec.

Coverage still matters. Untested paths stay invisible to the harness. Run a coverage tool alongside it.

Who should not use this

Teams with a clean, tested codebase should write real tests instead. You already have characterization. You need specifications.

Greenfield projects have no legacy behavior to lock. Start with proper tests.

Anyone who cannot tolerate false confidence should skip golden masters. They miss what they do not probe. Pair them with coverage and code review.

The takeaway

Lock behavior first. Draft probes with a free model. Refactor small. Verify every step. That sequence turns a scary refactor into a measured one.

Consider a payment parser that accepts JSON. Nobody remembers why amount is sometimes a string. A golden master with ten probes locks that behavior. The refactor becomes a search for the smallest change that keeps all ten digests identical.

The harness in this article is about 80 lines. It runs on any Python 3 repo. Adapt the command, adjust the budget, and let the digests do the arguing.