A Golden File Is Not a Test: The Byte-Level Refactor Safety Net

A golden file is not a test. It is a recording of current behavior. Tests assert what should happen. A golden file asserts what did happen. In a messy repo, that is the only spec you own.

The smallest safe change has a sharp definition. It changes nothing observable. Code diffs show edits, not behavior. Reviewers read patches, not execution. That gap explains the reviewer discussion on DEV. AI promoted every developer to reviewer. Nobody tested the reviewer. A fingerprint closes the gap. Record behavior before the patch. Diff behavior after. Let the digest do the arguing.

This workflow targets one function at a time. It works where comments are lies and tests are missing. It fails where behavior is non-deterministic. Use it before you trust any refactor, human or generated.

Why the patch review is not enough

A code review compares the patch with the surrounding code. It cannot compare behavior before and after. The human eye reads intent. The digest reads execution. For generated patches, that difference is fatal. Models produce plausible diffs. Plausibility is not preservation.

Step 1: Pick one function, not a module

A module is a swamp. A function is a bounded unit. Pick the function with the highest mystery score: branch count plus call count minus explicit tests.

import ast
from pathlib import Path

def mystery_score(path: Path, name: str) -> int:
    tree = ast.parse(path.read_text())
    func = next(
        n for n in ast.walk(tree)
        if isinstance(n, ast.FunctionDef) and n.name == name
    )
    return sum(
        isinstance(n, (ast.Call, ast.If, ast.Assign, ast.Attribute))
        for n in ast.walk(func)
    )

Score every function in the file. Sort descending. Take the top. That is your first candidate. Ignore the urge to refactor the whole chain.

Step 2: Capture real inputs

Golden files need realistic seeds. Synthetic inputs encode your assumptions. Real calls expose empty strings, nulls, and states only production produces.

Record calls with a wrapper at the boundary.

import functools
import json

def record_calls(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        with open('cases.jsonl', 'a') as fh:
            fh.write(json.dumps({
                'id': sum(1 for _ in open('cases.jsonl')),
                'args': list(args),
                'kwargs': kwargs,
            }) + '\n')
        return result
    return wrapper

The wrapper observes. It does not assert. It does not mutate. If the function touches state, record the state too.

Run it in staging for one day. Collect a few hundred cases. Remove the wrapper after collection. The corpus is raw material, not proof.

Step 3: Record the golden file

Execute every captured case. Hash the observable result. Include the return value and the exception type. Store the digests.

#!/usr/bin/env python3
import hashlib
import importlib
import json
import sys

def load_cases(path):
    return [json.loads(line) for line in open(path) if line.strip()]

def run_once(func, case):
    try:
        return {'ok': True, 'output': func(*case['args'], **case['kwargs'])}
    except Exception as exc:
        return {'ok': False, 'error': f'{type(exc).__name__}: {exc}'}

def digest(record):
    payload = json.dumps(record, sort_keys=True, default=str).encode()
    return hashlib.sha256(payload).hexdigest()

def main():
    module, name, corpus, golden, mode = sys.argv[1:6]
    func = getattr(importlib.import_module(module), name)
    rows = []
    for case in load_cases(corpus):
        rec = run_once(func, case)
        rows.append({'id': case['id'], 'digest': digest(rec)})
    if mode == 'record':
        json.dump(rows, open(golden, 'w'), indent=2)
        print(f'recorded {len(rows)} cases')
    else:
        expected = {r['id']: r['digest'] for r in json.load(open(golden))}
        changed = [r['id'] for r in rows if r['digest'] != expected.get(r['id'])]
        print(f'{len(rows) - len(changed)}/{len(rows)} unchanged')
        if changed:
            print('changed: ' + ', '.join(map(str, changed)))
            sys.exit(1)

if __name__ == '__main__':
    main()

Usage:

python fingerprint.py payments apply_discount cases.jsonl golden.json record
python fingerprint.py payments apply_discount cases.jsonl golden.json verify

The first run produces your behavior snapshot. Treat it like a crime-scene photo. It documents current bugs as well as current features.

A changed digest is a confession. It names the exact case that broke. That case, not the whole suite, gets your attention.

Step 4: Apply the smallest change

Now edit. One behavior-preserving mutation at a time. Rename a local. Extract an expression. Inline a constant. Re-run the verifier after each edit.

Keep edits smaller than the diff looks. If the edit touches three call sites, it is three edits. Verify after each. The verifier is cheap. The rollback is expensive.

A green verify means the edit is invisible. A red verify means you touched observable behavior. Red is not automatically bad. It means the change needs its own decision, not a rubber stamp.

This is where free model assistance earns a place. Drafting wrappers and initial patches is scaffolding. A free model produces that scaffolding fast. MonkeyCode's free model access and free server option cover this loop: draft, run, verify, iterate. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Verification stays mechanical. The model proposes. The digest decides.

Step 5: Judge the diff, not the patch

Compare two things. First, the fingerprint diff: which case IDs changed. Second, the code diff: what edit caused them.

Identical fingerprints mean behavior is locked. Review the code for style and intent. Different fingerprints mean behavior changed. Review it as a feature, not a refactor.

The decision table

Not every mess needs this loop. Match the tool to the repo state.

Repo state	Tool	Smallest safe change
No tests, visible call sites	Golden file fingerprint	Rename, extract, inline
Brittle tests, partial coverage	Targeted unit characterization	Behavior-preserving edit under tests
No logs, many call sites	Boundary wrapper first	Add a guard, not an edit
Trusted coverage exists	Normal TDD refactor	Anything the tests defend

When this workflow breaks

Golden files fail on non-determinism. Timestamps, random values, and network calls produce false reds. Sample them out or freeze them with dependency injection. If the function reads a clock, inject a fixed time. If it calls an API, record the response. Freeze the world, then fingerprint it.

Golden files lock bugs. If current behavior is the bug, the fingerprint defends it. Write a failing unit test first. Shift the boundary, then refactor.

Large corpora slow the loop. Keep the corpus under a few hundred cases. You want signal, not coverage theater.

Who should skip this

Skip it when trusted unit tests already exist. Skip it for known-wrong security-sensitive behavior. Skip it when side effects are uncapturable. A return-value fingerprint will not see the file write or the queue message. Wrap the boundary and assert on effects themselves.

The rule

A refactor is safe when the fingerprint is boring. Boring is a compliment. Identical digests mean merge. Changed digests mean you shipped a behavior change. Admit it. Name it. Test it. Merge it as a feature.

The smallest safe change is the one nobody has to trust on vibes. Next refactor, record the fingerprint first.