Pin Golden Call Logs Before One Messy-Repo Change

A messy module survives change only after behavior is pinned. Characterization tests record current outputs on known inputs. One structural edit can follow that recorded freeze.

Skip the freeze and refactors invent silent bugs. Golden call logs catch silent output drift early. This workflow treats present code as the spec.

The failure mode

Legacy files still mix I/O, branching, and formatting. Broad AI patches often rewrite those layers together. The file looks cleaner and then fails in production.

Unmeasured edits skip a replayable baseline for callers. Engineering work needs a frozen public call surface. Record that surface before any structure moves later.

What a characterization test is

The test does not encode any intended design. It freezes observed behavior on a finite seed. Later deltas must match that recorded freeze exactly.

The suite is a tripwire, not a design document. Replace it after the behavior is truly understood. Until then, the freeze remains the only contract.

Artifact: a golden call log

The listing below is proposed, not production history. It is proposed and unexecuted sample code only. Adapt the file paths to your own tree.

Consider a tangled alert router module as stand-in. It reads mixed ticket dictionaries from its callers. It returns one destination string per ticket.

# proposed stand-in: legacy_router.py

def route_alert(ticket):
    sev = ticket.get("sev") or "info"
    team = ticket.get("team") or "ops"
    tags = ticket.get("tags") or []
    if sev == "crit" and team != "sec":
        return f"pager:{team}"
    if "security" in tags:
        return "pager:sec"
    if sev == "warn":
        return f"slack:{team}"
    return f"log:{team}"

Do not extract a helper from this module yet. Pin the public call log first, without extracts. The log is the only safe contract today.

Step 1 — Inventory the public surface

List functions that callers actually import today first. Ignore private helpers during this first freeze pass. One public surface keeps the first log small.

rg -n "^def " legacy_router.py

Write those names into a short manifest file. One short manifest is enough for this pass. Revisit helpers after the public log stays green.

# proposed: surface.txt
legacy_router.route_alert

Step 2 — Sample a compact input set

Pick cases that hit each visible branch once. Keep the input set boring, finite, and replayable. Twelve tickets beat a generator nobody can replay.

# proposed: samples.py
SAMPLES = [
    {"sev": "crit", "team": "ops", "tags": []},
    {"sev": "crit", "team": "sec", "tags": []},
    {"sev": "info", "team": "ops", "tags": ["security"]},
    {"sev": "warn", "team": "ops", "tags": []},
    {"sev": "info", "team": "ops", "tags": []},
    {"sev": None, "team": None, "tags": []},
    {"sev": "crit", "team": "sec", "tags": ["security"]},
    {},
]

Missing keys change control flow in real routers. Null teams change the default destination path here. Tags that collide with severity change the winner.

Step 3 — Normalize, then record the golden log

Unstable fields make a freeze worthless fast. Sort keys and drop clocks before serialize. Random ids need stubs, not raw capture.

# proposed: normalize.py
import json
from collections import OrderedDict

def freeze_ticket(ticket):
    data = ticket if isinstance(ticket, dict) else {}
    items = sorted((str(k), data[k]) for k in data)
    return OrderedDict(items)

def dump_rows(rows, path):
    text = json.dumps(rows, indent=2, sort_keys=True)
    path.write_text(text + "\n")

Run the live function against the finite seed. Serialize each input and output pair as JSON. Commit that JSON file as the behavior freeze.

# proposed: record_golden.py
import json
from pathlib import Path
from legacy_router import route_alert
from samples import SAMPLES
from normalize import freeze_ticket, dump_rows

def record(path=Path("golden_routes.json")):
    rows = []
    for ticket in SAMPLES:
        frozen = freeze_ticket(ticket)
        rows.append({
            "input": frozen,
            "output": route_alert(dict(frozen)),
        })
    dump_rows(rows, path)

if __name__ == "__main__":
    record()

python record_golden.py

Inspect the JSON by hand before the commit. Garbage in the freeze becomes lasting law later. Delete bad rows before the git add step.

[
  {
    "input": {"sev": "crit", "tags": [], "team": "ops"},
    "output": "pager:ops"
  }
]

That snippet is a proposed shape only. Your freeze will contain every seed row. Commit the full file, not a partial paste.

Step 4 — Replay on every change

The test loads the committed freeze file first. It calls the same function with each input. It compares the returned strings with exact equality.

# proposed: test_characterize_router.py
import json
from pathlib import Path
from legacy_router import route_alert
from normalize import freeze_ticket

def test_golden_routes():
    rows = json.loads(Path("golden_routes.json").read_text())
    assert rows, "golden log must not be empty"
    for row in rows:
        ticket = dict(freeze_ticket(row["input"]))
        assert route_alert(ticket) == row["output"]

python -m pytest test_characterize_router.py -q

A red test means observed behavior moved somewhere. That failure is the suite doing its job. Do not edit the freeze to match a refactor.

Step 5 — Draft extra cases with a free model

Gaps in the seed still hide untested branches. A coding model can propose extra tickets from source. You still record outputs from the live module.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Use the model to suggest missing inputs only. Use the server to run the recorder and pytest. Run the recorder there if local pytest is noisy.

Paste the router source and the current seed. Ask for uncovered branches, not a full rewrite. Merge accepted proposals into SAMPLES by hand only.

Then re-run record_golden.py against the real live function. Never let the model invent expected output strings. The live module remains the only output oracle.

Proposed prompt, unlabeled as a production run:
Given legacy_router.py and SAMPLES, list tickets that hit
uncovered branches. Return JSON dicts only. Do not invent outputs.

Treat every suggested dict as a candidate, not a fact. Drop tickets that cannot occur in callers. Then record again from the live function.

Step 6 — Make the smallest safe change

One change still means one structural move only. Allowed moves include a local rename or constant extract. Stop after that single commit stays fully green.

Decision table for the first change

Candidate change	Touches I/O?	Touches branch order?	Allowed now?
Rename local sev to severity	No	No	Yes
Extract DEFAULT_TEAM = "ops"	No	No	Yes
Reorder crit and security checks	No	Yes	No
Swap the return for a logger call	Yes	No	No
Split route_alert into two functions	No	Maybe	Only after green replay

If the golden suite is green, apply one allowed row. Re-run the pytest suite immediately after that edit. If the suite fails, revert the change at once.

# proposed smallest change: extract a constant only
DEFAULT_TEAM = "ops"

def route_alert(ticket):
    sev = ticket.get("sev") or "info"
    team = ticket.get("team") or DEFAULT_TEAM
    tags = ticket.get("tags") or []
    if sev == "crit" and team != "sec":
        return f"pager:{team}"
    if "security" in tags:
        return "pager:sec"
    if sev == "warn":
        return f"slack:{team}"
    return f"log:{team}"

python -m pytest test_characterize_router.py -q
git add legacy_router.py
git commit -m "extract DEFAULT_TEAM after golden route freeze"

Do not batch three tidy-ups in one commit. Batching hides which edit drifted the output later. One commit keeps git bisect cheap later.

Step 7 — Expand the freeze after a green change

New helpers will need their own logs later. Do not start that work in the same commit. Public behavior stays pinned until callers move over.

When you add a sample, record the log again. Diff the JSON file against the last freeze. Only new rows should appear in that diff.

python record_golden.py
git diff golden_routes.json

Unexpected line changes mean the last edit leaked. Restore both the function and the freeze file. Repeat step six with a smaller structural edit.

Command checklist

1. Inventory public def names with a tight rg pass.
2. Seed a finite input list that covers each branch.
3. Record JSON outputs from the live function only.
4. Commit the JSON freeze before any structural edit.
5. Replay the freeze with pytest on every save.
6. Propose extra inputs from a model, never outputs.
7. Apply one allowed structural change, then stop.
8. Diff the golden file after any later re-record.

Run the checklist in that exact listed order. Skipping the record-before-edit step voids the suite. That order is the method, not a suggestion.

What belongs in the freeze

Pin return values, raised types, and stable strings. Skip wall-clock stamps and unordered hash seeds. Those fields need stubs before the first record.

# proposed: reject unstable values before commit
BANNED_SUBSTRINGS = ("T00:", "uuid:", "0x")

def assert_stable(output):
    text = str(output)
    for token in BANNED_SUBSTRINGS:
        assert token not in text

Call that guard inside the recorder path. Unstable goldens train the team to ignore red. Ignored red tests have no engineering value.

Limitations

Characterization tests can lock existing bugs in place. That tradeoff is acceptable for a first freeze. It is not a substitute for later intent tests.

Exact string compares fail on timestamps and unordered maps. Normalize clocks and sort keys before you serialize. Otherwise the suite flakes and then gets deleted.

The seed will miss some rare production branches. A model-suggested ticket is still only a guess. Production traces beat synthetic dicts when traces exist.

Free model access does not certify branch coverage. Free server runs do not replace human review. Someone still has to read the JSON freeze.

This workflow assumes a deterministic, mostly pure function. Networked routers need stubs around the freeze path. Time-based routers need injected clocks in the seed.

Who should not use this

Do not use this method on greenfield code. Write intent tests for those new modules instead. A golden log would freeze those early accidents.

Do not use this method to justify a large rewrite. The method allows only one small measured change. A rewrite still needs a real behavioral spec.

Do not use this when outputs must change on purpose. Update the product spec before touching the freeze. Then rebuild the golden log as an intentional act.

Do not outsource the oracle to any model. Models may draft extra input dictionaries for review. Only the running module writes those output strings.

What this does not claim

No runtime speedup numbers appear in this article. No model ranking appears in this article either. The only claimed value is a replayable behavior freeze.

If every product mention disappeared, the harness would still work. Local pytest remains sufficient for the whole loop. A remote runner is optional extra run capacity.

Stop after one green structural commit lands cleanly. Then pick the next allowed table row only. Messy repos shrink by measured steps, not mood.