Lock Observed Behavior Before One Messy-Repo Change

Messy repos fail before any extraction even starts. Observed behavior is the only contract you can trust. Characterization tests come before the smallest safe change.

Do not open a refactor pull request on vibes. Do not let a coding model rewrite the tree. Capture current outputs under frozen inputs first.

The failure mode this workflow targets

Untested modules mix I/O, cache, and formatting. Callers depend on incidental ordering and rounding. A cleanup extract then shifts one log line.

Production then sees the silent shift in logs. Characterization tests surface that shift before merge. The suite exists to make drift visible.

This method is not a greenfield design exercise. It is a containment method for code that already ships. The goal is a reversible cut, not a new architecture.

What you pin, and what you skip

Pin inputs that the module already consumes today. Pin outputs the rest of the system already observes. Do not pin private locals or dict identity.

A useful pin set stays small and stable. Typical pins appear in the list below. Extra pins become noise during later review.

1. Canonical fixture files used as inputs.
2. Return values and raised exception types.
3. Stdout and stderr byte streams.
4. Files written under a temp root.

Skip wall-clock timestamps unless callers print them. Skip unordered set iteration unless order already leaked. Skip cache object identity across repeated runs.

Step 1: Inventory the blast radius

List every public entry point you might touch. Count callers with search, not with memory.

rg -n "from quote import|import quote" --type py
rg -n "quote\.(price|render|flush)" --type py

Record three facts per entry point you find. Record the name, observed side effects, and known fixtures. Keep that inventory beside the tests as text.

Do not start the extract during this step. Search results are the only allowed evidence. Missing callers remain the usual production incident.

Step 2: Freeze the process envelope

Messy code reads cwd, env, and cache paths. Pin those values before the first test assertion. Unfrozen envelopes make golden files drift across machines.

# labeled proposal: envelope freeze for one module
from pathlib import Path


def freeze_envelope(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    monkeypatch.setenv("QUOTE_REGION", "us-east")
    monkeypatch.setenv("QUOTE_TZ", "UTC")
    monkeypatch.delenv("QUOTE_DEBUG", raising=False)
    (tmp_path / "cache").mkdir()

Treat this helper as a proposal until executed. Rename env keys to match the real module. Re-run pytest after each env key rename.

Step 3: Record one characterization harness

A characterization test is not a specification of intent. It records what the shipped code does today. A failing test means observed behavior has moved.

# tests/test_quote_characterize.py
import json
from pathlib import Path

import pytest

FIXTURES = Path(__file__).parent / "fixtures"
GOLDENS = Path(__file__).parent / "goldens"


def run_batch(quote, rows, capsys):
    results = []
    for row in rows:
        results.append(quote.price(row))
    captured = capsys.readouterr()
    return {
        "results": results,
        "stdout": captured.out.splitlines(),
        "stderr": captured.err.splitlines(),
    }


@pytest.mark.parametrize("name", ["batch_a.json", "batch_b.json"])
def test_quote_matches_golden(name, tmp_path, monkeypatch, capsys):
    freeze_envelope(tmp_path, monkeypatch)
    rows = json.loads((FIXTURES / name).read_text())
    import quote

    observed = run_batch(quote, rows, capsys)
    expected = json.loads((GOLDENS / name).read_text())
    assert observed == expected

Store golden files as pretty-printed JSON documents. Line-level diffs stay readable during code review. Add a hash check only as a secondary gate.

python -m pytest tests/test_quote_characterize.py -q
sha256sum tests/goldens/*.json

If the first run has no golden file, write it once. Review every field inside that golden file. Commit those goldens in a dedicated change.

Never regenerate goldens inside the refactor branch itself. A regenerated golden hides the behavior you just changed.

Step 4: Add a mutation canary

Characterization tests can miss a silent no-op harness. A canary proves those assertions can actually fail. Flip one observable field on purpose during setup.

def test_harness_detects_result_drift(tmp_path, monkeypatch, capsys):
    freeze_envelope(tmp_path, monkeypatch)
    import quote

    rows = [{"sku": "A-1", "qty": 2, "list_cents": 1999}]
    observed = run_batch(quote, rows, capsys)
    observed["results"][0]["total_cents"] += 1
    golden = json.loads((GOLDENS / "canary.json").read_text())
    assert observed != golden

If this test cannot fail, the pin set is weak. Add streams or files to the observed dict. Re-run until the canary fails on a one-cent edit.

Decision table: what to change after green

Observation	Smallest safe change	Do not do in the same PR
Duplicate tax math in two functions	Extract one pure helper	Rewrite the tax policy
Log lines interleaved with returns	Keep log order; extract formatter	Introduce async logging
Cache writes inside price()	Inject a cache protocol	Add a network cache
Fixture depends on cwd	Pass an explicit path argument	Clean every path at once

Change only one table row per pull request. Re-run the harness after that single row. If goldens move, stop and explain the delta in review.

Step 5: Use a free model only after green

A coding model is useful after the harness is green. It is not a substitute for pinned outputs. Feed it the inventory, golden paths, and one table row.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft that helper. The free server option can run the same pytest harness.

Neither statement names a model, quota, or machine. Treat both statements as availability claims only. Prompt the model with hard constraints, not style notes.

Propose one patch. Do not edit tests/goldens.
Keep quote.price signature unchanged.
Extract only the tax helper named in the table.
Stop after pytest tests/test_quote_characterize.py passes.

Reject any patch that regenerates golden files. Reject any patch that touches two table rows. Reject any patch that adds network calls.

If you already use MonkeyCode free models, keep the harness next to the patch. Compare golden hashes before you merge the helper.

Step 6: Apply the smallest safe change

Apply the extract by hand or from the draft patch. Limit the production diff to one helper module.

git checkout -b quote-tax-helper
git add quote/tax.py quote/price.py
git diff --stat
python -m pytest tests/test_quote_characterize.py -q

The expected result is a fully green suite. The golden file hashes must stay identical. If stdout order shifted, restore the old order.

Formatting-only drift still counts as behavior drift. Commit the code change without a golden update. A later golden update needs a review note naming the new behavior.

Artifact: a labeled test plan

Run this plan on a copy of the messy module. Label every result unexecuted until you run it.

1. Capture two real fixtures from staging logs.
2. Strip secrets, tokens, and customer names.
3. Freeze cwd, env, and the temp cache root.
4. Record goldens for return values and streams.
5. Add the mutation canary described above.
6. Confirm pytest fails after a one-cent flip.
7. Extract one helper listed in the table.
8. Re-run pytest and confirm hashes match.
9. Attach the inventory file to the pull request.

Time-box the first pass to one entry point. Extra entry points get their own harness files. Do not batch unrelated modules into one golden.

Limitations

Golden files encode today's bugs as well as features. They will still bless a wrong total. That is the method, not an accident.

Correctness work is a later, separate change with new tests. Do not mix a semantic fix into the extract.

The harness will not catch several classes of bugs. It misses races that need two threads. It misses behavior gated on a live network.

It also misses locale drift outside the frozen env. It misses iteration order you failed to pin.

Do not treat model patches as reviewed work. Raw diff size is not a quality signal. Unchanged goldens remain the actual quality signal.

Read the extracted helper anyway before merge. Name any leftover side effects in the inventory.

Who should not use this approach

Skip this workflow if the module has no callers. You can rewrite that module from scratch.

Skip this path if outputs contain secrets. Do not write secrets into golden files. Skip it if you need a semantic fix now.

This workflow freezes observed behavior on purpose. A semantic fix needs new tests, not recycled goldens.

Also skip it if you cannot freeze time, locale, and cwd. Unfrozen process envelopes produce flaky golden files. Flaky goldens train the team to ignore red tests.

What done looks like

Done means a green characterization test suite. Done also means exactly one structural cut today. A closed decision-table row completes the change.

A cleaner architecture diagram is not the done state. Keep the golden files in version control. Keep the inventory file next to them.

Delete the branch if goldens moved without a written reason. Unexplained golden drift means a failed extract.