Most legacy repos cannot be refactored safely on the first pass. You do not know which behaviors are load-bearing. Guessing produces regressions that surface three sprints later.
The fix is mechanical. Capture what the code actually does, freeze it as files, then make the smallest change that keeps those files identical. This post shows the harness, the normalization rules, and the review gate.
Everything below runs with Python 3.8+ and pytest. No plugins are required.
The baseline rule
A refactor changes structure without changing observable behavior. Observable means bytes on stdout, stderr, exit codes, and written files. If you cannot compare those, you are rewriting, not refactoring.
So the first commit is never the refactor. The first commit is the baseline harness. Keep it in its own pull request so reviewers can confirm it is behavior-neutral.
Step 1: Collect real inputs
Do not invent synthetic fixtures first. Pull 8 to 20 real inputs from logs, saved payloads, and bug reports. Store them under tests/cases as plain files.
Selection criteria:
- At least one input that triggers each early return.
- At least one input that hits the slowest path.
- At least one input that caused a past incident.
Small inputs are better. You want fast tests, not representative load.
Step 2: Capture one artifact per case
Run the real entry point as a subprocess. Capture exit code, stdout, and stderr into a single golden file.
# tests/test_characterization.py
import difflib
import os
import pathlib
import subprocess
import sys
import pytest
ROOT = pathlib.Path(__file__).resolve().parents[1]
GOLDEN = ROOT / 'tests' / 'golden'
CASES = sorted((ROOT / 'tests' / 'cases').glob('*.json'))
def render(proc):
return f'''exit={proc.returncode}
--- stdout ---
{proc.stdout}--- stderr ---
{proc.stderr}'''
def run_case(case_path):
env = dict(os.environ, TZ='UTC', PYTHONHASHSEED='0', LEGACY_SEED='0')
return subprocess.run(
[sys.executable, '-m', 'legacy.cli', '--input', str(case_path)],
cwd=ROOT, capture_output=True, text=True, timeout=30, env=env,
)
@pytest.mark.parametrize('case_path', CASES, ids=lambda p: p.stem)
def test_cli_matches_golden(case_path):
actual = render(run_case(case_path))
golden_file = GOLDEN / (case_path.stem + '.txt')
if os.environ.get('UPDATE_GOLDEN') == '1':
golden_file.write_text(actual)
pytest.skip('golden rewritten')
expected = golden_file.read_text()
if actual != expected:
diff = difflib.unified_diff(
expected.splitlines(True), actual.splitlines(True),
'golden', 'actual',
)
pytest.fail(''.join(diff))
Three details matter here. The subprocess uses the real argv path, not an imported function. The environment is pinned so locale and hash seed cannot drift. The failure message is a unified diff, not a bare assertion.
Step 3: Scrub only genuine nondeterminism
A noisy harness gets disabled within a week. Scrub the few fields that legitimately change between runs.
| Source | Symptom | Fix |
|---|---|---|
| Timestamps | diff on every run | normalize with a regex |
| Temp paths | random suffixes | replace with a placeholder |
| UUIDs | new id per run | mask with a fixed token |
| Dict ordering | field order flips | sort keys before printing |
| Locale | comma vs dot decimals | pin TZ and LC_ALL |
import re
SCRUBBERS = [
(re.compile(r'ts=\d{4}-\d{2}-\d{2}T[\d:.]+Z'), 'ts=<TS>'),
(re.compile(r'/tmp/tmp[a-z0-9_]{8}'), '<TMP>'),
(re.compile(r'\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b'), '<UUID>'),
]
def scrub(text):
for pattern, replacement in SCRUBBERS:
text = pattern.sub(replacement, text)
return text
Keep the scrubber list short and reviewable. Every rule you add hides a class of changes from the gate.
Step 4: Write goldens, then prove they can fail
Generate the baseline in one command, then commit it untouched.
mkdir -p tests/cases tests/golden
UPDATE_GOLDEN=1 python -m pytest tests/test_characterization.py -q
git add tests/golden && git commit -m 'pin observed CLI behavior'
python -m pytest tests/test_characterization.py -q # expect: 12 passed
A green suite proves nothing yet. Mutate one field the scrubber is supposed to catch, rerun, and watch it stay green. That is how you learn your regex never matched.
Step 5: Choose the smallest safe change
Rank candidate changes by blast radius, not by elegance.
| Change | Blast radius | Test cost | Do it now |
|---|---|---|---|
| Extract one pure predicate | one call site | low | yes |
| Rename a local variable | one function | none | yes |
| Move a module | import graph | medium | after baseline holds |
| Replace a framework | whole app | high | no |
| Change output formatting | every caller | high | no |
One change per commit. If the diff touches two call sites, split it into two commits.
Step 6: Extract one predicate
Start with a boolean expression buried in a loop.
# before: legacy/retention.py
for artifact in artifacts:
if artifact['age_days'] > 30 and artifact['size_mb'] > 50 and not artifact['pinned']:
to_evict.append(artifact)
After extraction:
# after: legacy/retention.py
def is_evictable(artifact):
return (
artifact['age_days'] > 30
and artifact['size_mb'] > 50
and not artifact['pinned']
)
for artifact in artifacts:
if is_evictable(artifact):
to_evict.append(artifact)
Check the shape of the change before running anything.
git diff --stat # expect: 1 file changed, +9 -1
python -m pytest tests/test_characterization.py -q
If the diff is larger than expected, revert. A refactor that grows six files is a redesign wearing a refactor label.
Step 7: Break it on purpose
Perturb the new function and confirm the harness fails.
python - <<'PY'
import pathlib
p = pathlib.Path('legacy/retention.py')
p.write_text(p.read_text().replace('> 30', '>= 30'))
PY
python -m pytest tests/test_characterization.py -q # must fail
git checkout legacy/retention.py
This is the cheapest regression proof you will ever buy. Run it once per extracted unit, and record the result in the pull request description.
Where free tooling fits
Two parts of this loop are tedious rather than difficult: replaying cases in a clean sandbox, and drafting normalization regexes from noisy diffs. Free model access can help with the second, and a free server option can host the first.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The free model access and free server option are operator-supplied availability claims. I have not benchmarked them here, and this article makes no claim about model quality, quotas, or uptime. Verify current limits, data retention, and terms before routing anything through them.
A practical split:
- Keep the golden files themselves local. They often contain real payload fragments.
- Send only scrubbed or synthetic payloads to a remote sandbox for replay runs.
- Treat free capacity as shared and non-guaranteed, so never put it on a release-blocking path.
Limitations and who should skip this
Golden-file harnesses are weak against nondeterministic systems. If concurrency, wall-clock timing, or network calls dominate behavior, diff noise will drown real regressions. Fix determinism first, or scope the harness to one pure submodule.
Skip this approach if:
- Output is already intentionally unstable and undocumented.
- The team will not review golden files as code in pull requests.
- You need coverage metrics for a compliance audit.
Start narrow
Pick the module nobody wants to touch. Capture 8 to 20 cases, commit the goldens, extract one predicate. Then stop and review before touching anything else.
The habit matters more than the harness. Behavior first, structure second, one change per commit.
Top comments (1)
Selecting real inputs from logs and past incidents instead of synthetic fixtures is the part that makes this approach trustworthy, and pinning the time zone and hash seed removes a lot of flaky diffs. One addition worth considering is a few tests for the normalization rules themselves, because a normalizer that is too aggressive can mask a genuine behavior change, which defeats the purpose of the golden files.