Characterization tests should precede every AI-assisted messy-repo refactor.
Capture one function's real calls before you rewrite anything.
Then change only that function body, nothing else.
Coding agents still assume behavior they never observed.
That guess is the usual source of silent branch deletions.
A recorded contract beats a confident but untested diff.
Why speculative rewrites fail
Messy modules mix policy, I/O, and forgotten edge cases.
A model rewrite often simplifies a branch you still need.
Callers keep compiling while observed production results drift.
Existing unit tests often miss the function you will touch.
They cover neighbors, not the actual impure core.
You need observed pairs from a run you already trust.
What you freeze
Freeze arguments, return values, and exception class names.
Do not freeze timestamps, process ids, or memory addresses.
Normalize those fields before you write JSONL rows.
Freeze one function, not the whole package import graph.
Callers stay untouched until the body change is green.
Signature changes belong in a later, separate pull request.
Why one function is the unit
A whole-file rewrite hides the first behavioral break.
One function keeps the diff inside a single review surface.
Stack several such changes only after each oracle stays green.
Shared helpers used by other functions are out of scope.
Move helper extraction to a later characterization pass.
Characterize the helper itself before you rename it.
Artifact: a JSONL probe
The example below is a proposal, not a production trace.
It records one pricing helper during a fixture script.
Adapt field names to your real module before you run it.
1. Isolate the target
Pick the function with the highest local complexity.
Prefer a function your tests already invoke indirectly.
Do not start with a public API you plan to rename.
# proposal: messy.py — current behavior, not a rewrite
from decimal import Decimal
def quote_line(qty, unit_cents, coupon):
if qty < 0:
raise ValueError("qty")
raw = qty * unit_cents
if coupon == "HALF" and qty >= 2:
raw = raw // 2
if coupon == "CAP":
raw = min(raw, 999)
return Decimal(raw) / Decimal(100)
2. Wrap with a recorder
Keep the original function under a private name.
Write one JSONL line per call, including exceptions.
Skip writes when PROBE_JSONL is unset in the environment.
# proposal: probe.py
import json
import os
from messy import quote_line as _quote_line
LOG = os.environ.get("PROBE_JSONL", "")
def _normalize(value):
if isinstance(value, dict):
return {k: _normalize(v) for k, v in sorted(value.items())}
if isinstance(value, (list, tuple)):
return [_normalize(v) for v in value]
return value
def quote_line(qty, unit_cents, coupon):
record = {
"qty": qty,
"unit_cents": unit_cents,
"coupon": coupon,
}
try:
result = _quote_line(qty, unit_cents, coupon)
record["ok"] = True
record["result"] = str(result)
return result
except Exception as exc:
record["ok"] = False
record["exc_type"] = type(exc).__name__
record["exc_msg"] = str(exc)
raise
finally:
if LOG:
row = json.dumps(_normalize(record), sort_keys=True)
with open(LOG, "a", encoding="utf-8") as fh:
fh.write(row + "\n")
Route imports through the wrapper only during capture.
Do not ship the wrapper to production processes.
Require PROBE_JSONL in the capture shell before pytest.
3. Capture from a known suite
Run the suite you already trust, not a new driver.
The probe must see the same imports the tests use.
Set PROBE_JSONL in the same shell as pytest.
# proposal: capture once from the existing fixture
export PYTHONPATH="$PWD:$PWD/probe_path"
export PROBE_JSONL="$PWD/quote_line.jsonl"
rm -f "$PROBE_JSONL"
python -m pytest tests/test_checkout.py -q
wc -l "$PROBE_JSONL"
Reject an empty log, because zero rows mean a missed probe.
Fix import routing before you generate any tests.
4. Deduplicate and freeze rows
Duplicate calls add noise without adding new contracts.
Sort keys so equivalent objects collapse to one row.
# proposal: freeze_jsonl.py
import json
import sys
seen = set()
src_path, dst_path = sys.argv[1], sys.argv[2]
with open(src_path, encoding="utf-8") as src, open(dst_path, "w", encoding="utf-8") as dst:
for line in src:
row = json.loads(line)
key = json.dumps(row, sort_keys=True)
if key in seen:
continue
seen.add(key)
dst.write(key + "\n")
print(len(seen))
python freeze_jsonl.py quote_line.jsonl quote_line.frozen.jsonl
The frozen file is the oracle for this refactor.
Commit it before any model sees the function body.
Do not edit rows to match a desired future design.
5. Turn rows into tests
Replay every frozen row against the live function.
Success rows compare Decimal values, not formatted strings.
Error rows compare exception type and message text.
# proposal: test_quote_line_char.py
import json
from decimal import Decimal
from pathlib import Path
import pytest
from messy import quote_line
ROWS = Path(__file__).with_name("quote_line.frozen.jsonl")
def load_rows():
lines = ROWS.read_text(encoding="utf-8").splitlines()
return [json.loads(line) for line in lines]
@pytest.mark.parametrize("row", load_rows())
def test_quote_line_matches_frozen_row(row):
if row["ok"]:
got = quote_line(row["qty"], row["unit_cents"], row["coupon"])
assert got == Decimal(row["result"])
return
with pytest.raises(Exception) as ei:
quote_line(row["qty"], row["unit_cents"], row["coupon"])
assert type(ei.value).__name__ == row["exc_type"]
assert str(ei.value) == row["exc_msg"]
Run the new tests before you edit the body.
They must pass against the current messy implementation.
python -m pytest tests/test_quote_line_char.py -q
6. Allow one body change
Now the model may touch quote_line and nothing else.
Reject patches that edit callers, types, or module layout.
Re-run the characterization file after the patch lands.
# proposal: inspect the patch scope
git diff --name-only
git diff -- messy.py
python -m pytest tests/test_quote_line_char.py tests/test_checkout.py -q
A passing oracle plus unchanged callers is the release gate.
Do not merge a multi-file cleanup in the same change.
Choose the capture suite
Use the smallest test module that already reaches the function.
A full integration run is valid if that is your only path.
Do not write new callers only to feed the probe.
Seed data must stay deterministic across capture and replay.
Pin fixture files, clock stubs, and locale settings first.
Otherwise JSONL rows will flake and block the refactor.
When two rows collide
Two rows with equal inputs must produce equal outcomes.
A mismatch means hidden state, not a JSON formatting issue.
Stop the refactor until you name that hidden state.
Drop rows that embed timestamps after you normalize them.
Keep rows that encode business rules, including errors.
Decision table for smallest safe change
Use the table as a review checklist, not a slogan.
One violated row means the change is too large.
Reviewers should reject scope before they debate style.
| Edit | After JSONL freeze | Merge in this PR |
|---|---|---|
Rewrite quote_line body |
Allowed | Yes, if tests pass |
| Extract a private helper in-file | Allowed | Yes, if tests pass |
| Change argument names or types | Forbidden | No |
| Catch a new exception class | Forbidden | No |
| Edit call sites | Forbidden | No |
| Reformat unrelated files | Forbidden | No |
| Add logging with new output | Forbidden | No |
Diff budget
The pull request should list one production file, plus tests.
Characterization files count as tests, not as product code.
A second production file means the change is too wide.
If a model rewrites a number
Models often rewrite rounding that your capture already recorded.
Treat every numeric drift as a failed characterization row.
Restore the old result unless a human files a spec change.
Spec changes need new rows, not silent JSONL edits.
Add a second test file for the new intended behavior.
Keep the old oracle until callers migrate on purpose.
Where a free coding model fits
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is open source, with free model access.
It also provides a free server option for isolated runs.
Paste the frozen tests and the one function into the prompt.
Ask for a body-only rewrite that keeps every row green.
Discard any patch that expands beyond messy.py.
The server is useful when local Python versions conflict.
It is not a substitute for the frozen JSONL oracle.
Reviewer commands
Reviewers should run three commands before they approve.
Check git names, then run both pytest selections.
Approve only when both selections pass and names stay tight.
git diff --name-only HEAD
python -m pytest tests/test_quote_line_char.py -q
python -m pytest tests/test_checkout.py -q
Limitations
JSONL probes fail on unserializable arguments and returns.
Sockets, ORM instances, and open files need explicit adapters.
Without adapters, the log is incomplete and misleading.
This method does not prove full functional correctness.
It only locks observed behavior from one capture run.
Missing branches stay missing if the suite never hit them.
Do not use this approach for cryptographic code changes.
Do not use it when you must change a public signature.
Do not use it if you cannot run the original suite.
Concurrent functions may log interleaved and racy JSONL rows.
Add a mutex around the write, or capture in one thread.
Timing-sensitive code needs a different oracle entirely.
Checklist
- Select one target function and one capture command.
- Record the JSONL file, then freeze a deduplicated copy.
- Commit the characterization tests that replay frozen rows.
- Change only the function body, then re-run tests.
- Reject any diff that touches callers or types.
The core result is a small, reversible body edit.
The oracle stays in git after the model session ends.
Start the next function only after this gate passes.
Top comments (0)