Messy modules break under wide AI-driven rewrites. Public behavior drifts while the helpers look cleaner.
Freeze the module as a black box first. Then change one private helper, and nothing else.
This article is a protocol, not a memoir. It uses a fixture directory as the oracle. The worked example is labeled as unexecuted sample code.
Why full-file rewrites fail
A dirty module mixes parsing, I/O, and policy. A single rewrite often touches every mixed layer. Reviewers cannot see which bytes were required.
Unit tests written after the rewrite are circular. They encode the new guess, not the old contract. Characterization records what the file already does.
The gate in one line
No helper edit ships without a green module harness. The harness compares returns, stdout, and output files. Anything outside that budget is a new project.
Why three channels beat one
Return values miss print and logging side effects. Stdout text still misses files under output/. Output hashes miss exception types on failure paths.
A cleaner helper that changes error spelling still fails. Callers often parse exception text in brownfield code. Keep the error string in the ledger.
Step 1 — Rank inbound imports
Do not start with the God object. Start with a leaf module that few callers import. Count inbound references with a cheap scan.
# proposed ranking helper — unexecuted sample
from pathlib import Path
import ast, collections, sys
def count_inbound(root: Path) -> list[tuple[int, str]]:
names = {}
inbound = collections.Counter()
files = list(root.rglob("*.py"))
for path in files:
if path.name == "__init__.py":
continue
names[path.stem] = str(path)
for path in files:
try:
tree = ast.parse(path.read_text(encoding="utf-8"))
except SyntaxError:
continue
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node.module:
head = node.module.split(".")[-1]
if head in names and path.stem != head:
inbound[head] += 1
if isinstance(node, ast.Import):
for alias in node.names:
head = alias.name.split(".")[-1]
if head in names and path.stem != head:
inbound[head] += 1
ranked = sorted((inbound[m], names[m]) for m in names)
return ranked[:15]
if __name__ == "__main__":
root = Path(sys.argv[1])
for n, p in count_inbound(root):
print(f"{n:4d} {p}")
Low inbound count means a smaller blast radius. High inbound count means you freeze that file later. Rank the files first, then edit second.
Dynamic imports will under-count some leaf modules. Treat the ranking as a filter, not a proof. Confirm the chosen file with a grep pass.
Step 2 — List public entry points
Treat underscore-prefixed names as private helper functions. Do not characterize those private helpers just yet. Private code is the later edit surface.
# proposed public lister — unexecuted sample
import ast
from pathlib import Path
def public_functions(path: Path) -> list[str]:
tree = ast.parse(path.read_text(encoding="utf-8"))
out = []
for node in tree.body:
if isinstance(node, ast.FunctionDef) and not node.name.startswith("_"):
out.append(node.name)
return out
A messy file with six public functions is still tractable. A file with thirty public functions is not this protocol. Split that file into two characterization jobs.
Methods on a large class need the same rule. Characterize the public methods on that class. Leave _ methods for the later edit.
Step 3 — Build a fixture workspace
Copy a small input directory for every public call. Keep those fixtures tiny, local, and fully deterministic. Ban network calls, clocks, and home-directory writes.
fixtures/
case_invoice_v1/
input/
invoice.txt
env.json
case_invoice_empty/
input/
invoice.txt
env.json
Each case passes kwargs through env.json. All paths inside those kwargs stay workspace-relative. Do not pass absolute home-directory paths here.
{
"kwargs": {
"path": "input/invoice.txt",
"out_dir": "output"
}
}
env.json holds argv and environment keys only. It must not hold any production secrets. Commit the fixtures, but never commit production data.
Two tight cases beat twenty vague ones. Cover one happy path and one error path. Add more cases only after the first helper lands.
Step 4 — Record the black-box ledger
The ledger stores return JSON, stdout text, and output hashes. That triple is the locked module contract. Skip internal call traces in this protocol.
# proposed harness — unexecuted sample
from __future__ import annotations
import hashlib, io, json, os, shutil, traceback
from contextlib import redirect_stdout
from pathlib import Path
import importlib.util
def load_module(path: Path):
spec = importlib.util.spec_from_file_location(path.stem, path)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod
def hash_tree(root: Path) -> dict[str, str]:
out = {}
if not root.exists():
return out
for p in sorted(root.rglob("*")):
if p.is_file():
rel = str(p.relative_to(root)).replace("\\", "/")
out[rel] = hashlib.sha256(p.read_bytes()).hexdigest()
return out
def run_case(mod, fn_name: str, case_dir: Path, work: Path) -> dict:
shutil.copytree(case_dir / "input", work / "input")
env = json.loads((case_dir / "env.json").read_text())
fn = getattr(mod, fn_name)
buf = io.StringIO()
old_cwd = os.getcwd()
record = {"fn": fn_name, "case": case_dir.name}
try:
os.chdir(work)
with redirect_stdout(buf):
result = fn(**env.get("kwargs", {}))
record["ok"] = True
record["result"] = result
record["error"] = None
except Exception as exc:
record["ok"] = False
record["result"] = None
record["error"] = f"{type(exc).__name__}: {exc}"
record["trace"] = traceback.format_exc()
finally:
os.chdir(old_cwd)
record["stdout"] = buf.getvalue()
record["files"] = hash_tree(work / "output")
return record
def write_golden(path: Path, record: dict) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(record, indent=2, sort_keys=True, default=str))
Run the recorder once against current code. Store the goldens next to those fixtures. After that first run, goldens stay read-only.
python harness.py record messy_invoice.py fixtures goldens
Label this command as a proposed interface. Wire the record and check commands yourself.
Sample module under test
The sample below is a messy invoice helper file. It mixes parsing, printing, and disk writes. Do not rewrite the whole messy file.
# messy_invoice.py — unexecuted sample, not production code
from pathlib import Path
def parse_invoice(path: str) -> dict:
text = Path(path).read_text(encoding="utf-8")
return _parse_body(text)
def write_summary(data: dict, out_dir: str) -> str:
vendor = _normalize_vendor(data.get("vendor", ""))
total = data.get("total", "0")
line = f"{vendor}:{total}\n"
print(f"summary {vendor}")
dest = Path(out_dir) / "summary.txt"
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(line, encoding="utf-8")
return str(dest)
def parse_invoice_or_raise(path: str) -> dict:
data = parse_invoice(path)
if "total" not in data:
raise ValueError("missing total")
return data
def _parse_body(text: str) -> dict:
out = {}
for raw in text.splitlines():
if ":" not in raw:
continue
key, value = raw.split(":", 1)
out[key.strip().lower()] = value.strip()
return out
def _normalize_vendor(name: str) -> str:
return " ".join(name.strip().title().split())
Public functions are parse_invoice, write_summary, and parse_invoice_or_raise. The later edit target is _normalize_vendor only. Goldens must cover both parse and write.
Sample golden record
Keep golden files boring, sorted, and byte stable. Stringify non-JSON values with a default encoder.
{
"case": "case_invoice_v1",
"error": null,
"files": {
"summary.txt": "<sha256>"
},
"fn": "write_summary",
"ok": true,
"result": "output/summary.txt",
"stdout": "summary Acme Corp\n"
}
The hash in this block is a placeholder. Compute the real hashes on your own machine. Never paste production invoice text into goldens.
Step 5 — Check before every edit
The check command diffs new records against goldens. Any mismatch is a failed refactor gate. Do not update goldens to silence a helper rewrite.
# proposed checker fragment — unexecuted sample
def check_record(golden: dict, actual: dict) -> list[str]:
misses = []
for key in ("ok", "result", "error", "stdout", "files"):
if golden.get(key) != actual.get(key):
misses.append(key)
return misses
Print the failing ledger keys, and nothing else. Do not dump entire trees in CI logs. Keep the failure signal small and local.
Step 6 — Change one private helper
Open the messy file after the harness is green. Then pick exactly one underscore-prefixed helper function. Keep every public signature frozen during the edit.
Allowed edits in this loop:
- Rename local variables inside that one helper.
- Extract a pure slice of that helper.
- Fix one branch that goldens already cover.
Forbidden edits in this loop:
- Do not add any new public functions.
- Do not change exception types on public paths.
- Do not write extra files under
output/. - Do not regenerate goldens to match a rewrite.
If the helper needs new behavior, add a new case first. Record that new case on the old code. Then edit the helper against that case.
Failure analysis of a clean-looking patch
Suppose a model rewrites _normalize_vendor during cleanup. It also changes the missing-total error text. The harness fails on error and maybe stdout.
That mismatch is the protocol working as designed. Cleanup that alters caller-visible strings is a behavior change. Revert the patch and shrink the diff.
A second failure mode is extra debug logging. A debug print breaks the stdout channel. Strip logging or route it to stderr outside the ledger.
Where a free coding model fits
A model is useful after the ledger exists. It is not useful as the first reader of a dirty file. Prompt it with the helper body and the failing keys only.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option. Use those only against the helper, not the goldens. Keep the harness on your machine as the judge.
A proposed prompt shape follows for the helper.
Public signatures must stay byte-stable.
Goldens already lock result, stdout, and output hashes.
Rewrite only `_normalize_vendor(name: str) -> str`.
Do not add files or logging.
Return a unified diff limited to that function.
Reject any diff that touches harness.py or goldens/. Re-run the check command after applying the patch.
If the check fails, revert the patch. Do not negotiate with the locked goldens.
Commit the harness and goldens in one change. Commit the helper patch in a second change. Reviewers can then ignore the oracle and judge the helper.
Decision table
| Situation | Action |
|---|---|
| Inbound count is 0–2 | Characterize now |
| Inbound count is high | Defer the file |
| Public functions exceed 12 | Split the job |
| Helper has no golden path | Add a fixture first |
| Diff spans two helpers | Split into two patches |
| Goldens need a refresh | Stop. Behavior changed. |
The table is a filter, not a score. Skip cells that do not match your repo.
CI one-liner
Run the check in the same job as unit tests. Fail the job on any ledger mismatch. Do not auto-refresh goldens in CI.
python harness.py check messy_invoice.py fixtures goldens
That command is a proposed interface only. Keep it synchronous and local. Remote flakiness means the fixtures leaked a clock.
What this protocol does not claim
This protocol does not prove full functional correctness. It proves the public surface did not move. Hidden bugs stay hidden if fixtures miss them.
It does not replace type checkers or linters. Those tools catch shape errors the ledger never sees. Run them after the harness, not instead of it.
It does not handle threads, sockets, or GUI state. Those paths need different oracles, not this ledger. Do not stretch this fixture design onto them.
Who should not use this
Skip this if the module is already a pure function with tests. Skip this if legal rules forbid copying production-like fixtures. Skip this if every public call hits a paid API.
Skip this if you cannot pin the runtime. Python minor versions can change exception text strings. Pin the interpreter before you record goldens.
A proposed 25-minute loop
This timebox is a planning aid, not a measured result.
- Rank inbound imports for about ten minutes.
- Record two small fixtures for ten minutes.
- Edit one private helper for five minutes.
- Re-run check, then revert on any mismatch.
Stop the loop at exactly one helper. A second helper is a second loop. Wide rewrites are how messy files stay messy.
Limits of the sample code
The ranking sample ignores dynamic import calls. The harness assumes JSON-friendly public return values. File hashing skips empty directories on purpose.
Clock-dependent helpers will flake under this ledger. Inject a clock through kwargs in env.json. If you cannot inject, do not characterize that path here.
Bytes and datetime objects need an encoder. Add default=str only when the old code already printed them. Do not normalize values the callers never saw.
Close
Record the messy module as a black box. Then change exactly one private helper function.
The recorded ledger is the only gate. The model is optional labor after the gate.
Point a free model at one helper after goldens lock. The local harness still decides the merge.
Top comments (1)
Your approach to handling messy modules by treating them as black boxes until you've incrementally refined their helpers is an elegant strategy for managing complexity. The emphasis on using a fixture directory for characterization is particularly insightful, as it promotes isolation and repeatability during refactoring. Additionally, considering inbound import counts to help prioritize changes is a practical tactic that many teams could benefit from. If this project needs further support in refining the implementation or developing additional tools around this protocol, I’d be glad to discuss a paid collaboration. What are your thoughts on integrating automated tests into this workflow to ensure stability as you refactor?