The Shape Census: Triage a Messy Module Before You Refactor It

Picture a billing module with a 900-line invoice function, four boolean flags threaded through it, and a test suite that has been skipped since a migration nobody remembers. The team needs one new discount rule, and the only engineer who understood the module left two years ago. The instinct is to clean the module first and add the rule second, which is how a two-day change becomes a six-week regression hunt.

There is a cheaper order of operations. Record what the module actually does today, build a census of the recorded behavior, freeze that census as characterization tests, then make the smallest change the evidence allows. Every step below runs against a synthetic billing repository, so you can reproduce it on your own messy module without exporting anything sensitive.

Why "read it and clean it up" fails on untested modules

Reading code tells you what a branch could do, not what it does on the inputs your users actually send. On an untested module those two things diverge fast, and the divergence is exactly where refactors break.

• You cannot tell which branches execute, so deleting "dead" code removes behavior you never observed.
• You cannot tell which output fields are load-bearing, because downstream consumers read dictionaries by key without any written contract.
• A rename is a one-line diff for a reviewer and a breaking change for a regex-based consumer in another repository.
• Refactor pull requests that mix structural moves with behavior tweaks force reviewers to approve both at once, which nobody can honestly do.

A call ledger replaces those guesses with counts, and counts survive code review arguments.

Step 1: Record a call ledger at the module boundary

Pick the seams you intend to touch, meaning the functions whose callers you control today. Wrap each one in a decorator that logs arguments, result, status, and latency to JSON Lines, and keep the recording scoped to a short window.

# ledger.py
import functools, json, pathlib, time, uuid

LEDGER = pathlib.Path("ledger/calls.jsonl")

def record(seam: str):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            started = time.perf_counter()
            status, result = "ok", None
            try:
                result = fn(*args, **kwargs)
            except Exception as exc:
                status = "error"
                result = {"__error__": type(exc).__name__, "message": str(exc)}
            payload = {
                "call_id": uuid.uuid4().hex[:12],
                "seam": seam,
                "status": status,
                "args": args,
                "kwargs": kwargs,
                "result": result,
                "elapsed_ms": round((time.perf_counter() - started) * 1000, 3),
            }
            LEDGER.parent.mkdir(parents=True, exist_ok=True)
            with LEDGER.open("a", encoding="utf-8") as fh:
                print(json.dumps(payload, default=str, sort_keys=True), file=fh)
            if status == "error":
                raise
            return result
        return wrapper
    return deco

Run the test suite and a staging job, then check what you captured. Redact or hash customer identifiers before the file leaves the machine, and never ship a raw ledger to a repository.

python -m pytest tests/ -q
wc -l ledger/calls.jsonl
python tools/triage.py ledger/calls.jsonl > ledger/triage.md

Step 2: Build a shape census instead of reading every output

Thousands of recorded results are unreadable, but their structure is small. Fingerprint each result by types and dictionary keys, ignoring values entirely, then count fingerprints per seam. The figures below come from a synthetic recording of roughly 4,000 calls across three seams, and you should read them as a format illustration rather than a benchmark.

# tools/triage.py
import collections, json, sys

def shape(value, depth=0):
    if depth > 4:
        return "..."
    if isinstance(value, dict):
        inner = ",".join(f"{k}:{shape(v, depth + 1)}" for k, v in sorted(value.items()))
        return "{" + inner + "}"
    if isinstance(value, (list, tuple)):
        return "[" + shape(value[0], depth + 1) + "]" if value else "[]"
    return type(value).__name__

rows = collections.defaultdict(lambda: collections.Counter())
statuses = collections.defaultdict(collections.Counter)
for line in open(sys.argv[1], encoding="utf-8"):
    call = json.loads(line)
    statuses[call["seam"]][call["status"]] += 1
    rows[call["seam"]][shape(call["result"])] += 1

print("| seam | calls | errors | distinct shapes | top shape share |")
print("| --- | --- | --- | --- | --- |")
for seam, counter in sorted(rows.items()):
    total = sum(counter.values())
    top = counter.most_common(1)[0][1] / total
    print(f"| {seam} | {total} | {statuses[seam]['error']} | {len(counter)} | {top:.0%} |")

seam	calls	errors	distinct shapes	top shape share
compute_invoice	2,140	31	3	96%
apply_adjustments	1,502	118	11	61%
format_receipt	388	0	7	43%

The third column is the one that changes your plan. A seam with three shapes is a small, bounded surface you can freeze quickly; a seam with eleven shapes is where the flags hide and where a rename will hurt. Use the census to choose the order of work rather than your intuition.

Calls	Distinct shapes	Top shape share	Verdict
High	1-3	Above 90%	Freeze and extract in the same day
High	4-8	60-90%	Freeze first, extract only one branch per PR
High	9+	Below 60%	Document behavior, do not restructure yet
Low	Any	Any	Collect more traffic before deciding

Step 3: Turn the recording into characterization tests

Characterization tests lock in current behavior, bugs included, so normalization has to remove the fields you genuinely cannot predict. Timestamps, generated identifiers, and trace headers get dropped; numeric money fields get rounded; everything else must match exactly.

# tests/test_characterization.py
import json, pathlib, pytest
from billing.legacy import compute_invoice

CASES = [json.loads(line) for line in pathlib.Path("ledger/calls.jsonl").read_text().splitlines()]
OK = [c for c in CASES if c["seam"] == "compute_invoice" and c["status"] == "ok"]

def normalize(result):
    result = dict(result)
    for volatile in ("invoice_id", "created_at", "trace_id"):
        result.pop(volatile, None)
    if "amounts" in result:
        result["amounts"] = {k: round(v, 2) for k, v in result["amounts"].items()}
    return result

@pytest.mark.parametrize("case", OK, ids=lambda c: c["call_id"])
def test_matches_recorded_behavior(case):
    actual = normalize(compute_invoice(*case["args"], **case["kwargs"]))
    assert actual == normalize(case["result"])

For the noisy seams, assert the shape rather than the values, and keep the census report under version control so drift is visible. A CI step that regenerates ledger/triage.md and fails on a byte difference is usually enough.

python tools/triage.py ledger/calls.jsonl > /tmp/triage.md
diff -u ledger/triage.md /tmp/triage.md

Step 4: Make the smallest change the census allows

Three rules keep the diff reviewable and the risk bounded. Apply them literally, because every shortcut here has a cost you will pay during an incident.

1. One structural move per commit; extraction and renaming never share a pull request.
2. Extract only behind a name that already appears in the census as a distinct shape.
3. If the regenerated census changes shape counts, revert instead of adjusting the tests to pass.

The actual change is usually smaller than the preparation suggests, which is the point.

-        if customer.get("plan") == "legacy" and order["total"] > 500:
-            discount = round(order["total"] * 0.07, 2)
-            apply_discount(order, discount)
+        if qualifies_for_legacy_discount(customer, order):
+            apply_discount(order, legacy_discount_amount(order))

The acceptance conditions are mechanical: the characterization suite is green, the census diff is empty, and the new rule has at least one recorded case of its own before you merge.

Where free model access and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The parts of this workflow that are genuinely tedious are also mechanical: clustering the shape fingerprints into a short list, drafting the parametrize scaffolding from a ledger sample, and writing the triage table you reviewed above. That is the kind of work where a model with tool access saves real time, and the operator describes both free model access and a free server option, with an allowance on the order of ten million tokens, so you can try the loop before committing a budget to it. Verify current terms on the project page, since availability and limits change and this article was not written to track them.

The free server option matters for one specific reason. A replay harness that only runs on a laptop records whenever someone remembers to run it, whereas moving the same container to a hosted instance gives you a stable endpoint and a repeatable recording window. Keep secrets out of the container, keep the ledger window short, and treat the hosted run as a staging job rather than a production dependency.

Limits, and who should skip this approach

• Nondeterministic seams without a clean normalization function produce flaky characterization tests, and flaky tests get skipped.
• Modules whose real behavior lives in database triggers or queues cannot be frozen at a Python function boundary alone.
• Recording inputs may capture personal data, so legal review belongs before the first ledger file, not after.
• Low-traffic seams need weeks to reach useful coverage, which may exceed your deadline.
• Teams that cannot deploy a recording wrapper to staging have no cheap path here, and should write the missing tests by hand instead.

Checklist before you push

1. The ledger window is short, redacted, and excluded from the repository.
2. Every seam you plan to touch has at least one recorded error case.
3. Normalization removes only fields you can defend in review.
4. The census report is committed and the CI diff check passes.
5. The pull request contains one structural move and nothing else.
6. A reviewer who has never seen the module can explain the change after reading the filter to the third commit? they should not need that filter at all.

Start with the seam that has the fewest shapes, not the one that annoys you most, and let the census pick the order of work for you. If you run the harness on your own repo this week, the triage table will tell you more about your legacy module than a week of reading it will.