Characterize a Messy Module Before One Safe Change

Characterization tests beat a clever refactor every time. Pin observable output before you touch the module.

A messy file hides sort order and truncation rules. One extract without a baseline will shift those rules.

The failure mode

Developers often rewrite a helper that looks messy. The rewrite drops an implicit sort or default name.

Callers then fail on empty jobs or long errors. The refactor looked clean and still shipped a bug.

Core rule

Write tests first, then change one pure seam. Leave mixed formatting and defaults alone until later.

This article uses a small jobs summary module. The same sequence applies to any tangled helper.

Artifact: a messy jobs summary

The module below is intentionally compact and tangled. It formats job dicts into a stable text report.

Do not clean it before the tests exist. Copy it into jobs_summary.py as the subject.

# jobs_summary.py
import time


def summarize(jobs, now=None):
    if now is None:
        now = time.time()
    lines = []
    failed = 0
    for job in jobs:
        name = job.get("name") or "unnamed"
        status = job.get("status", "unknown")
        started = job.get("started_at")
        err = job.get("error") or ""
        if len(err) > 40:
            err = err[:37] + "..."
        dur = 0
        if started is not None:
            dur = int(now - started)
        if status == "failed":
            failed += 1
        lines.append((status, name, dur, err))
    order = {"failed": 0, "running": 1, "ok": 2, "unknown": 3}
    lines.sort(key=lambda row: (order.get(row[0], 9), row[1]))
    header = f"jobs={len(jobs)} failed={failed}"
    body = []
    for status, name, dur, err in lines:
        extra = f" err={err}" if err else ""
        body.append(f"{status} {name} {dur}s{extra}")
    if not body:
        return header
    return header + "\n" + "\n".join(body)

What the tests must pin

Characterization tests record behavior, including ugly parts. They do not judge whether the behavior is ideal.

Inject now so duration math stays deterministic. Never call time.time() inside a characterization test.

Decision table

Input shape	Locked observation
[]	header only, jobs=0 failed=0
missing name	label is unnamed
name is ""	also unnamed, because of or
missing status	status is unknown
error longer than 40	slice 37 chars, then ...
empty error	no err= suffix on the line
mixed statuses	failed, then running, then ok
unknown status label	sort key falls through to 9
started_at present	duration is int(now - started_at)
started_at missing	duration is 0s
status == "failed"	counted in the header only

Each row is one assertion, not a redesign prompt. If production depends on a quirk, the test keeps it.

Workflow

1. Freeze the public function only

Test summarize only, not imagined private helpers. Private extracts come after the baseline is green.

Name the file test_jobs_summary.py. Keep it next to the messy module.

2. Cover the edges in the table

Empty input is a first-class characterization case. Missing keys and long errors belong in the same file.

Add the empty-string name case before any extract. That or chain is a frequent silent change.

3. Run the suite until it is boring

Green tests are the lock, not a style win. Re-run after every edit, including comment-only edits.

4. Make the smallest safe change

Extract one function that the tests already imply. Stop when the suite is green and the diff is tiny.

Characterization tests

The tests below freeze strings, not object graphs. Exact text is the contract for this helper.

# test_jobs_summary.py
import unittest
from jobs_summary import summarize

NOW = 1_700_000_040


class SummarizeCharacterization(unittest.TestCase):
    def test_empty_jobs_header_only(self):
        self.assertEqual(summarize([], now=NOW), "jobs=0 failed=0")

    def test_missing_name_becomes_unnamed(self):
        text = summarize([{"status": "ok"}], now=NOW)
        self.assertIn("ok unnamed 0s", text)

    def test_empty_name_also_unnamed(self):
        text = summarize([{"name": "", "status": "ok"}], now=NOW)
        self.assertIn("ok unnamed 0s", text)

    def test_missing_status_is_unknown(self):
        text = summarize([{"name": "build"}], now=NOW)
        self.assertIn("unknown build 0s", text)

    def test_error_truncated_at_forty(self):
        err = "x" * 41
        text = summarize(
            [{"name": "sync", "status": "failed", "error": err}],
            now=NOW,
        )
        self.assertIn("err=" + ("x" * 37) + "...", text)
        self.assertNotIn("x" * 41, text)

    def test_empty_error_omits_suffix(self):
        text = summarize(
            [{"name": "sync", "status": "ok", "error": ""}],
            now=NOW,
        )
        self.assertEqual(text, "jobs=1 failed=0\nok sync 0s")

    def test_sort_failed_running_ok_then_name(self):
        jobs = [
            {"name": "b", "status": "ok"},
            {"name": "a", "status": "running"},
            {"name": "c", "status": "failed"},
            {"name": "a", "status": "failed"},
        ]
        text = summarize(jobs, now=NOW)
        lines = text.splitlines()[1:]
        self.assertEqual(
            lines,
            ["failed a 0s", "failed c 0s", "running a 0s", "ok b 0s"],
        )

    def test_unknown_status_sorts_after_known(self):
        jobs = [
            {"name": "z", "status": "weird"},
            {"name": "a", "status": "ok"},
        ]
        lines = summarize(jobs, now=NOW).splitlines()[1:]
        self.assertEqual(lines, ["ok a 0s", "weird z 0s"])

    def test_duration_uses_injected_now(self):
        jobs = [{"name": "etl", "status": "ok", "started_at": NOW - 9}]
        text = summarize(jobs, now=NOW)
        self.assertEqual(text, "jobs=1 failed=0\nok etl 9s")

    def test_failed_count_ignores_other_statuses(self):
        jobs = [
            {"name": "a", "status": "failed"},
            {"name": "b", "status": "ok"},
            {"name": "c", "status": "failed"},
        ]
        header = summarize(jobs, now=NOW).splitlines()[0]
        self.assertEqual(header, "jobs=3 failed=2")


if __name__ == "__main__":
    unittest.main()

Commands

Run the suite from the repo root with unittest.

python -m unittest test_jobs_summary.py -v

A useful check is a deliberate mutation of truncation. Change the slice to 30 characters and rerun tests.

The long-error case must fail if the pin is real. Restore the original slice before the extract commit.

# temporary mutation inside jobs_summary.py, then:
python -m unittest test_jobs_summary.py -v
# expect test_error_truncated_at_forty to fail, then revert

Record the failure name in the review notes. A pin that never fails is not a pin.

Smallest safe change

After the pin, extract truncation and nothing else. Do not retouch sort keys in the same diff.

def _truncate_error(err):
    if len(err) > 40:
        return err[:37] + "..."
    return err

Call _truncate_error(err) from the existing loop. Keep summarize as the only public entry point.

Do not rename keys or change sort order yet. Do not replace or with a None check yet.

Re-run unittest. The output strings must match exactly. If a line drifts, revert and shrink the extract.

A second later commit can extract duration. That split keeps git blame honest.

git add test_jobs_summary.py jobs_summary.py
git commit -m "test: characterize summarize output"
# then, after the extract:
git add jobs_summary.py
git commit -m "refactor: extract error truncation only"

Two commits make a later revert cheap and obvious. Mixed commits hide which edit broke a caller.

Using a model without skipping the pin

A coding model can suggest extra edge cases quickly. It cannot replace a committed, locally executed baseline.

MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Paste the messy function and the decision table only. Ask for candidate cases, not a rewritten module.

Keep generated tests out of main until they pass locally. Discard any case that asserts a behavior you did not observe.

The free server does not change the review duty. You still own the exact strings in the characterization file.

Reject a generated extract that also “fixes” empty names. That is a behavior change hiding in a refactor.

What this method does not do

It does not fix product bugs you intend to keep. Characterization will freeze a truncation bug if you allow it.

Split that work: pin first, then add a behavior-change test. Do not mix a bugfix and an extract in one diff.

Non-deterministic clocks, networks, and maps will flake. Inject time, and sort any set-driven output explicitly.

Full-file snapshots also drift on header wording. Prefer line-level asserts for the rows you care about.

Who should skip this

Skip this if the module has no callers yet. Greenfield code can use ordinary TDD instead.

Skip this if you need a new report format now. That is a behavior change, not a safe refactor.

Skip this if output includes raw timestamps from the wall clock. Stabilize time first or the snapshots will thrash.

Skip this if legal copy must change with the extract. Frozen strings would block a required wording update.

Failure analysis

The usual break is an unstable sort of equal keys. The sample module sorts by status, then by name.

Two failed jobs named alike will keep input order. Python’s sort is stable; pin that if callers depend on it.

Add a duplicate-name case when the real log shows ties. Do not assume uniqueness from the happy path.

Another break is or versus missing empty strings. job.get("name") or "unnamed" treats empty as unnamed.

Add a case with "name": "" before any extract. Models often “fix” that line without a test.

Integer truncation of duration is another silent edit. int(9.9) is 9; a round() swap will fail the pin.

Checklist

1. Copy the messy function with no cosmetic edits.
2. Inject clocks, paths, and other non-deterministic inputs.
3. Fill a decision table from real caller traces.
4. Lock public output with exact string asserts.
5. Mutate one rule and confirm a named test fails.
6. Restore the mutation, then extract a single helper.
7. Re-run the suite and stop if any line drifts.

Close

Messy repos reward patience more than tasteful helpers. Lock the text, extract one seam, then stop.

Commit the characterization file before the extract commit. If a free model drafts cases, keep the assertions yours.