Characterization Tests Before the First Messy-Repo Edit

Do not edit a messy module on sight. Freeze current outputs with characterization tests first. Then apply the smallest behavior-preserving change.

Naive cleanup rewrites names and control flow together. Overlapping branches then change winners without warning. A fixture matrix records those winners before any extract.

This walkthrough uses a constructed ticket router. It is not live production telemetry. Copy the harness into a throwaway branch. Swap the keys for your own module.

Why messy edits fail

Legacy routers mix disk I/O with string rules. Two predicates can match a single ticket. Authors recall the intended winning branch. The interpreter keeps positional order instead.

A rename can look pure during review. Winner order still moves under the tests. Characterization tests pin winners, not hoped design.

The term comes from legacy-code practice. You record what the system does today. You do not claim that behavior is correct.

What the tests must pin

Pin inputs, filesystem presence, and exact outputs. Skip internal call graphs on day one. Skip assertions about correct product policy.

This router needs three pinned fields. The subject string is field one. The priority token is field two. Override-file presence is field three.

The output is one queue name. Empty strings and mixed case still count. Colliding VIP and urgent rows matter most.

Hidden time and network reads break pins. Inventory those seams before you freeze rows. Stub them or the hash will drift.

The messy module

The listing is a teaching fixture only. Treat every line as untrusted legacy code. Do not tidy it before the matrix is green.

# ticket_router.py — constructed example, not production
from __future__ import annotations

import json
import os
import re
from typing import Optional

OVERRIDES_PATH = "overrides.json"


def route(subject: Optional[str], priority: str = "normal") -> str:
    subject = subject or ""
    if os.path.exists(OVERRIDES_PATH):
        with open(OVERRIDES_PATH, encoding="utf-8") as handle:
            data = json.load(handle)
        if subject in data:
            return str(data[subject])
    lowered = subject.lower().strip()
    if "refund" in lowered or "chargeback" in lowered:
        return "billing"
    if re.search(r"\b2fa\b|otp|login", lowered):
        return "identity"
    if priority == "urgent":
        return "oncall"
    if lowered.startswith("[vip]"):
        return "concierge"
    return "general"

Urgent tickets beat the VIP prefix here. An override file beats every keyword. A missing file is a separate path. Those three facts are the refactor risk.

Keyword search uses a lowered copy. Override lookup uses the raw subject. That split is easy to destroy during cleanup.

Numbered workflow

1. Inventory the observable surface

List every external input the function reads. Include files, defaults, and argument fallbacks. Put them in a table, not prose.

Input	Source	Default
subject	positional argument	empty string
priority	keyword argument	"normal"
overrides.json	process working directory	file absent

Stop if you find network or clock reads. Stub those seams before you freeze rows. Record the stub policy beside the table.

2. Build a collision-heavy fixture matrix

Enumerate collisions instead of happy paths. Include empty subject and mixed case. Include VIP text plus urgent priority.

Keep each row as JSON, not a spreadsheet. JSON diffs in git without extra tools. One row must document override-file presence.

Add a row for None subject as well. The function coerces None to empty string. That coercion is behavior, not a style nit.

3. Record outputs, then hash the document

Run the current function against every row. Write the observed queue into the row. Hash the finished document with SHA-256.

Commit that file before any production edit. The hash is a tripwire, not a metric. Later diffs must explain each hash change.

Do not hand-edit observed queues to look nicer. Pretty queues hide the surprising VIP rule. Surprising rules are why you pin first.

4. Extract one tiny pure helper

Choose a change that cannot reorder branches. Subject stripping is a reasonable first extract. Leave file I/O inside route for now.

Re-run the matrix after the extract. The hash must match the committed snapshot. If it drifts, revert and shrink the edit.

Name the helper after the transform only. Do not name it after a business queue. Business names invite extra policy in the helper.

5. Only then consider a second extract

Do not chain extracts in one commit. One helper per commit keeps bisect cheap. Repeat inventory if a new file read appears.

If the second extract needs new stubs, stop. Update the matrix in its own commit. Then extract in the following commit.

Artifact: the characterization harness

The script below is a runnable example. Execute it from a clean working directory. It writes route_matrix.json and prints a hash.

# characterize_router.py — constructed harness
from __future__ import annotations

import hashlib
import json
from pathlib import Path

import ticket_router

FIXTURES = [
    {"subject": None, "priority": "normal", "overrides": False},
    {"subject": "", "priority": "normal", "overrides": False},
    {"subject": "Refund please", "priority": "normal", "overrides": False},
    {"subject": "CHARGEBACK on order", "priority": "urgent", "overrides": False},
    {"subject": "Need 2FA reset", "priority": "normal", "overrides": False},
    {"subject": "otp code failed", "priority": "urgent", "overrides": False},
    {"subject": "[VIP] late shipment", "priority": "normal", "overrides": False},
    {"subject": "[VIP] late shipment", "priority": "urgent", "overrides": False},
    {"subject": "hello", "priority": "normal", "overrides": False},
    {"subject": "hello", "priority": "normal", "overrides": True},
    {"subject": "Refund please", "priority": "normal", "overrides": True},
]

OVERRIDE_PAYLOAD = {"hello": "concierge", "Refund please": "oncall"}


def apply_overrides(enabled: bool) -> None:
    path = Path(ticket_router.OVERRIDES_PATH)
    if enabled:
        path.write_text(json.dumps(OVERRIDE_PAYLOAD), encoding="utf-8")
        return
    if path.exists():
        path.unlink()


def run_matrix() -> list[dict]:
    rows = []
    for fixture in FIXTURES:
        apply_overrides(bool(fixture["overrides"]))
        queue = ticket_router.route(fixture["subject"], fixture["priority"])
        rows.append({**fixture, "queue": queue})
    apply_overrides(False)
    return rows


def main() -> None:
    rows = run_matrix()
    blob = json.dumps(rows, indent=2, sort_keys=True) + "\n"
    Path("route_matrix.json").write_text(blob, encoding="utf-8")
    digest = hashlib.sha256(blob.encode("utf-8")).hexdigest()
    print(digest)
    print(blob)


if __name__ == "__main__":
    main()

Run it once against the untouched module.

python characterize_router.py
git add route_matrix.json characterize_router.py ticket_router.py
git commit -m "Freeze ticket router characterization matrix"

Paste the printed digest into your notes. Do not copy a hash from this article. This article never executed the script on your disk.

A pytest wrapper can lock the file bytes. Compare the regenerated blob to git contents. Avoid asserting Python dict identity across versions.

# test_router_characterization.py — constructed example
from pathlib import Path
import json
import characterize_router


def test_matrix_matches_committed_snapshot():
    rows = characterize_router.run_matrix()
    blob = json.dumps(rows, indent=2, sort_keys=True) + "\n"
    committed = Path("route_matrix.json").read_text(encoding="utf-8")
    assert blob == committed

Decision table for the first extract

Candidate change	Reorders branches?	Allowed in commit one?
Extract strip/lower helper	No, if call sites stay ordered	Yes
Merge VIP and urgent rules	Yes	No
Inline the override file	Yes, missing-file path changes	No
Rename returned queue strings	Yes, snapshot bytes will fail	No

Only the first row is in scope. Everything else waits for a later matrix. Write that limit in the commit message.

The smallest safe change

Extract subject normalization only. Keep regex and file I/O in route. This listing is a proposal, not a measured refactor.

def normalize_subject(subject: Optional[str]) -> str:
    return (subject or "").lower().strip()


def route(subject: Optional[str], priority: str = "normal") -> str:
    raw = subject or ""
    if os.path.exists(OVERRIDES_PATH):
        with open(OVERRIDES_PATH, encoding="utf-8") as handle:
            data = json.load(handle)
        if raw in data:
            return str(data[raw])
    lowered = normalize_subject(raw)
    if "refund" in lowered or "chargeback" in lowered:
        return "billing"
    if re.search(r"\b2fa\b|otp|login", lowered):
        return "identity"
    if priority == "urgent":
        return "oncall"
    if lowered.startswith("[vip]"):
        return "concierge"
    return "general"

Override lookup still uses the raw subject. That preserves exact-key file behavior. The matrix should stay byte-identical after this extract.

If you lower the lookup key as well, stop. That is a second behavior change. Split it after the first hash stays stable.

Drafting extra rows without trusting intent

Drafting extra collision rows is tedious work. A coding model can suggest missing collisions. You still run every suggestion against current code.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. Those two facts are the only product claims here. Do not treat a model suggestion as intended policy.

A useful prompt stays narrow and local. Paste the inventory table, not the wish list. Ask for colliding subjects and priority pairs only.

Reject rows that describe how routing should work. Characterization records current winners, including ugly ones. Intent tests come after the module is understood.

Use free model access for collision ideas only. Keep harness execution on your own machine. The snapshot remains the source of truth.

Limitations

Characterization tests freeze bugs as well as features. That is the point, and the cost. A green matrix does not mean correct product policy.

The harness stubs only one file path. Hidden reads of cwd still leak. Parallel pytest workers can clobber overrides.json.

SHA-256 of JSON depends on separators. Always dump with sort_keys=True. Always append a trailing newline for stability.

Models will invent correct queues for VIP tickets. Current code sends VIP-plus-urgent to oncall. Your tests must keep that surprising winner.

This method does not replace contract tests. After behavior becomes intentional, switch assertions. Until then, pin the bytes you observed.

Unicode subjects can change after runtime upgrades. Pin the Python minor version in the commit. Re-freeze if your interpreter changes.

Who should not use this approach

Do not use this on a greenfield API. You can write intent tests from scratch there. Characterization is for inherited, under-specified modules.

Do not freeze security-sensitive parsers this way. Current behavior may be the vulnerability. Pinning it would encode the hole.

Do not run the harness against live override files. Point the path at a temporary directory. Never commit customer subjects from real tickets.

Skip the extract if the matrix is still drifting. Stabilize I/O first, then extract. One moving file path invalidates every hash.

Teams without git history should not start here. The hash only helps if you can revert. Save the matrix before the helper lands.

Closing rule

The core rule stays small and strict. Freeze outputs, then extract one pure helper. Re-hash the matrix after that extract.

Keep route_matrix.json in the extract commit. If the hash changes, shrink the extract. Do not explain a drift with a rename story.