Bug-for-Bug: Stamp Every Frozen Behavior With a Quirk ID Before You Refactor

Characterization tests freeze bugs, not intent. That is fine while you refactor. It becomes a liability one sprint later.

The next engineer reads assert total_cents("1.005") == 100 and cannot tell whether that is a business rule or a known defect. So the assertion survives forever.

Fix it cheaply: give every frozen behavior a quirk ID, a verdict, and an owner. Then gate CI on that ledger.

1. The failure mode is ambiguity, not the test

A characterization suite answers one question. Does the code still do what it did yesterday? That is exactly what you want before moving a parser or splitting a module.

The suite does not answer the second question. Should it? Every assertion is silent on that point.

So the smallest safe change becomes a judgment call. People choose differently under deadline. Behavior drifts.

The ledger fixes this by separating two decisions. Freeze the behavior now. Decide its fate explicitly, on the record.

2. The quirk ledger has three fields

One JSON file in the repo, one row per frozen oddity:

• id — stable label, e.g. Q-014.
• verdict — preserve, fix-later, or fix-now.
• owner — a human, not a team alias.

A fix-now quirk never ships in the same diff as the refactor. It gets its own PR with its own tests and its own review.

That rule alone prevents most silent behavior changes. The refactor PR stays mechanical and reviewable.

3. Step 1: capture the baseline before touching anything

Install pytest, then run the module through a snapshot test. Pin the Python version in CI so output formatting does not move under you.

python -m pip install pytest
python -m pytest -q tests/characterization --snapshot-update

Keep these tests boring. Feed fixed inputs, assert exact outputs. No mocking frameworks unless the call order matters.

If a test needs the clock, inject a fixed datetime. Never call datetime.now() inside a characterization test. You want deterministic output, not realistic timing.

4. Step 2: tag each frozen test and emit a machine-readable report

A pytest marker carries the quirk ID. A conftest hook collects the results into JSON. This is the whole harness.

# conftest.py
import json
from pathlib import Path
import pytest


def pytest_addoption(parser):
    parser.addoption("--quirk-report", default="reports/quirks-run.json")


def pytest_configure(config):
    config.addinivalue_line(
        "markers",
        "quirk(qid, verdict): frozen behavior tied to docs/quirks.json",
    )


@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.when != "call":
        return
    marker = item.get_closest_marker("quirk")
    if marker is None:
        return
    item.session._quirk_rows.append(
        {
            "id": marker.kwargs["qid"],
            "verdict": marker.kwargs["verdict"],
            "test": item.nodeid,
            "passed": report.passed,
        }
    )


def pytest_sessionstart(session):
    session._quirk_rows = []


def pytest_sessionfinish(session, exitstatus):
    path = Path(session.config.getoption("--quirk-report"))
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(session._quirk_rows, indent=2))

Now the test itself documents the defect. The comment is optional. The marker is not.

# tests/characterization/test_invoice_rounding.py
import pytest
from billing import total_cents


@pytest.mark.quirk(qid="Q-014", verdict="preserve")
def test_half_cent_rounds_down():
    # Observed in the 2026-09 billing export. Kept bug-for-bug.
    assert total_cents("1.005") == 100

Run it and look at the report before writing any production code.

python -m pytest -q --quirk-report reports/quirks-run.json

5. Step 3: fail CI when the ledger and the suite disagree

Three conditions break the build. A tagged test with no ledger row. A ledger row with no test. A row with no owner.

# tools/check_quirks.py
import json
import sys
from pathlib import Path

ledger = {q["id"]: q for q in json.loads(Path("docs/quirks.json").read_text())}
seen = json.loads(Path("reports/quirks-run.json").read_text())
seen_ids = {q["id"] for q in seen}

unknown = sorted(seen_ids - ledger.keys())
orphans = sorted(ledger.keys() - seen_ids)
no_owner = sorted(qid for qid, q in ledger.items() if not q.get("owner"))

deferred = sum(1 for q in ledger.values() if q.get("verdict") == "fix-later")

print(f"unknown={unknown} orphans={orphans} no_owner={no_owner} deferred={deferred}")
sys.exit(1 if (unknown or orphans or no_owner) else 0)

Add a ratchet on the deferred count. Read the previous number from a committed file and fail when it grows. Frozen debt becomes visible instead of ambient.

6. Step 4: make the smallest safe change

The gate is not a feeling about diff size. It is four checks on one commit.

python -m pytest -q tests/characterization
python tools/check_quirks.py
git diff --stat main...HEAD -- 'src/*.py'
git diff main...HEAD -- 'src/*.py' | grep -c '^[-+]' || true

Accept the change when all four hold. Every quirk test still passes. The ledger is consistent. Only the target module moved. The touched-line count stayed inside the number you agreed before starting.

If a quirk test breaks, stop. Either the refactor is wrong, or the quirk needs a fix-now PR first. Do not edit the assertion to make the diff green.

7. Where a free model and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator reports two things I used in this workflow: free model access, and a free server option. Treat both as supplied availability claims, not guarantees.

Free model access helps at step 1, on the module nobody wants to read. Ask for candidate assertions over a pasted function, then keep the ones you can trace to real output. Most drafts over-specify internals. You throw those away.

The free server helps at step 3, where the baseline suite runs as a batch. Set a fixed clock, run serially, and store reports/quirks-run.json as an artifact. That is a scheduling problem, not a compute problem.

Do not use it for timing-sensitive benchmarks. Shared machines make latency numbers meaningless.

8. Verdicts, in plain terms

Situation	Verdict	Ships with the refactor?
Behavior is contractual, even if ugly	preserve	Yes, tests unchanged
Behavior is a defect with a real cost	fix-later	Yes, ledger row only
Behavior is a defect you must fix now	fix-now	No, separate PR first
Behavior is unreachable dead code	delete the test	Yes, with the code

9. Limitations and who should skip this

This adds two files and a CI step. On a five-file repo it is overhead. Skip it.

It also assumes output is observable and deterministic. If your module talks to a live third party, record responses first; otherwise the ledger will churn.

Teams that cannot name an owner per quirk should not adopt it. The ledger degrades into a second, worse backlog.

Finally, a quirk ID is not documentation of why. Link an incident or a ticket when one exists. Do not invent a rationale you cannot source.

Start with one module and one ledger file. Refactor the next thing tomorrow with the gate already in place.