Three Gates Before the Refactor: Freeze Order, Drafted Tests, Flake Census

Most messy-repo refactors fail at the test layer, not the code layer. The tests pass on your laptop and fail in CI, so you revert and learn nothing.

The fix is ordering. Rank modules first. Draft characterization tests second. Reject unstable tests third. Only then does a smallest safe change exist.

Below is that workflow as three gates, with runnable code for each.

Why the test layer breaks first

A characterization test records what the code does now. It does not record what the code should do. That distinction dies the moment a test asserts a clock, a hash order, or a random value.

Order matters for the same reason. Ranking first stops you freezing a module nobody imports. A census last stops you freezing instability into the suite forever.

Gate 1 — Freeze order: rank before you test

Step 1: measure churn from git history. Step 2: measure import fan-in with the ast module. Step 3: multiply them.

High churn plus high fan-in means the module is both volatile and load-bearing. That is where a wrong refactor costs the most.

#!/usr/bin/env python3
"""freeze_order.py - rank modules for characterization work.

Usage: python freeze_order.py /path/to/repo [days]
"""
import ast
import collections
import pathlib
import subprocess
import sys

repo = pathlib.Path(sys.argv[1]).resolve()
days = sys.argv[2] if len(sys.argv) > 2 else "365"
SKIP = {".venv", "venv", "node_modules", "site-packages", ".git", "build", "dist"}


def dotted(path: pathlib.Path) -> str:
    parts = list(path.relative_to(repo).with_suffix("").parts)
    if parts and parts[-1] == "__init__":
        parts = parts[:-1]
    return ".".join(parts)


files = [p for p in repo.rglob("*.py") if not SKIP & set(p.parts)]

fan_in = collections.Counter()
for path in files:
    try:
        tree = ast.parse(path.read_text(encoding="utf-8", errors="replace"))
    except SyntaxError:
        continue
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for alias in node.names:
                fan_in[alias.name] += 1
        elif isinstance(node, ast.ImportFrom) and node.module:
            fan_in[node.module] += 1
            for alias in node.names:
                fan_in[f"{node.module}.{alias.name}"] += 1

log = subprocess.run(
    ["git", "log", f"--since={days} days ago", "--name-only", "--pretty=format:"],
    cwd=repo, capture_output=True, text=True, check=True,
).stdout
churn = collections.Counter(
    line.strip() for line in log.splitlines() if line.strip().endswith(".py")
)

rows = []
for path in files:
    commits = churn.get(str(path.relative_to(repo)), 0)
    if commits == 0:
        continue
    imports = fan_in.get(dotted(path), 0)
    rows.append((commits * (1 + imports), commits, imports, str(path.relative_to(repo))))

rows.sort(reverse=True)
print(f"{'score':>7} {'commits':>7} {'fan-in':>6}  module")
for score, commits, imports, name in rows[:15]:
    print(f"{score:>7} {commits:>7} {imports:>6}  {name}")

Read the table with three rules. Top rows with no tests get frozen first. Top rows already frozen get skipped this round. Low-churn, high-fan-in rows get an interface seal only, not a rewrite.

Gate 2 — Draft characterization tests, then reject some

Writing fifty characterization tests by hand is slow and boring. A free model can draft them fast, and speed is the only thing you should trust here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use MonkeyCode's free model access as a drafting step, not an approval step. You still own the accept-or-delete decision for every test it produces.

The prompt contract matters more than the model. Ask for tests that call public functions with concrete inputs and assert exact return values. Forbid assertions on log text, wall-clock durations, and mock internals of the module under test. Request plain pytest.

# Prompt template (unexecuted example - adapt before use)
Here is one module from a legacy codebase.
Write pytest tests that pin its CURRENT behaviour.
Rules:
- Call only functions listed as public.
- Use concrete literal inputs. No randomness. No time.time().
- Assert exact return values or raised exception types.
- Do not mock the module under test.
- Do not assert on log messages, ordering of sets, or durations.
- Every test must fail loudly if behaviour changes.
Module:
<source>

Now the rule that saves the refactor. Run the drafted tests against unmodified code. A drafted test that fails immediately is a spec leak. The model wrote what it thinks the code should do, not what it does.

Delete spec leaks. Never edit production code to satisfy a drafted test. That inverts the entire point of characterization.

Gate 3 — The flake census

One green run proves almost nothing. Run the frozen suite twenty times under varied hash seeds and time zones, then classify every test by its outcome set.

#!/usr/bin/env bash
# census.sh - run the frozen suite N times under varied seeds and clocks
set -u
SUITE="${1:-tests/characterization}"
RUNS="${2:-20}"
rm -f run-*.xml log-*.txt
for i in $(seq 1 "$RUNS"); do
  if [ $((i % 2)) -eq 0 ]; then TZ=UTC; else TZ=Asia/Tokyo; fi
  PYTHONHASHSEED="$i" TZ="$TZ" \
    python -m pytest "$SUITE" -q -p no:cacheprovider \
      --junitxml="run-$i.xml" >"log-$i.txt" 2>&1
done
echo "completed runs: $RUNS"

The loop avoids any third-party plugin. --junitxml ships with pytest, so the classifier below needs only the standard library.

#!/usr/bin/env python3
"""verdicts.py - classify every test from the junit XML census."""
import collections
import glob
import xml.etree.ElementTree as ET

seen = collections.defaultdict(set)
for path in glob.glob("run-*.xml"):
    for case in ET.parse(path).iter("testcase"):
        nodeid = f"{case.get('classname')}::{case.get('name')}"
        tags = {child.tag for child in case}
        if "skipped" in tags:
            seen[nodeid].add("skip")
        elif tags & {"failure", "error"}:
            seen[nodeid].add("fail")
        else:
            seen[nodeid].add("pass")


def verdict(outcomes: set) -> str:
    if outcomes == {"pass"}:
        return "keep"
    if "fail" in outcomes and "pass" in outcomes:
        return "quarantine"
    if outcomes <= {"fail", "skip"}:
        return "spec-leak"
    return "review"


for nodeid, outcomes in sorted(seen.items()):
    print(f"{verdict(outcomes):11} {sorted(outcomes)} {nodeid}")

Illustrative output from a synthetic three-module repo, with numbers invented for shape only and no benchmark claim:

verdict	tests	action
keep	31	freeze, then refactor
quarantine	6	mark xfail(strict=False) and exclude from the gate
spec-leak	4	delete before refactoring
review	2	inspect by hand

The twenty-run loop is also a good fit for a free server option. It is idle compute, not interactive work, so a remote machine suits it better than your laptop.

The smallest safe change, defined numerically

With keep tests frozen, the change is now budgeted. One extraction per pull request. Zero signature changes. No renamed public functions.

# budget check before opening the PR
git diff --stat
# one extracted function, zero signature edits
git diff -U0 | grep -E '^[-+]def ' || echo "no signature churn"

Re-run the census after the change. The keep set must stay keep and the quarantine list must not grow. If quarantine grows, you changed behaviour, whatever the tests say.

Limitations and who should skip this

Characterization tests freeze bugs along with behaviour. Treat them as a refactor contract, not a specification. Delete them progressively after a real suite exists.

The census costs twenty full suite runs. On a slow suite that is hours, which is exactly why the remote loop matters. Env-var variation cannot seal external calls, so record fixtures for anything network-bound.

Skip this if the repo already has a meaningful, deterministic test suite. Skip it if a rewrite is genuinely cheaper than comprehension. Skip it if the code has no runnable entry point, because you cannot census what cannot start.

Free-tier offerings change. Verify current availability before planning work around them.

Start with one module from the top of the freeze table. Keep the census output. It is the only honest baseline you will ever get for that code.