Hold Out a Third of Your Corpus: The Characterization Gate for a Messy Refactor

The gate is simple: pin behavior on inputs you can see, then grade the refactor on inputs you cannot. A refactor that passes the visible corpus and fails the hidden one was never safe. It was fitted.

Most characterization suites rot this way. You write tests against the three files you have open. The model, or you, then patches code until those three pass. Nothing proves the other forty inputs still behave.

This post builds a fingerprint harness with a canary/holdout split. Then it sets a rule for the smallest change that can honestly ship.

1. Define the behavior surface before you touch code

A refactor preserves observable behavior. So you must name what is observable, or the word means nothing.

For a CLI-shaped tool, four surfaces cover most of it:

Surface	Captured as	Catches
Exit code	integer	swallowed exceptions, new early returns
stdout bytes	SHA-256	reordered fields, changed rounding, lost whitespace
stderr presence	boolean	new warnings, debug prints left behind
files written	path + SHA-256 list	changed output paths, dropped side files

Deliberately excluded: internal call order, log timings, object identity. Those change under every refactor. Pinning them buys you failing tests and no safety.

Write that list down first. Every later argument about "did we break it" resolves against this table, not against memory.

2. Split the corpus before you look at results

Take your input corpus. Sort the filenames. Give every third case to the holdout set. The rule is deterministic, so it is reproducible, and it is chosen before any code changes.

corpus/
  case_000/input.txt
  case_001/input.txt
  ...

Canary gets two-thirds. Holdout gets one-third. The holdout stays closed until the refactor is finished. Not after step three. After the last commit.

Why one-third and not ten percent? Because one-third of thirty cases is ten, and ten is enough to see a pattern failure. One or two holdout cases can fail by coincidence and teach you nothing.

If you have fewer than twelve cases, stop. Do not refactor. Collect inputs first.

3. Record the baseline on the unmodified tree

Run the harness on a clean checkout. Commit the JSON. This file is the contract.

#!/usr/bin/env python3
"""fingerprint.py <repo> <corpus> <out.json> - pin observable behavior."""
import hashlib, json, subprocess, sys, tempfile
from pathlib import Path

ROOT = Path(sys.argv[1]).resolve()
CORPUS = Path(sys.argv[2]).resolve()
OUT = Path(sys.argv[3]).resolve()
TIMEOUT = 30


def sha(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def fingerprint(case: Path) -> dict:
    work = Path(tempfile.mkdtemp(prefix="fp-"))
    cmd = [sys.executable, str(ROOT / "tool.py"), str(case / "input.txt")]
    try:
        proc = subprocess.run(cmd, capture_output=True, cwd=work, timeout=TIMEOUT)
    except subprocess.TimeoutExpired:
        return {"timeout": True}
    files = sorted(
        [p.relative_to(work).as_posix(), sha(p.read_bytes())]
        for p in work.rglob("*")
        if p.is_file()
    )
    return {
        "exit": proc.returncode,
        "stdout": sha(proc.stdout),
        "stderr_present": bool(proc.stderr.strip()),
        "files": files,
    }


def split(cases: list) -> dict:
    return {
        "canary": [c for i, c in enumerate(cases) if i % 3 != 0],
        "holdout": [c for i, c in enumerate(cases) if i % 3 == 0],
    }


cases = sorted(p for p in CORPUS.iterdir() if (p / "input.txt").is_file())
result = {name: {c.name: fingerprint(c) for c in group}
          for name, group in split(cases).items()}
OUT.write_text(json.dumps(result, indent=2, sort_keys=True))
print(f"canary={len(result['canary'])} holdout={len(result['holdout'])}")

The script is short on purpose. It has no dependencies, no fixtures, and no framework. You can run it in CI or on a laptop.

Run it twice on the untouched tree. If the two JSON files differ, you have nondeterminism. Fix that before any refactor, or every later failure is ambiguous.

4. Set the smallest safe change rule

The point of a gate is to constrain scope, not to bless ambition. Use the behavior surface table as a budget.

Change touches	Allowed in one commit	Evidence required
One function body, no signature	yes	canary + holdout match
One function signature, internal callers only	yes	callers updated in same commit
Module moved to a new path	no	separate commit, imports only
Output ordering or formatting	no	explicit human sign-off
Public function signature	no	not a refactor; it is a redesign

Row five is the one people argue about. A signature change is a behavior change by definition. Ship it as its own reviewed change with its own tests, not folded into a cleanup.

5. Replay, then read the two numbers separately

After the change, run the harness again. Compare per split, never in aggregate.

import json, sys

base = json.load(open(sys.argv[1]))
now = json.load(open(sys.argv[2]))

exit_code = 0
for name in ("canary", "holdout"):
    drifted = [k for k, v in base[name].items() if now[name].get(k) != v]
    print(f"{name}: {len(drifted)}/{len(base[name])} drifted")
    for k in drifted[:5]:
        print(f"  - {k}")
    exit_code |= bool(drifted)

sys.exit(exit_code)

Three outcomes matter.

1. Canary 0, holdout 0. The change is behavior-preserving on this corpus. Commit it.
2. Canary 0, holdout > 0. The change is wrong. The corpus was broad; the tests were narrow. Revert the commit, do not patch the harness.
3. Canary > 0. You already know what to do. Inspect the diff before touching a single assertion.

Outcome two is the one that saves you. Without a holdout split, it looks exactly like outcome one.

6. Apply the revert rule, not the negotiation rule

When holdout drifts, the instinct is to update the expected values. Resist it once.

Holdout drift cause	Action
Renamed output key	revert; rename in a separate commit
Changed float formatting	revert; formatting is a behavior change
Dropped file write	revert; a side effect disappeared
Genuine pre-existing bug	revert, then fix bug separately, then re-baseline
Corpus case is invalid input	remove the case, re-baseline on clean tree

Only the last row permits touching the baseline. And it requires re-running step three on an unmodified tree, so the contract stays honest.

Where free model access fits, and where it does not

The harness above is mechanical. The slow part is the corpus and the first draft of the fingerprint script — and that is where a free model tier is genuinely useful. I used MonkeyCode's free model access to draft the harness and to propose corpus edge cases, then read every line before running it. The operator also describes a free server option, which matters here for one reason: the replay must run on a clean machine, not your laptop with a stale virtualenv. A throwaway environment makes step three reproducible.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use the free tooling for the disposable parts. Do not let a model decide which behavior counts as observable. That table is a design decision, and it is yours.

On availability: the operator states a free token allowance in the millions-per-account range, plus the free server option. Quotas and hardware change without notice. Verify current terms on the project page before you plan around them.

Limitations and who should skip this

• The fingerprint is byte-level. It will flag cosmetic changes you intended. That is the price of catching the ones you did not.
• Thirty cases is a floor, not a target. Real tools need cases drawn from production traffic, not from your imagination.
• Timing, memory, and concurrency are not covered. This gate says nothing about performance regressions.
• Skip this if your tool has no stable text output. A UI-heavy service needs snapshot diffs of rendered state instead.
• Skip this if the refactor changes the public contract. Then you want migration tests, not a preservation gate.

The short version

Pick four observable surfaces. Sort your corpus and hold out every third case. Baseline on a clean tree, twice. Change one thing. Replay both splits. If the holdout moves, revert.

The hidden third is the only honest reviewer you get for free. If you try the harness, keep the corpus small and the rule strict at first — and tell me which holdout case caught you.

Further reading: Martin Fowler on characterization tests, and Michael Feathers, "Working Effectively with Legacy Code", chapter 13.