A Test That Cannot Fail Is Decoration: Gating Model-Drafted Characterization Tests

Most refactors do not break because tests are missing. They break because the tests pin the wrong surface. Model-drafted tests make this worse, not better: they are fast, plausible, and confident about internals.

My rule for messy repos is simple. Draft many candidate tests. Admit only three-gate survivors. Then make the smallest safe change.

The three gates are determinism, assertion surface, and sentinel failure. A candidate test must pass all three before it can guard a refactor. A test that cannot fail is decoration.

Below is the workflow I use, with a harness sketch you can run before trusting any thresholds.

Step 0: Record one observed baseline

Before writing a single test, capture what the code does today. Run the entrypoint once and store the raw output.

python -m yourapp.report --account 42 > baseline/report.json
sha256sum baseline/report.json

That hash is your reference point for the whole session. If the refactor changes the bytes, you will know. Not "roughly the same JSON" — the same bytes, or an explained diff.

Keep this file out of version control churn. You want diff noise to mean something.

Step 1: Draft candidates with a free model

This is where free model access helps most. Drafting candidate characterization tests is volume work, and volume is cheap. Judgment is not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use a prompt contract, not a vague request. The contract below forces the model onto the observable surface and demands observed values.

You are drafting characterization tests for an unchanged module.
Rules:
1. Import only the public API listed below. Never touch names starting with _.
2. Each test performs one call and asserts one observable value or exact output string.
3. Expected values must come from the observed run I pasted. Do not guess.
4. No mocks of the module under test. Patch only network and clock.
5. Name each test after the behavior, not the function: test_rounds_discount_to_cents.
Public API: calculate_invoice, render_report
Observed run: <paste baseline output>

Ask for fifteen candidates, expect eight usable ones. Reject anything that asserts on a private attribute, a log line, or an exception message you did not paste.

Run the gate script on the candidates before you read them closely. Reading first biases you toward keeping the clever ones.

Step 2: Determinism gate

A characterization test that flips on its own is worse than no test. It teaches the team to ignore red.

Run every candidate twenty times with a different PYTHONHASHSEED. Require twenty passes.

# tools/gate_candidate_tests.py (sketch, tighten thresholds for your repo)
import os, subprocess, sys
from pathlib import Path

RUNS = 20

def run_pytest(target: Path, seed: str) -> int:
    env = {**os.environ, "PYTHONHASHSEED": seed}
    cmd = [sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", str(target)]
    return subprocess.run(cmd, env=env, capture_output=True).returncode

def determinism_gate(test_file: Path, runs: int = RUNS) -> bool:
    failures = [n for n in range(runs) if run_pytest(test_file, str(n)) != 0]
    print(f"determinism: {runs - len(failures)}/{runs} pass")
    return not failures

Twenty runs is a floor, not a proof. It catches hash-order dependence, unseeded randomness, and real clock reads. It will not catch a branch that fires once a month.

Use a free server for this step if your laptop sleeps. A gate that dies at run nine and reports "unknown" is a gate you will skip next time. Any always-on box works; the point is an uninterrupted twenty-run window.

Step 3: Surface gate

Now check what the assertions actually touch. Parse the test file and flag private attribute access.

import ast

def surface_violations(test_file: Path) -> list[str]:
    tree = ast.parse(test_file.read_text())
    bad = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Attribute) and node.attr.startswith("_"):
            bad.append(f"line {node.lineno}: private attribute .{node.attr}")
    return bad

This is syntactic, so it misses indirect access through helpers. Treat a clean report as permission to look, not proof of safety.

Two more manual checks matter. Does the test assert on a log message? Logs are not API. Does it assert on an exception type without a pinned message? That is fine for control flow, weak for refactors.

Prefer assertions on returned data and rendered output. Those survive most internal moves.

Step 4: Sentinel gate

The most important check. A characterization test must fail when behavior changes.

Write a sentinel patch: one deliberate, tiny behavior change. Apply it in a throwaway clone. The test must go red.

--- a/pricing.py
+++ b/pricing.py
@@
-    return round(subtotal * (1 - rate), 2)
+    return round(subtotal * (1 - rate), 1)

import shutil, tempfile

def sentinel_gate(repo: Path, test_file: Path, sentinel: Path) -> bool:
    with tempfile.TemporaryDirectory() as tmp:
        work = Path(tmp) / "repo"
        shutil.copytree(repo, work, ignore=shutil.ignore_patterns(".git"))
        applied = subprocess.run(["git", "apply", "--unsafe-paths", str(sentinel)],
                                 cwd=work, capture_output=True)
        if applied.returncode != 0:
            print("sentinel patch did not apply; fix the sentinel first")
            return False
        red = run_pytest(work / test_file.relative_to(repo), "0") != 0
        print(f"sentinel: {'caught the change' if red else 'DECORATION'}")
        return red

One sentinel per behavior you claim to protect. A test that stays green after a rounding change protects nothing about rounding.

Step 5: Make the smallest safe change

Only survivors guard the edit. Then the refactor rule applies: one edit, one full suite run.

Green suite plus unchanged baseline hash means you moved code, not behavior. Red suite means you learned something cheap.

Keep the change smaller than your confidence. If a test goes red, do not edit the test to match the new code. That is how refactors turn into silent behavior changes nobody reviews.

Decision table

Gate result	Action
Determinism fails	Quarantine; find the nondeterministic source first
Private attribute asserted	Rewrite against returned data
Passes all but sentinel stays green	Delete the test; it protects nothing
Passes all three	Keep; this one guards the edit
Red after a legitimate fix	Remove the test; characterization is not a spec

That last row matters. Characterization tests freeze current behavior, bugs included. If the frozen behavior is a real defect, file it and delete the test.

Limitations and who should skip this

• The sentinel gate only catches changes your sentinel touches. Choose sentinels per behavior, not one global patch.
• Twenty runs cannot find rare-path flakiness. Shorten the interval or widen the run count for hot paths.
• The AST check is syntactic. Helper-mediated private access passes it.
• Model-drafted tests encode the model's reading of your output. Paste raw observations, not summaries.
• Skip this if the target module has no runnable entrypoint. Build one first; gates need something to run.
• Skip it for a module you are actively rewriting anyway. Gates pay off on code you intend to keep.

If you want to try the drafting step without spending anything, run the prompt contract against a free model, then let the three gates decide what survives. The gates are the part worth keeping.