Characterize Failure Tokens Before You Extract a Predicate

A messy validator is unsafe until reject paths freeze. Characterization of failures still beats a pretty extract. Change one predicate only after the matrix holds.

The damage pattern

Happy-path tests miss most damage from refactors. Callers depend on failure shape, not structure.

Exit codes remain part of the public contract. Messages remain part of operator grep habits.

A tidy diff can still swallow a reject branch. A rename can still alter a stderr line. A helper extract can still change an exit code.

What this method freezes

This workflow freezes observable rejection before structure. It then permits one predicate extract, nothing more.

Three fields define each characterization row in this method. Those fields are payload, exit code, and stderr token.

Do not freeze helper names or internal layout. Do not freeze comment text or import order.

Teaching fixture, not production history

The next module is teaching code, not production. Treat outputs as unexecuted examples until you run them.

# example_validator.py — illustrative fixture only
import json
import sys

ALLOWED_KINDS = {"user", "service", "batch"}
MAX_NAME = 32


def run(argv):
    if len(argv) < 2:
        print("usage: validator <payload.json>", file=sys.stderr)
        return 2
    path = argv[1]
    try:
        with open(path, encoding="utf-8") as handle:
            payload = json.load(handle)
    except OSError:
        print("unreadable payload", file=sys.stderr)
        return 2
    except json.JSONDecodeError:
        print("invalid json", file=sys.stderr)
        return 2
    if not isinstance(payload, dict):
        print("object required", file=sys.stderr)
        return 1
    kind = payload.get("kind")
    name = payload.get("name")
    dry = payload.get("dry_run", False)
    if kind not in ALLOWED_KINDS:
        print("unknown kind", file=sys.stderr)
        return 1
    if not isinstance(name, str) or not name:
        print("name required", file=sys.stderr)
        return 1
    if len(name) > MAX_NAME:
        print("name too long", file=sys.stderr)
        return 1
    if any(ch in name for ch in " /\\"):
        print("name has path chars", file=sys.stderr)
        return 1
    if dry not in (True, False):
        print("dry_run must be bool", file=sys.stderr)
        return 1
    print(f"ok:{kind}:{name}:{int(dry)}")
    return 0


if __name__ == "__main__":
    raise SystemExit(run(sys.argv))

The fixture mixes parsing, policy, and printing on purpose. That mix is the usual local mess. Extra keys are accepted. Nested objects are not.

Encode the reject matrix as data

Store the matrix beside the fixture, not comments. Comments drift during later cleanup and review passes. Checked-in matrix files get grepped by the harness.

{
  "cases": [
    {"id": "no_args", "mode": "no_file", "exit": 2, "stderr_token": "usage: validator"},
    {"id": "missing_path", "mode": "missing", "exit": 2, "stderr_token": "unreadable payload"},
    {"id": "bad_json", "raw": "{", "exit": 2, "stderr_token": "invalid json"},
    {"id": "array_root", "payload": [1, 2], "exit": 1, "stderr_token": "object required"},
    {"id": "bad_kind", "payload": {"kind": "admin", "name": "ok"}, "exit": 1, "stderr_token": "unknown kind"},
    {"id": "empty_name", "payload": {"kind": "user", "name": ""}, "exit": 1, "stderr_token": "name required"},
    {"id": "long_name", "payload": {"kind": "user", "name": "n012345678901234567890123456789012"}, "exit": 1, "stderr_token": "name too long"},
    {"id": "path_chars", "payload": {"kind": "user", "name": "acme/ops"}, "exit": 1, "stderr_token": "name has path chars"},
    {"id": "dry_string", "payload": {"kind": "user", "name": "ok", "dry_run": "true"}, "exit": 1, "stderr_token": "dry_run must be bool"},
    {"id": "happy", "payload": {"kind": "batch", "name": "nightly", "dry_run": true}, "exit": 0, "stdout_token": "ok:batch:nightly:1"}
  ]
}

Happy path stays in the same table on purpose. A reject-only file hides accidental accept flips. One table shows both directions after each edit.

Harness that records tokens, not vibes

The harness below is an unexecuted example. Copy it, then run it on your tree. Do not trust pasted output as measured evidence.

# reject_matrix.py — illustrative characterization harness
import json
import subprocess
import sys
import tempfile
from pathlib import Path

ROOT = Path(__file__).resolve().parent
SCRIPT = ROOT / "example_validator.py"
MATRIX = ROOT / "reject_matrix.json"
SNAPSHOT = ROOT / "reject_matrix.snapshot.json"


def run_case(case):
    with tempfile.TemporaryDirectory() as tmp:
        payload_path = Path(tmp) / "payload.json"
        argv = [sys.executable, str(SCRIPT)]
        mode = case.get("mode")
        if mode == "no_file":
            pass
        elif mode == "missing":
            argv.append(str(Path(tmp) / "missing.json"))
        else:
            if "raw" in case:
                payload_path.write_text(case["raw"], encoding="utf-8")
            else:
                payload_path.write_text(json.dumps(case["payload"]), encoding="utf-8")
            argv.append(str(payload_path))
        proc = subprocess.run(argv, capture_output=True, text=True)
        return {
            "id": case["id"],
            "exit": proc.returncode,
            "stdout": proc.stdout.strip(),
            "stderr": proc.stderr.strip(),
        }


def token_ok(actual, case):
    if case.get("stderr_token") and case["stderr_token"] not in actual["stderr"]:
        return False
    if case.get("stdout_token") and case["stdout_token"] not in actual["stdout"]:
        return False
    return actual["exit"] == case["exit"]


def main(write=False):
    cases = json.loads(MATRIX.read_text(encoding="utf-8"))["cases"]
    rows = [run_case(case) for case in cases]
    report = []
    failed = 0
    for case, actual in zip(cases, rows):
        ok = token_ok(actual, case)
        failed += int(not ok)
        report.append({"id": case["id"], "ok": ok, **actual})
    if write:
        SNAPSHOT.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
    print(json.dumps({"failed": failed, "rows": report}, indent=2))
    return 1 if failed else 0


if __name__ == "__main__":
    write = "--write" in sys.argv
    raise SystemExit(main(write=write))

Token checks beat full-string equality for this fixture. Usage lines may gain a path later. Exit codes still must match exactly.

Numbered workflow

1. Inventory reject classes from current behavior

List every reject class the module currently emits. Use stderr tokens and exit codes as labels.

Do not inventory intended design from memory. Inventory what HEAD prints today.

python3 example_validator.py
echo $?
python3 example_validator.py /tmp/missing.json
echo $?
printf '{' > /tmp/bad.json
python3 example_validator.py /tmp/bad.json
echo $?

2. Turn each class into a tiny payload

Turn each class into a JSON fixture file. Keep each payload tiny and obviously invalid.

One payload per class keeps later diffs readable. Combined payloads hide which check moved.

3. Run the harness once against HEAD

Run the harness on the current tree first. Save the report as the locked snapshot.

python3 reject_matrix.py --write
git add reject_matrix.json reject_matrix.snapshot.json example_validator.py
git commit -m "Lock reject tokens before predicate extract"

Uncommitted snapshots do not protect later diffs. A dirty matrix is not a gate.

4. Classify each row before touching code

Use the table below as a stop rule. Skip extracts that need new behavior.

Observed row	Safe next edit	Unsafe next edit
Token and exit stable	Extract one boolean predicate	Rename stderr text
Extra keys already accepted	Leave extra-key policy untouched	Add a surprise denylist
dry_run rejects strings	Keep bool check in the same place	Coerce "true" to boolean
Path chars rejected	Extract has_path_chars only	Expand to Unicode slash folds
Happy row still prints ok:	Keep print format unchanged	Switch to JSON logs
Missing file returns 2	Keep IO errors as 2	Merge IO errors into 1

If the wanted change is policy, stop this workflow. Characterization protects behavior, not product intent. Policy needs an explicit spec, not a snapshot.

5. Extract one predicate, not a layer

Extract one boolean helper, not a new layer. Keep print statements and return codes in place.

def has_path_chars(name):
    return any(ch in name for ch in " /\\")

Wire it in one call site only. Do not move JSON loading in the same patch. Do not move exit-code mapping in the same patch.

    if has_path_chars(name):
        print("name has path chars", file=sys.stderr)
        return 1

That is the whole structural change. Anything larger is a second change.

6. Re-run the matrix as a byte gate

The snapshot must match on tokens and exits. Any stderr drift is a failed extract.

python3 reject_matrix.py
python3 - <<'PY'
import json
from pathlib import Path
live = json.loads(Path("reject_matrix.snapshot.json").read_text())
# Re-write live only after a deliberate matrix update.
print("locked_rows", len(live))
PY
git diff --exit-code -- example_validator.py reject_matrix.json

If the helper extract is correct, only the validator file changes. Matrix files stay quiet. Quiet matrix files are the passing signal.

7. Stop after one green extract

Do not chain a second extract in the same change. One green matrix is the whole change.

A second extract hides which edit broke a token. Split commits keep the bisect cheap.

Where a free model belongs

A coding model can propose the predicate extract. It cannot own the reject matrix.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use that pair to draft the helper on a scratch tree after HEAD is locked.

Keep the matrix runner on the repo you actually ship. Treat model output as a patch candidate. Merge only when reject_matrix.py reports zero failures.

Prompt the model with the frozen table, not with vibes. Paste the one helper boundary. Forbid stderr edits in the same request.

Propose a patch that extracts has_path_chars only.
Do not change stderr strings.
Do not change exit codes.
Do not touch JSON loading.
Stop after one helper.

If the patch edits more than the predicate, discard it. Breadth is the usual failure mode here.

Failure analysis the matrix still misses

This table does not cover locale. It does not cover concurrent writers. It does not cover symlink tricks on the payload path.

Unicode homoglyphs can still pass the path check. MAX_NAME counts Python characters, not graphemes. Extra keys remain untested policy.

dry_run missing is not the same as false. The fixture treats missing as false today. Record that fact as a row if callers depend on it.

Add that row only as a new commit. Expanding the matrix is not an extract. Mixing both hides the cause of a miss.

Commands for a disposable loop

Use a throwaway clone for model drafts. Keep the locked snapshot in the real clone.

git clone --local . /tmp/validator-scratch
cd /tmp/validator-scratch
python3 reject_matrix.py --write
# apply a model patch here, then:
python3 reject_matrix.py

Compare exits between clones with the same matrix file. Different tokens mean the patch is not small. Delete the scratch clone after the gate fails or passes.

Limitations

This method does not replace typed contracts. It does not replace fuzzing. It does not replace a real parser test suite.

Token matching can hide extra stderr noise. A second line can appear and still pass. Tighten tokens only when operators grep exact lines.

Subprocess tests miss in-process monkeypatches. They also miss import-time side effects. If loading the module prints, add an import row.

The workflow assumes deterministic stderr. Timestamps in errors break snapshots. Strip clocks before you freeze tokens.

Who should not use this approach

Do not use this as an authorization test. Do not use this as a crypto parser test. Do not use this as a privacy boundary test.

Teams without a committed snapshot should not start extracting. Teams changing product policy should not hide behind characterization. Teams that cannot run subprocesses locally should pick in-process tests.

If the validator already has stable unit tests, use those first. A reject matrix is for untested mess. It is not a badge for clean modules.

Closing rule

Freeze failure tokens, then extract one predicate. Re-run the matrix. Stop.

If a scratch model loop helps, keep it behind that gate. The matrix stays the authority, not the diff.