The Smallest Safe Change Is a Measured Property

The smallest safe change is a measured property. It is not a line count. It is not a gut feeling. It is the difference between two test runs. Identical output means the change is safe. Different output means behavior changed. This protocol makes that measurement explicit.

Why Line Counts Fail

A one-line change can reorder a side effect. A 400-line change can be pure renames. Diffs hide risk. They show text movement, not behavior change. Only tests reveal behavior.

Consider a rename. The diff shows one line changed. The behavior is identical. Now move a logging call before a validation check. The diff shows two lines moved. The behavior changed. Users see a log line for invalid input. Line counts cannot distinguish these cases.

Characterization tests lock current behavior. Then every edit becomes a controlled experiment. Run before. Run after. Compare the output. The verdict is objective.

The Protocol

Five steps. Each produces a number or a verdict. Follow them in order. Do not skip Step 3.

Step 1: Slice by Behavior

Open the target file. List every public function. List every side effect. List every state transition. Ignore private helpers. They will change during the refactor. Give each slice its own test file.

A good slice is small. It has one entry point. It has a clear input and output. It has no hidden global state. If a slice touches a database, mock the database. If it reads a file, mock the file.

Step 2: Generate Characterization Tests

Send one slice to a free model. Ask for tests that record current behavior. Do not ask for improvements. Do not ask for refactors. Ask for observations only.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access can draft the test skeleton. You verify every assertion against the source. The model is a typist, not an oracle. Delete wrong assertions. Add missing ones.

Good characterization tests use real inputs. They capture real outputs. They do not assert style. They do not assert intent. They assert facts.

Step 3: Measure the Baseline

Run the slice tests. Record the pass count. Record the coverage percentage. Store both in a file. This is your baseline. The protocol is only as strong as this number.

Below 70% coverage, stop and add tests. Untested branches hide behavior changes. The digest comparison cannot see what the tests cannot see.

Step 4: Make the Smallest Edit

Change one behavior. Run the tests again. Green means the edit is safe. Red means a hidden dependency. Revert or split the edit. Then try again.

One behavior per edit. One edit per run. This is the discipline that makes the measurement meaningful.

Step 5: Compare the Digests

Run the same tests after the edit. Compare the full output. Exit codes are not enough. Warnings and logs matter too. Identical output means the refactor preserved behavior.

The Artifact: A Before/After Digest Check

This script automates the comparison. It hashes the complete test output. It does not trust exit codes alone.

#!/usr/bin/env python3
"""Prove a refactor is small: compare full test output before and after."""
import hashlib
import json
import subprocess
import sys
from pathlib import Path

BASELINE = Path(".refactor_baseline.json")

def digest(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def snapshot(command: list[str]) -> dict:
    result = subprocess.run(command, capture_output=True, text=True)
    return {
        "returncode": result.returncode,
        "stdout": digest(result.stdout),
        "stderr": digest(result.stderr),
    }

def save(snap: dict) -> None:
    BASELINE.write_text(json.dumps(snap, indent=2))

def load() -> dict:
    return json.loads(BASELINE.read_text())

def main() -> int:
    args = sys.argv[1:]
    if not args:
        print("usage: refactor_proof.py <test command> [--save]")
        return 2
    if "--save" in args:
        args.remove("--save")
        save(snapshot(args))
        print("baseline saved")
        return 0
    before = load()
    after = snapshot(args)
    if before == after:
        print("SAFE: behavior unchanged")
        return 0
    print("UNSAFE: behavior changed")
    return 1

if __name__ == "__main__":
    raise SystemExit(main())

Usage:

python refactor_proof.py pytest tests/test_slice.py --save   # before
# make the smallest edit
python refactor_proof.py pytest tests/test_slice.py          # after

The script returns 0 only when the output matches. CI can use it as a gate. The digest catches warnings, ordering, and stderr noise. Exit codes miss all of that.

The baseline file is small. It holds three values: returncode, stdout hash, stderr hash. Commit it to the repository. Reviewers can see what you locked. The script records failing runs too. A red baseline is still a baseline. It locks the failure output.

A Worked Example

Suppose parse_config() has 200 lines. It reads a file, validates keys, and returns a dict. You want to extract the validation into a helper.

Slice it first. Write test_parse_config.py. Generate tests with the free model. Verify every assertion. Run the baseline. Coverage is 85%. Good.

Extract the validation helper. Run the comparison. The output is identical. SAFE. Merge.

Now rename a variable inside the helper. Run again. A warning appears. UNSAFE. The warning is a deprecation from the old code path. You found a hidden behavior. Fix it or document it.

That is the workflow. Small slices. Measured verdicts. No guessing.

Try the same protocol on a 1,000-line module. The slices multiply. Each slice gets a baseline. Each edit gets a verdict. The total time is longer. The risk per edit stays flat. That is the trade-off: more setup, less surprise.

Reading the Verdict

SAFE means the slice's observable behavior is unchanged. Merge the edit. Move to the next slice.

UNSAFE means something changed. Read the diff. Find the hidden dependency. The test output usually names it.

UNSAFE can also mean flaky tests. Run the baseline twice. If the baseline is unstable, fix the tests first. Never compare against a moving target.

Limitations

Characterization tests lock current behavior. Current behavior can be wrong. If the code has bugs, the tests bless them. The protocol proves stability, not correctness.

Generated tests need human review. Free models hallucinate assertions. They invent expected values that do not exist. Verify every line before saving the baseline.

The digest is all-or-nothing. One changed warning fails the check. That is strict by design. It forces you to understand every difference.

Who Should Not Use This

Skip this protocol for greenfield code. Write real behavioral tests instead. Skip it for throwaway scripts. The overhead is not worth it. Skip it when the team already has strong tests. Characterization tests would duplicate them.

Running It on a Free Server

MonkeyCode's free server option can host this harness. The workflow is identical. The value is the protocol, not the host. Run the baseline locally. Run the comparison anywhere.

Take one messy file this week. Run the protocol. Let the digest be the judge.