Time Your Safety Net First: A Budget Gate for Characterization Tests in Messy Repos

A characterization suite you skip is not a safety net. It is documentation. Before you refactor a messy repo, measure how long the net takes, cap it, and only then start making one change per commit.

This post is about the step before the refactor: timing the net. It includes a script, a budget table, a pre-commit gate, and the cases where the whole idea falls apart.

The rule

If the net takes longer than your patience, you will stop running it. That is the real failure mode. Not a wrong test. Not a clever bug. Just a suite you quietly stop executing around commit six.

So the first commit in a messy-repo refactor is not code. It is a measurement. Then a budget. Then a gate that refuses to let you drift.

Step 1: Baseline the net three times

One run tells you little. A cold cache or a busy laptop skews it. Take the median of three runs and ignore the mean.

#!/usr/bin/env bash
# scripts/net-budget.sh — time the characterization suite, fail if over budget.
set -euo pipefail

CMD="${NET_CMD:-./scripts/test-net.sh}"
BUDGET_FILE="${BUDGET_FILE:-.net-budget}"
RUNS="${RUNS:-3}"

budget_seconds=$(tr -d '[:space:]' < "$BUDGET_FILE")

times=()
for _ in $(seq 1 "$RUNS"); do
  start=$(date +%s%N)
  if ! bash -c "$CMD" >/tmp/net.out 2>/tmp/net.err; then
    echo "FAIL: net is red. Do not refactor." >&2
    tail -n 20 /tmp/net.err >&2
    exit 1
  fi
  end=$(date +%s%N)
  times+=( $(( (end - start) / 1000000 )) )
done

IFS=$'\n' sorted=($(sort -n <<<"${times[*]}")); unset IFS
mid=$(( ${#sorted[@]} / 2 ))
median_ms=${sorted[$mid]}
budget_ms=$(( budget_seconds * 1000 ))

printf 'median=%dms budget=%dms runs=%s\n' "$median_ms" "$budget_ms" "$RUNS"

if (( median_ms > budget_ms )); then
  echo "FAIL: net too slow to run before every micro-change." >&2
  exit 1
fi
echo "PASS: net is inside budget."

The script fails on a red net before it reports any timing. A fast red suite is still a stop sign.

Example output, with placeholder numbers for format only:

median=7400ms budget=10000ms runs=3
PASS: net is inside budget.

One portability note: date +%s%N works on Linux, not on stock macOS. Use gdate from coreutils, or time with python3 -c 'import time; print(time.time_ns())'.

Step 2: Write the budget, tie it to change size

A budget is a number in a file. Keep it boring.

echo 10 > .net-budget

Then tie the number to how often you will run it. The table below is a starting point to calibrate against your own baseline, not a measurement of your repo.

Planned change	Max net runtime	Why
Rename or move one symbol	10 s	Run 20+ times per hour without friction
Extract one function	30 s	Run per micro-commit, twice on red
Move a module across packages	60 s	Run twice per move, once after import fixes
Anything above 60 s	Split the net first	The gate will lose to your deadline

If the baseline exceeds the budget, do not raise the budget. Shrink the net.

Step 3: Shrink the net before you shrink the code

Split the suite into a fast core and a slow tail. The core guards behavior at the seam you are about to cut. The tail runs on push, not on every save.

With pytest, markers plus --durations do this in one command.

# Fast core: characterization tests only, no sleeps, no network.
python -m pytest tests/char -q -m "not slow" --durations=10

Read the ten slowest tests. In most messy repos, two or three tests own half the wall clock. Usually they sleep, hit a real socket, or start a container. Replace the socket with a local fixture and the median drops without weakening the net.

Tag aggressively. -m "not slow" is a contract: anything that sleeps for more than a second gets the marker. Review that contract in the pull request.

Step 4: Gate every micro-commit

Measure once, then enforce. A pre-commit hook is enough.

#!/usr/bin/env bash
# .git/hooks/pre-commit
set -euo pipefail
NET_CMD=./scripts/test-net-fast.sh ./scripts/net-budget.sh

Run chmod +x .git/hooks/pre-commit and note it in the repo README. Hooks are local, so also run the same script in CI to catch machines that skip hooks. CI should run it with a loose budget and warn, not block. Slow shared runners are normal, and you do not want to train people to bypass the job.

Step 5: Keep the change smaller than the net

Four rules carried us through the ugly parts:

1. One seam per commit. If the diff touches two call paths, split it.
2. Never rename with a move. Renames hide inside moves in every diff viewer.
3. No test edits in a move commit. If a test must change, that is a separate commit with its own reason.
4. On red, revert first. git checkout -- . then re-cut the change smaller. Do not debug the net and the change at the same time.

Rule four is the one people break. A red net after a small move is information about the move, not about the test.

Where a free model and a free server change the loop

Drafting characterization tests is a volume game. You need many attempts, and most attempts are wrong in boring ways: assertions on formatting, mocked return values, one missing edge case.

The operator of this account states that MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For this workflow, the relevant effect is economic, not magical. Cheap attempts mean you can draft ten characterization tests for one function and keep the two that survive a mutation check. That is the useful pattern, and it does not require a specific model name to be true.

One honest caveat about the free server. Timing measured on a contended shared machine is not a stable baseline. Set the budget on the machine where you actually refactor, and treat server timings as a separate, looser number. Do not copy a budget between environments and call it a gate.

Limitations and who should skip this

This approach has narrow assumptions, and they do not always hold.

• If your safety net is a genuine integration suite that cannot be split, a 10-second budget is fiction. Skip the gate and use a slower, deliberate cadence instead.
• If the suite is flaky rather than slow, timing it is the wrong first move. Fix flakes before you enforce a budget, or you will blame the clock for a race.
• If the net depends on a shared database or a staging environment, the median will move with the environment. Budget per environment or not at all.
• If your team already runs the net on every save, you do not need this post. You need a smaller diff habit.

The takeaway

Measure the net. Write one number in one file. Fail the commit when the number is exceeded. Then make the smallest change the net can actually cover.

Start by pointing NET_CMD at your own suite and setting a budget you currently violate. The first failing run tells you more about your repo than any refactor plan will.