The Falsifier First: A 90-Minute Spike With a Kill Switch

A scene that plays out on many teams: a shared document, three engineers, and one question that has been open since Monday — should the image pipeline move to a different resize library? By Thursday the document holds eleven comments, two benchmark screenshots taken on different laptops, and still no decision. Nobody spent more than a few minutes actually measuring anything, because measuring felt like the expensive part.

The team in that scene is a composite, assembled from the kind of argument most of us have sat through. The workflow below is the part worth testing on your own repository. Its claim is narrow: a ninety-minute spike fails not because the box is too short, but because nobody wrote down, in advance, what evidence would end it.

A hypothesis produces output; a falsifier produces knowledge

Most spikes open with a hypothesis and then start building, which quietly guarantees that ninety minutes will end with something to show. That something is usually a prototype nobody can judge, and the argument resumes on Friday with better screenshots. The missing piece is a command that, if it succeeds, proves the hypothesis wrong.

A falsifier has one property a hypothesis never has: it can run without you. It is a shell invocation with a numeric threshold, an exit code, and a variance bound, so the decision is made by the evidence rather than by whoever is most tired. Write it before you write any implementation code, and freeze the threshold by minute twenty.

The contract, written before the first command

Keep the contract in a plain environment file, because a file you can read in a terminal is a file you will actually fill in. Nothing here needs a schema or a service; the point is that the numbers exist before the work does.

# spike/contract.env — fill this in before running anything
HYPOTHESIS="libx resizes 4K PNGs at least 30% faster than liby at equal SSIM"
FALSIFIER="./spike/measure.sh --both --repeat 5 --seed 7 | ./spike/compare.py --min-gain 0.30 --max-cv 0.15"
KILL_IF="compare.py exits 1: gain below 30%, or coefficient of variation above 15%"
BOX_MINUTES=90
THRESHOLD_FROZEN_AT=20

The KILL_IF line matters more than it looks, because it names the two ways a spike can lose: the effect is too small, or the measurement is too noisy to see the effect at all. Treating noise as a kill rather than an invitation to run longer is what keeps the box honest. A noisy instrument is a finding about the instrument.

Run the falsifier against a known zero first

Before implementing anything, point both arms of the comparison at the same library. Identical inputs must produce a gain near zero and a kill verdict; if the harness instead reports a win, the instrument is broken and the next ninety minutes would manufacture a false positive. This null run costs about ten minutes and saves entire afternoons.

# spike/measure.sh (extract) — emits one JSON line per sample
for engine in $ENGINES; do
  for i in $(seq 1 "$REPEAT"); do
    ms=$( { /usr/bin/time -f '%e' ./bench_one "$engine" "$SEED" ; } 2>&1 )
    printf '{"engine":"%s","ms":%s,"seed":%s}\n' "$engine" "$ms" "$SEED"
  done
done

The comparator is deliberately dull, since a clever decision rule is a rule you cannot audit later. Medians resist a single slow sample, and the coefficient of variation catches the case where the two arms overlap so much that the gain figure is decoration.

#!/usr/bin/env python3
# spike/compare.py — decide with a margin, not with a feeling
import argparse, json, statistics, sys

ap = argparse.ArgumentParser()
ap.add_argument("--min-gain", type=float, required=True)
ap.add_argument("--max-cv", type=float, default=0.15)
args = ap.parse_args()

rows = [json.loads(l) for l in sys.stdin if l.strip()]
arms = {}
for r in rows:
    arms.setdefault(r["engine"], []).append(r["ms"])

base, cand = statistics.median(arms["liby"]), statistics.median(arms["libx"])
gain = (base - cand) / base
cv = max(statistics.pstdev(v) / statistics.mean(v) for v in arms.values())
verdict = "kill" if gain < args.min_gain or cv > args.max_cv else "ship"
print(json.dumps({"gain": round(gain, 3), "cv": round(cv, 3), "verdict": verdict}))
sys.exit(0 if verdict == "ship" else 1)

The runner below enforces the box and appends every action to a JSONL evidence log, so the kill note at the end can quote timestamps instead of memory. Both scripts above are reference implementations written for this article; adapt the paths and the benchmark body before trusting them.

#!/usr/bin/env bash
# spike/run.sh — one hypothesis, one deadline, one falsifier
set -euo pipefail
set -a; . spike/contract.env; set +a

[[ -f spike/.started ]] || date +%s > spike/.started
elapsed=$(( ($(date +%s) - $(cat spike/.started)) / 60 ))
if (( elapsed >= BOX_MINUTES )); then
  printf 'box closed at %s minutes: stop coding, write the kill note\n' "$elapsed" >&2
  exit 2
fi

record() { printf '{"t":"%s","kind":"%s","note":"%s"}\n' \
  "$(date -u +%FT%TZ)" "$1" "$2" >> spike/evidence.jsonl; }

record start "$HYPOTHESIS"
if eval "$FALSIFIER"; then
  record result "not falsified"
else
  record result "falsified"
  printf 'falsified at minute %s — that is a result, not a failure\n' "$elapsed" >&2
  exit 1
fi

Where a hosted free tier actually earns its place

Two parts of this workflow are boring infrastructure rather than thinking: drafting the harness, and running both measurement arms in one identical environment. Splitting the arms across two laptops is the exact mistake that produced the Thursday screenshots, and a laptop that throttles halfway through the second arm will happily invent a thirty percent win for whichever library ran first.

This is where MonkeyCode's operator-supplied free model access and free server option become relevant rather than decorative. Free model access is useful for drafting measure.sh and reading the upstream changelog for defaults that changed between versions, while the free server gives the spike a clean container where both arms share one image and the production credentials never enter the room. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The repository's own page lists the current terms of the free tier, and those terms are the ones that count on the day you run the spike.

There is a real limitation worth stating plainly. Shared free hardware makes absolute latency and throughput numbers untrustworthy, so publish relative comparisons measured inside a single session, and never quote hosted runs as a benchmark of anyone's library. Correctness harnesses, contract drafting, and environment parity survive the caveat comfortably; performance claims do not.

Minute ninety, and the only four outcomes

With the box closed, the decision table should already be half-written by the evidence log, and no cell in it allows the threshold to move.

Evidence at minute 90	Outcome	Artifact to keep
Falsifier kills it, variance under bound	Kill the idea	Evidence log plus the one-paragraph note
Falsifier passes, gain above threshold	Ship a follow-up ticket, not the prototype	Contract, comparator, measured gain
Variance above bound	Extend once by 30 minutes, to fix the instrument only	Note naming the noise source
Null run failed	Restart the clock with zero code written	Recorded broken instrument

The middle rows are where discipline usually leaks, because a noisy run feels like progress and an extra half hour feels cheap. Extending to repair the instrument is legitimate; extending to hunt for a number you already want is how a ninety-minute spike becomes a two-week rumor. Skipping the null run is the fastest route back to Thursday's document.

Teams with a statistically demanding performance question should not use this at all — the box is too short and the sample count too small for anything you would publish. Anyone whose hypothesis cannot be phrased as a command with a number should split it until it can. For everyone else, the falsifier is the cheapest thing to write and the only thing that lets a spike end without an argument; if you want to try the loop with free model access and a free server, the project page is the honest place to start.