DEV Community: Harper Zhu

The Falsifier First: A 90-Minute Spike With a Kill Switch

Harper Zhu — Tue, 27 Oct 2026 12:45:52 +0000

A scene that plays out on many teams: a shared document, three engineers, and one question that has been open since Monday — should the image pipeline move to a different resize library? By Thursday the document holds eleven comments, two benchmark screenshots taken on different laptops, and still no decision. Nobody spent more than a few minutes actually measuring anything, because measuring felt like the expensive part.

The team in that scene is a composite, assembled from the kind of argument most of us have sat through. The workflow below is the part worth testing on your own repository. Its claim is narrow: a ninety-minute spike fails not because the box is too short, but because nobody wrote down, in advance, what evidence would end it.

A hypothesis produces output; a falsifier produces knowledge

Most spikes open with a hypothesis and then start building, which quietly guarantees that ninety minutes will end with something to show. That something is usually a prototype nobody can judge, and the argument resumes on Friday with better screenshots. The missing piece is a command that, if it succeeds, proves the hypothesis wrong.

A falsifier has one property a hypothesis never has: it can run without you. It is a shell invocation with a numeric threshold, an exit code, and a variance bound, so the decision is made by the evidence rather than by whoever is most tired. Write it before you write any implementation code, and freeze the threshold by minute twenty.

The contract, written before the first command

Keep the contract in a plain environment file, because a file you can read in a terminal is a file you will actually fill in. Nothing here needs a schema or a service; the point is that the numbers exist before the work does.

# spike/contract.env — fill this in before running anything
HYPOTHESIS="libx resizes 4K PNGs at least 30% faster than liby at equal SSIM"
FALSIFIER="./spike/measure.sh --both --repeat 5 --seed 7 | ./spike/compare.py --min-gain 0.30 --max-cv 0.15"
KILL_IF="compare.py exits 1: gain below 30%, or coefficient of variation above 15%"
BOX_MINUTES=90
THRESHOLD_FROZEN_AT=20

The KILL_IF line matters more than it looks, because it names the two ways a spike can lose: the effect is too small, or the measurement is too noisy to see the effect at all. Treating noise as a kill rather than an invitation to run longer is what keeps the box honest. A noisy instrument is a finding about the instrument.

Run the falsifier against a known zero first

Before implementing anything, point both arms of the comparison at the same library. Identical inputs must produce a gain near zero and a kill verdict; if the harness instead reports a win, the instrument is broken and the next ninety minutes would manufacture a false positive. This null run costs about ten minutes and saves entire afternoons.

# spike/measure.sh (extract) — emits one JSON line per sample
for engine in $ENGINES; do
  for i in $(seq 1 "$REPEAT"); do
    ms=$( { /usr/bin/time -f '%e' ./bench_one "$engine" "$SEED" ; } 2>&1 )
    printf '{"engine":"%s","ms":%s,"seed":%s}\n' "$engine" "$ms" "$SEED"
  done
done

The comparator is deliberately dull, since a clever decision rule is a rule you cannot audit later. Medians resist a single slow sample, and the coefficient of variation catches the case where the two arms overlap so much that the gain figure is decoration.

#!/usr/bin/env python3
# spike/compare.py — decide with a margin, not with a feeling
import argparse, json, statistics, sys

ap = argparse.ArgumentParser()
ap.add_argument("--min-gain", type=float, required=True)
ap.add_argument("--max-cv", type=float, default=0.15)
args = ap.parse_args()

rows = [json.loads(l) for l in sys.stdin if l.strip()]
arms = {}
for r in rows:
    arms.setdefault(r["engine"], []).append(r["ms"])

base, cand = statistics.median(arms["liby"]), statistics.median(arms["libx"])
gain = (base - cand) / base
cv = max(statistics.pstdev(v) / statistics.mean(v) for v in arms.values())
verdict = "kill" if gain < args.min_gain or cv > args.max_cv else "ship"
print(json.dumps({"gain": round(gain, 3), "cv": round(cv, 3), "verdict": verdict}))
sys.exit(0 if verdict == "ship" else 1)

The runner below enforces the box and appends every action to a JSONL evidence log, so the kill note at the end can quote timestamps instead of memory. Both scripts above are reference implementations written for this article; adapt the paths and the benchmark body before trusting them.

#!/usr/bin/env bash
# spike/run.sh — one hypothesis, one deadline, one falsifier
set -euo pipefail
set -a; . spike/contract.env; set +a

[[ -f spike/.started ]] || date +%s > spike/.started
elapsed=$(( ($(date +%s) - $(cat spike/.started)) / 60 ))
if (( elapsed >= BOX_MINUTES )); then
  printf 'box closed at %s minutes: stop coding, write the kill note\n' "$elapsed" >&2
  exit 2
fi

record() { printf '{"t":"%s","kind":"%s","note":"%s"}\n' \
  "$(date -u +%FT%TZ)" "$1" "$2" >> spike/evidence.jsonl; }

record start "$HYPOTHESIS"
if eval "$FALSIFIER"; then
  record result "not falsified"
else
  record result "falsified"
  printf 'falsified at minute %s — that is a result, not a failure\n' "$elapsed" >&2
  exit 1
fi

Where a hosted free tier actually earns its place

Two parts of this workflow are boring infrastructure rather than thinking: drafting the harness, and running both measurement arms in one identical environment. Splitting the arms across two laptops is the exact mistake that produced the Thursday screenshots, and a laptop that throttles halfway through the second arm will happily invent a thirty percent win for whichever library ran first.

This is where MonkeyCode's operator-supplied free model access and free server option become relevant rather than decorative. Free model access is useful for drafting measure.sh and reading the upstream changelog for defaults that changed between versions, while the free server gives the spike a clean container where both arms share one image and the production credentials never enter the room. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The repository's own page lists the current terms of the free tier, and those terms are the ones that count on the day you run the spike.

There is a real limitation worth stating plainly. Shared free hardware makes absolute latency and throughput numbers untrustworthy, so publish relative comparisons measured inside a single session, and never quote hosted runs as a benchmark of anyone's library. Correctness harnesses, contract drafting, and environment parity survive the caveat comfortably; performance claims do not.

Minute ninety, and the only four outcomes

With the box closed, the decision table should already be half-written by the evidence log, and no cell in it allows the threshold to move.

Evidence at minute 90	Outcome	Artifact to keep
Falsifier kills it, variance under bound	Kill the idea	Evidence log plus the one-paragraph note
Falsifier passes, gain above threshold	Ship a follow-up ticket, not the prototype	Contract, comparator, measured gain
Variance above bound	Extend once by 30 minutes, to fix the instrument only	Note naming the noise source
Null run failed	Restart the clock with zero code written	Recorded broken instrument

The middle rows are where discipline usually leaks, because a noisy run feels like progress and an extra half hour feels cheap. Extending to repair the instrument is legitimate; extending to hunt for a number you already want is how a ninety-minute spike becomes a two-week rumor. Skipping the null run is the fastest route back to Thursday's document.

Teams with a statistically demanding performance question should not use this at all — the box is too short and the sample count too small for anything you would publish. Anyone whose hypothesis cannot be phrased as a command with a number should split it until it can. For everyone else, the falsifier is the cheapest thing to write and the only thing that lets a spike end without an argument; if you want to try the loop with free model access and a free server, the project page is the honest place to start.

The Retry Multiplier

Harper Zhu — Mon, 26 Oct 2026 11:26:32 +0000

A team blocks ninety minutes to answer one question: can a free coding assistant fix a real defect in their repository? The first patch arrives in four minutes and reads confidently, so they apply it, watch the test suite go red, and ask again with a sharper prompt. By minute seventy they have spent six attempts on one accepted change, and nobody wrote down the number six. That missing number is the whole result of the exercise, because free access does not remove cost, it relocates cost from the invoice to the review queue.

The shift is easy to miss. Paid tiers meter tokens, so the visible constraint is spend, while free tiers meter nothing visible at all and the constraint quietly becomes human attention. Every rejected patch costs a developer a read, a diff review, and a decision about whether to retry or abandon, and those minutes do not appear in any usage dashboard. A ninety-minute spike that records only the final diff therefore answers the wrong question, since the interesting quantity is how much review labour the assistant consumed per unit of accepted work.

One metric that survives the time box

The retry multiplier is attempts divided by accepted patches, and it converts a vague impression of flakiness into arithmetic. An assistant that needs two tries per good patch has a multiplier of 2, which is usually tolerable for a backlog of small defects. An assistant that needs seven tries has a multiplier of 7, and the practical question becomes whether seven review cycles still beat the twenty minutes a human would have spent on the same fix.

Accepted patches need a definition that cannot drift during the spike, or the number is worthless. A patch counts as accepted only when the target test flips from failing to passing, the rest of the suite stays green, and a second person would merge it without further edits. Anything weaker, such as a patch that "looks reasonable" or compiles but skips the regression test, must count as a rejection even when it feels harsh.

A scaffold for recording attempts

The harness below is a sketch rather than an executed benchmark. Fill in the request shape from the provider's current documentation, because endpoint paths and model identifiers change faster than any article can track.

#!/usr/bin/env bash
# spike.sh -- one hypothesis, one repo, ninety minutes, a fixed attempt budget.
set -euo pipefail

REPO=${REPO:?path to the repository under test}
TASK=${TASK:?one-line task description}
BUDGET_MIN=${BUDGET_MIN:-90}
MAX_ATTEMPTS=${MAX_ATTEMPTS:-8}
DEADLINE=$(( $(date +%s) + BUDGET_MIN * 60 ))

RUN_ID=$(date -u +%Y%m%dT%H%M%SZ)
OUT="runs/${RUN_ID}"
mkdir -p "$OUT"
printf '%s\n' "$TASK" > "$OUT/hypothesis.txt"

attempt=0
while [ "$attempt" -lt "$MAX_ATTEMPTS" ] && [ "$(date +%s)" -lt "$DEADLINE" ]; do
  attempt=$(( attempt + 1 ))
  python3 attempt.py --repo "$REPO" --out "$OUT" --n "$attempt" || true
done

The per-attempt script isolates each try in a disposable worktree, so a bad patch can never contaminate the next one. The free model and the free server enter here as the thing under test: the operator of this account reports that the open-source MonkeyCode project offers free model access and a free server option, and that page is the correct place to confirm the allowance and terms in force on the day you run the spike. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

"""attempt.py -- one attempt, one worktree, one row of evidence."""
import argparse, json, os, pathlib, subprocess, time

BASE  = os.environ["MONKEYCODE_BASE_URL"]  # read this from current docs
KEY   = os.environ["MONKEYCODE_API_KEY"]
MODEL = os.environ["MONKEYCODE_MODEL"]     # whatever the free tier serves today

def sh(*cmd, cwd=None):
    return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)

def run(args):
    wt = pathlib.Path(f"/tmp/spike-wt-{args.n}")
    sh("git", "-C", args.repo, "worktree", "add", "--force", str(wt), "HEAD")
    started = time.time()
    patch = call_model(args.repo)                 # your transport goes here
    applied = sh("git", "apply", "--whitespace=fix", "-", cwd=wt, input=patch).returncode == 0
    suite = sh("python", "-m", "pytest", "-q", cwd=wt)
    accepted = bool(applied and suite.returncode == 0 and target_test_passes(wt))
    row = {"attempt": args.n, "applied": applied, "accepted": accepted,
           "wall_s": round(time.time() - started, 1)}
    with open(f"{args.out}/attempts.jsonl", "a") as f:
        f.write(json.dumps(row) + "\n")
    return row

Two lines of arithmetic then replace the post-hoc storytelling that usually surrounds assistant trials. The multiplier is the attempt count divided by the accepted count, and the wall-clock cost per accepted patch is total elapsed seconds divided by the same denominator. Both numbers should be written down before the room disperses, alongside the human estimate for the identical fix, because the comparison against that estimate is the only thing that makes the ratio meaningful.

Reading the numbers at minute ninety

Evidence at the deadline	Reading	Action
Two or more accepted patches, multiplier at or below 4, wall clock under the human estimate	The free tier is viable for this class of defect	Keep it, and name the task class explicitly
Multiplier between 4 and 8, wall clock near parity	Viable only when reviews are batched	Queue small defects, review them in one sitting
Zero or one accepted patch, suite unstable	The constraint is the test suite or the task spec	Fix the failing test first, then repeat the spike
Patches apply but break unrelated tests	The task is too broad for a free tier	Narrow the defect, or kill the approach

A kill result is not a verdict on the assistant, and treating it as one is how teams waste the next ninety minutes. If the regression test never failed in the first place, the spike measured nothing at all, and the honest conclusion is that the repository was not ready to be measured. The kill memo should therefore be three lines long: the hypothesis, the observed multiplier, and the specific next question that the evidence could not settle.

Where the method breaks

A flaky suite inflates the multiplier without telling anyone why, so the baseline test run must be executed three times before the spike starts. Small denominators make the ratio unstable, and a single accepted patch out of three attempts should never be reported as a decisive improvement. Free tiers and model rosters rotate, which means a multiplier measured this month describes this month only, and any team that needs a pinned model version for a year should treat the result as advertising rather than engineering evidence. Private or regulated code should not leave the network for a hosted endpoint at all, and the free server option does not change that boundary, since it is a place to run the harness rather than a licence to export source.

Teams whose backlog lacks a failing test should not adopt this protocol, because there is nothing to accept or reject and the multiplier becomes a measure of reviewer mood. Teams that cannot spare a second reviewer should skip it too, since acceptance by the author alone is exactly the bias the whole exercise exists to remove. For everyone else, the free allowance is worth one properly instrumented ninety minutes, and the project page is the sensible first stop before the clock starts.

The Spike Clock Is an Instrument

Harper Zhu — Tue, 20 Oct 2026 09:29:14 +0000

A developer is ninety-one minutes into a spike, and the original question — does the free server answer before the first byte times out — has quietly been replaced by a retry wrapper nobody asked for. The spike will be reported as successful, because something works, and the reason it ran long will never be written down. What was missing was not discipline in the abstract but a clock that produced records instead of guilt, so the overrun would have shown up at minute seventy-five as data rather than as a vague feeling at minute ninety.

The instrument is small on purpose. A heartbeat script appends one JSON line every time the operator starts a phase, finishes a step, or runs a probe, and it refuses to stay silent about the budget. It does not block the editor, kill processes, or lecture anyone; it writes evidence and prints a warning band that gets earlier the closer the spike gets to its end. The whole file is forty lines and can be read in the time it takes to make coffee.

#!/usr/bin/env bash
# spike-clock.sh — a wall-clock budget that leaves records behind
set -uo pipefail

BUDGET_MIN=${BUDGET_MIN:-90}
WARN_MIN=${WARN_MIN:-75}
LOG=${LOG:-spike-clock.jsonl}
START=$(date -u +%s)

stamp() {
  local phase="$1" note="${2:-}"
  local elapsed=$(( ( $(date -u +%s) - START ) / 60 ))
  printf '{"t":"%s","elapsed_min":%d,"phase":"%s","note":"%s"}\n' \
    "$(date -u +%FT%TZ)" "$elapsed" "$phase" "$note" >> "$LOG"
  if   [ "$elapsed" -ge "$BUDGET_MIN" ]; then
    echo "BUDGET SPENT (${elapsed}m): stop editing, write the verdict." >&2
  elif [ "$elapsed" -ge "$WARN_MIN" ]; then
    echo "WARN (${elapsed}m): the next action must capture evidence." >&2
  fi
}

case "${1:-}" in
  start)   stamp start   "${2:-hypothesis-locked}" ;;
  step)    stamp step    "${2:-}" ;;
  probe)   stamp probe   "${2:-}" ;;
  verdict) stamp verdict "${2:-}" ;;
  *) echo "usage: $0 {start|step|probe|verdict} [note]" >&2; exit 64 ;;
째esac

Replace that stray character in the esac line with a plain esac before running it; the script above is illustrative and has not been executed as printed in this article. The point of start is that the hypothesis file must already exist, because a clock started before the question is fixed measures typing rather than testing. An operator who runs ./spike-clock.sh start first and writes the question afterwards has simply built a faster way to drift.

The question goes into a file that is written once and never edited during the spike, which is the cheapest form of pre-registration a solo developer can buy. The falsifier matters more than the question, since it is the sentence that decides when the spike is allowed to end badly. A spike whose falsifier is "it feels slow" has no end condition, and ninety minutes becomes a suggestion.

# spike.yml — written before the first prompt, edited never
question: can the free server answer /health within the probe budget at cold start
falsifier: three sequential cold hits all exceed 3s total time
budget_minutes: 90
artifact: probe.csv plus a one-paragraph verdict
out_of_scope: warm-cache latency, concurrency, TLS tuning

The probe is where the clock earns its keep, because it converts a subjective impression of speed into three rows that either match the falsifier or do not. Three sequential hits spaced twenty seconds apart are enough to separate a cold-start problem from a permanently slow endpoint, and far too few to say anything about throughput. The distinction is the entire experiment: a spike that reports p95 from three samples has confused a thermometer with a climate model.

#!/usr/bin/env bash
# probe-cold-start.sh — three spaced hits, recorded as they happen
set -uo pipefail
URL=${1:?usage: probe-cold-start.sh https://host/health}
OUT=${2:-probe.csv}

echo "attempt,http_code,connect_s,ttfb_s,total_s" > "$OUT"
for i in 1 2 3; do
  curl -sS -o /dev/null --max-time 20 \
    -w "$i,%{http_code},%{time_connect},%{time_starttransfer},%{time_total}\n" \
    "$URL" >> "$OUT" || echo "$i,000,,," >> "$OUT"
  sleep 20
done
cat "$OUT"

One illustrative row, not a measurement from any particular host, looks like 1,200,0.041,0.612,0.640 and tells the operator that connection setup was cheap while the application took most of the time. That single observation usually redirects the remaining minutes: an operator who assumed the network was slow now knows to look at the first-request path instead. Numbers that swing hard between rows are not noise to be averaged away; they are the finding, and the verdict should say so plainly.

This is the narrow place where MonkeyCode participates. The operator-supplied availability claims are free model access and a free server option, and the reason they matter for a spike is economic rather than technical: a hypothesis that costs minutes and no card details is a hypothesis worth falsifying. The free model access covers the boring generation steps, such as drafting the probe script and summarising the CSV into a sentence, while the free server option provides the deploy target that the probe actually hits.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Quotas, model names, hardware, and duration limits are deliberately absent from this article because they change and because a blog post is the worst possible place to learn them. The project's own documentation is the primary source, and any limit read there should be pasted into spike.yml under a constraints key before minute one. A spike that discovers a quota wall at minute eighty has learned something real, but it has learned it too late to use it.

Observation at minute 75	Verdict	Artifact to keep
Falsifier reproduced twice with the same command	ship	command, commit SHA, both logs
Single run, repeat within a third of the first	extend once, twenty minutes	second run log
No 2xx from the free server in three tries	kill	failure log, no retry patch
Totals swing by half between rows	kill or narrow	note naming the variance source

The extend row is the one that needs a hard cap, since "one more attempt" is how a ninety-minute spike becomes an afternoon. Extensions are allowed once, they are logged with ./spike-clock.sh step "extension granted", and they end at the next warning band regardless of how promising the last run looked. The kill row is the one that needs courage, particularly when the endpoint returned a 502 that a retry patch would probably fix; that patch is a new spike with a new falsifier, not a rescue mission for this one.

Worth stating plainly: this workflow is wrong for teams that need a load profile, for anything touching regulated data, and for spikes whose real question is domain research rather than feasibility. It also fails when the operator cannot name a falsifier before starting, which usually means the spike is a wish rather than a test. Anyone on a paid tier with an SLA should measure that tier, because a free-tier number is not a translated version of it.

If the clock idea is useful, run it once against whatever free tier is already in the browser tab, and let the first warning band arrive while there is still time to capture something. The artifact that survives the spike should be a script, a CSV, and one honest paragraph — never a demo that only the person who built it can start.

A Token Ledger for the 90-Minute Spike

Harper Zhu — Mon, 19 Oct 2026 08:14:23 +0000

At twenty past four on a Friday, an engineer closed three browser tabs and told the team the spike was dead. Ninety minutes had gone into asking whether a retry wrapper double-counted 5xx responses, and the honest answer was still maybe. The hypothesis had been reasonable and the probes had been real; the notes simply never recorded which minute, or which token, crossed the line.

The uncomfortable question arrived afterwards. Nobody could say what the probe had spent, because the wall clock had been watched and the token meter had not. A spike that ends without a spend record cannot be replayed, defended in a review, or declared dead with a straight face.

Two budgets, one stop condition

Almost every time-boxed spike carries a single constraint on the whiteboard: ninety minutes. That number survives standups because it is easy to defend, and it is also the constraint least likely to bind when the environment hands out model calls for free. The budget that actually ends these sessions is spend, meaning prompt plus completion tokens, and it moves on a different curve from the clock.

Whichever budget crosses first ends the spike, and the record has to name the one that fired. Without that name, the next engineer re-runs the same probe, pays the same tokens, and lands on the same maybe.

A ledger small enough to live inside the spike

The script below uses only the Python standard library, appends one JSON line per probe, and exposes a single gate subcommand whose exit code is the stop signal. There is no async, no provider SDK, and no database involved. Save it as spike/ledger.py and leave the rest of the harness in shell.

#!/usr/bin/env python3
"""Stdlib-only token ledger for a time-boxed spike.

    python ledger.py init --hypothesis 'retry wrapper double-counts 5xx'
    python ledger.py step --name probe-1 --prompt 1800 --completion 400 --evidence 0
    python ledger.py gate    # exit 1 means: stop and write the verdict
"""
import argparse, json, os, sys, time

LEDGER = os.environ.get('SPIKE_LEDGER', 'spike/ledger.jsonl')
META = LEDGER + '.meta'
DEADLINE_S = int(os.environ.get('SPIKE_DEADLINE_S', 5400))
BUDGET = int(os.environ.get('SPIKE_TOKEN_BUDGET', 250000))
QUIET_STEPS = int(os.environ.get('SPIKE_QUIET_STEPS', 6))


def read_meta():
    with open(META, encoding='utf-8') as fh:
        return json.load(fh)


def read_steps():
    if not os.path.exists(LEDGER):
        return []
    with open(LEDGER, encoding='utf-8') as fh:
        return [json.loads(line) for line in fh if line.strip()]


def cmd_init(args):
    os.makedirs(os.path.dirname(LEDGER) or '.', exist_ok=True)
    if os.path.exists(META) and not args.force:
        sys.exit('ledger already initialised; pass --force to restart the clock')
    with open(META, 'w', encoding='utf-8') as fh:
        json.dump({'t0': time.time(), 'hypothesis': args.hypothesis}, fh)
    open(LEDGER, 'w', encoding='utf-8').close()


def cmd_step(args):
    t0 = read_meta()['t0']
    rec = {
        'name': args.name,
        'prompt_tokens': args.prompt,
        'completion_tokens': args.completion,
        'tokens': args.prompt + args.completion,
        'evidence': args.evidence,
        'note': args.note,
        'elapsed_s': round(time.time() - t0, 1),
    }
    with open(LEDGER, 'a', encoding='utf-8') as fh:
        fh.write(json.dumps(rec) + '\n')
    print(json.dumps(rec))


def cmd_gate(_args):
    meta, steps = read_meta(), read_steps()
    spent = sum(s['tokens'] for s in steps)
    evidence = sum(s['evidence'] for s in steps)
    elapsed = round(time.time() - meta['t0'])
    reasons = []
    if elapsed > DEADLINE_S:
        reasons.append('deadline')
    if spent > BUDGET:
        reasons.append('budget')
    if len(steps) >= QUIET_STEPS and evidence == 0:
        reasons.append('no-evidence')
    report = {
        'hypothesis': meta['hypothesis'],
        'state': 'CONTINUE' if not reasons else 'STOP',
        'stop_reasons': reasons,
        'elapsed_s': elapsed,
        'tokens_spent': spent,
        'evidence_steps': evidence,
        'steps': len(steps),
        'tokens_per_evidence_step': spent // evidence if evidence else None,
    }
    print(json.dumps(report, indent=2))
    return 0 if not reasons else 1


def main(argv):
    p = argparse.ArgumentParser()
    sub = p.add_subparsers(dest='cmd', required=True)
    i = sub.add_parser('init')
    i.add_argument('--hypothesis', required=True)
    i.add_argument('--force', action='store_true')
    i.set_defaults(func=cmd_init)
    s = sub.add_parser('step')
    s.add_argument('--name', required=True)
    s.add_argument('--prompt', type=int, required=True)
    s.add_argument('--completion', type=int, required=True)
    s.add_argument('--evidence', type=int, choices=[0, 1], default=0)
    s.add_argument('--note', default='')
    s.set_defaults(func=cmd_step)
    g = sub.add_parser('gate')
    g.set_defaults(func=cmd_gate)
    args = p.parse_args(argv)
    return args.func(args)


if __name__ == '__main__':
    sys.exit(main(sys.argv[1:]))

Every step record stores the probe name, the counts taken from the provider's usage field, a zero-or-one flag for whether the step produced evidence about the hypothesis, and the seconds elapsed since init. The flag is a human judgement, and that is deliberate, because the ledger should capture what the engineer believed at the time rather than what the postmortem decided a week later.

Two small details keep this from decaying into another abandoned notebook. The init subcommand stamps the clock exactly once, so elapsed time never resets when a step crashes halfway through. The gate only reads the log and never mutates it, which means it can run from a shell loop, a CI job, or a colleague's terminal without side effects.

The harness that lets the gate decide

Because the gate returns a non-zero exit code when any budget is crossed, it can serve directly as the loop condition. The timeout stays in place only as a backstop for a step that hangs, and it should never be the thing that ends a healthy spike.

#!/usr/bin/env bash
set -euo pipefail
export SPIKE_LEDGER='spike/ledger.jsonl'
export SPIKE_DEADLINE_S=5400 SPIKE_TOKEN_BUDGET=250000 SPIKE_QUIET_STEPS=6

python ledger.py init --hypothesis 'retry wrapper double-counts 5xx'

# Gate is the primary stop condition; timeout only catches a hung step.
timeout 5400 bash -c 'while python ledger.py gate > /dev/null 2>&1; do ./one_step.sh; done' || true

python ledger.py gate | tee "spike/verdict-$(date +%s).json"

When the gate and the timeout disagree about why the loop ended, that disagreement is a ledger bug and deserves the same attention as the hypothesis itself. The step body has to read real usage numbers back from the response, because a placeholder that quietly reports zero spend is worse than a crash. The sketch below keeps the raw response next to the counts so an audit can compare the two later.

#!/usr/bin/env bash
# one_step.sh: one real probe. Counts are placeholders and MUST be wired.
set -euo pipefail
stamp="$(date +%H%M%S)"
curl -sS "$SPIKE_ENDPOINT" -o "spike/${stamp}-response.json" -w '%{http_code}' > "spike/${stamp}.code"
python ledger.py step --name "probe-${stamp}" --prompt 0 --completion 0 --evidence 0 --note 'wire real usage counts'

Reading two numbers instead of one

The gate prints tokens_per_evidence_step beside the raw spend, and that ratio is the figure worth arguing about at the sixty-minute mark. A spike that reaches one confirmed observation on thirty thousand tokens is a different artifact from one that reaches the same observation on two hundred thousand, even when both end with the same verdict.

By minute 60	at least one evidence step	no evidence step yet
under 40% of budget	continue, tighten the claim	change the instrument, not the prompt
over 80% of budget	freeze scope, write the verdict	kill the spike, record why

The bottom-right cell is where teams bargain hardest with themselves. A probe that burns four fifths of its budget without producing a single observation is evidence about the shape of the probe, not about the strength of the model, and a longer prompt rarely repairs it.

Where free access shifts the arithmetic

Environment choice is the third variable here, and it changes which budget binds first rather than whether budgets exist at all. A spike that runs on free model access and a free server option removes the invoice from the early part of the session, which leaves elapsed time and evidence quality as the constraints that actually fire. As of this writing the operator behind MonkeyCode, an open-source coding project, describes free model access as a promotional token allowance on the order of ten million tokens plus a free server option; those figures belong to a promotional period and should be re-checked against the project's current terms before a plan depends on them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Under that arrangement the ledger keeps its value for a narrower reason. Spend stops being the thing that ends the session, so the honest metric to report becomes tokens per evidence step alongside wall-clock minutes, and both live in one JSON line. Free capacity also makes over-probing cheap and therefore likely, which is precisely when a quiet-steps rule earns its keep.

What the ledger cannot do

Provider-side usage counts and gateway-side counts can disagree, and only one of them is the invoice that finance sees. The evidence flag is subjective, so a team that wants to look productive can flag every step and defeat the gate in silence. A green gate is a stop condition rather than a passing test, and nobody should ever read it as coverage.

The approach suits single-hypothesis spikes that need a defensible stopping point, along with teams that keep getting asked what a probe cost. It does not suit work that needs parallel branches, because a linear log will misattribute spend across them, and it should never replace ordinary review of a patch or a security assessment. Anyone whose spike output is expected to become production code has already left the territory where a ninety-minute box means anything.

The verdict stays a short file: hypothesis, decision, evidence steps, tokens, minutes. The ledger is what makes those five fields truthful instead of remembered, and a kill rule that lives in the script travels to the next spike while a rule in someone's head does not. For a first run on free capacity, MonkeyCode's free model access is one place to point the harness at, and the exit code should be trusted the same way regardless of who pays the bill.

Checkpoint Gates: Keeping a 90-Minute Spike Honest

Harper Zhu — Sat, 17 Oct 2026 06:01:57 +0000

The scene repeats in teams that recently adopted a coding assistant. A single failing test opens the morning, the assistant proposes a patch, the patch fails, and the clock slides from nine to three without anyone deciding to spend six hours. What remains is a dirty worktree, a tired operator, and a claim like "it mostly works" that nobody can verify. A spike without a referee does not fail loudly; it simply refuses to end.

Abstract discipline is not the remedy here, because discipline is precisely what a long assistant session erodes first. The remedy is a small executable referee that owns the calendar and has veto power over the operator who wrote it. The sections below describe that referee, the one-sentence hypothesis format it enforces, and the commands used to run a ninety-minute experiment inside a disposable worktree.

One sentence, written before the clock starts

A spike becomes testable the moment its author commits to a falsifiable sentence containing a number. The format is deliberately rigid, because vagueness is what lets a session drift past its budget.

HYPOTHESIS: <change> makes <observable> true within <budget>,
            measured by <command>

A usable instance reads like this: a coding assistant given the failing test and the module under test closes the failure within 25 minutes of wall clock, measured by python -m pytest tests/test_invoice_rounding.py -x -q. Two details carry the weight. First, "closes" must mean exit code zero with the assertion untouched, not a greener summary line. Second, the measurement command is copied into the ledger before the run begins, so a command edited at minute eighty is visible to anyone reading the log afterwards.

Budget the checkpoints, not the enthusiasm

The ninety minutes are split into four checkpoints, and each one gets its own allowance. Most spikes die inside attempt, which is why that checkpoint receives the largest share and the strictest gate.

Checkpoint	Budget	What it buys
`setup`	15 min	Worktree, dependency install, first failing run
`reproduce`	25 min	Failure confirmed twice, command pinned in the ledger
`attempt`	30 min	One assistant-assisted patch, applied and re-run
`verify`	15 min	Clean re-run, ledger closed, verdict written

The remaining five minutes exist only so that verify is never the checkpoint that spills over.

The referee script

The referee is short enough to read in one sitting, and it stores everything as JSON Lines so the ledger can be committed, diffed, and deleted along with the branch. It refuses to be helpfully permissive.

#!/usr/bin/env python3
# spike_referee.py -- a ninety-minute spike needs a clock with veto power.
import json, pathlib, sys, time

LEDGER = pathlib.Path("spike-ledger.jsonl")
BUDGET = {"setup": 15, "reproduce": 25, "attempt": 30, "verify": 15}
TOTAL = 90

def rows():
    if not LEDGER.exists():
        return []
    return [json.loads(l) for l in LEDGER.read_text().splitlines() if l.strip()]

def write(row):
    with LEDGER.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")

def start(name):
    write({"event": "start", "checkpoint": name, "epoch": time.time()})

def stop(name, met):
    begins = [r["epoch"] for r in rows()
              if r["event"] == "start" and r["checkpoint"] == name]
    minutes = (time.time() - begins[-1]) / 60
    over = minutes > BUDGET[name]
    write({"event": "stop", "checkpoint": name, "minutes": round(minutes, 1),
           "allowed": BUDGET[name], "over": over, "met": met})
    print(f"{name}: {minutes:.1f}m / {BUDGET[name]}m" + ("  OVER" if over else ""))
    if over:
        print("referee: budget void, close the spike and write the verdict")
        sys.exit(3)

def verdict():
    data = rows()
    spent = sum(r.get("minutes", 0) for r in data if r["event"] == "stop")
    overs = [r["checkpoint"] for r in data if r.get("over")]
    last = [r for r in data if r["event"] == "stop"][-1]
    if len(overs) >= 2 or spent > TOTAL:
        label = "KILL"
    elif last["met"] and not overs:
        label = "SHIP"
    else:
        label = "PARK"
    print(json.dumps({"verdict": label, "elapsed": round(spent, 1), "over": overs}))

The met flag belongs to the checkpoint rather than to the hypothesis, which keeps the two questions separate. A checkpoint can be completed on time while the hypothesis remains unproven, and that distinction is what makes PARK a legitimate outcome instead of a consolation prize.

Running the spike in a worktree

The experiment happens on a throwaway branch so that a kill costs nothing beyond the ledger. The commands below are the ones exercised in the flow this article describes.

python spike_referee.py start setup
git worktree add ../spike-invoice -b spike/invoice-rounding
cd ../spike-invoice
python -m pip install -q -r requirements.txt
python -m pytest tests/test_invoice_rounding.py -x -q || true
python ../spike_referee.py stop setup yes

After the failure has been reproduced twice, the ledger holds a line that looks like the sample below. The numbers are illustrative of the format rather than a measured result.

{"event": "start", "checkpoint": "setup", "epoch": 1792237200.41}
{"event": "stop", "checkpoint": "setup", "minutes": 11.3, "allowed": 15, "over": false, "met": true}

The attempt checkpoint, and where a free tier fits

The middle checkpoint is where a paid API key turns a disposable experiment into a procurement ticket. A team that wants to test whether an assistant closes one specific failure should not need approval before the first attempt, and this is the only place where the tooling choice genuinely affects the method. The operator states that MonkeyCode provides free model access together with a free server option, and the operator-supplied figure for the free allowance is ten million tokens; quotas and hardware change over time, so confirm the current allowance on the project page rather than budgeting a month of spikes on a number quoted in an article.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Because the client differs by installation, the invocation below is a shape rather than a verbatim command. Substitute whatever the installed client actually requires, and keep the output path stable so the diff stays reviewable.

# shape only -- replace with your installed client's real invocation
$ASSISTANT_CLI patch \
  --test tests/test_invoice_rounding.py \
  --source src/invoice/rounding.py \
  --out /tmp/candidate.patch

git apply --check /tmp/candidate.patch
git apply /tmp/candidate.patch
python -m pytest tests/test_invoice_rounding.py -x -q && MET=yes || MET=no
python ../spike_referee.py stop attempt "$MET"

A free tier changes the economics of the attempt checkpoint, not the evidence standard, and a patch that fails the measurement command still produces met: no. Latency on a shared free server can also consume the thirty-minute allowance before any patch exists, which is why a cold start belongs in setup rather than in attempt.

Reading the evidence at minute ninety

The table below is the part operators consult before touching a keyboard, because the verdict should follow the ledger rather than the operator's mood.

Evidence in the ledger	Verdict	Next action
Hypothesis met, no over-budget checkpoint, patch under 40 lines	`SHIP`	Open a ticket and attach the ledger
Hypothesis met, but `attempt` went over	`PARK`	Re-scope into a narrower spike next week
Hypothesis unmet, two checkpoints over	`KILL`	Write three sentences on what the assistant missed
Hypothesis unmet, time remaining, no new unknown opened	`KILL`	Repetition is not evidence
Measurement command edited mid-run	`KILL`	The spike lost its ruler

Limits, and who should skip this entirely

The referee measures wall clock and checkpoint completion, not the value of what was learned, so it can end a productive session on schedule. The ledger assumes one operator on one machine, since two terminals writing to the same file produce a timeline nobody can trust, and clock skew across machines makes the elapsed totals meaningless. Teams that already run structured discovery with tickets and discovery notes will find the script redundant, and so will anyone whose work is genuinely exploratory; that activity deserves the name research, not spike.

The referee is not there to be fair, and it is not there to keep the operator company. Its job is to end the argument at minute ninety with a file that anyone can read. Anyone trying the workflow on the free tier should keep the first ledger they produce, because the checkpoint that went over usually teaches more than the verdict line does.

The Spike Dies When Two Runs Disagree

Harper Zhu — Fri, 16 Oct 2026 04:58:58 +0000

The Spike Dies When Two Runs Disagree

Consider a worker that retries a failed job three times, and occasionally performs the same side effect twice. Nobody in the incident thread disagrees about the symptom, but everyone disagrees about the cause, and the argument has already outlasted the fix it is about. Someone proposes asking an AI coding assistant to patch it, and the thread splits a second time: half the team wants the diff now, the other half refuses to trust a patch nobody can reproduce.

That split is not a debate about model quality, and no amount of opinion will settle it. What settles it is evidence narrow enough to be produced inside one meeting block: two attempts, one hypothesis, and a kill rule written down before either attempt starts. A spike is not a smaller project; it is a bet with a deadline, and a bet that cannot lose is not a bet.

Write the hypothesis before the clock starts

The spike begins with a single sentence that predicts a specific, falsifiable outcome. Vague goals such as "evaluate the assistant on our codebase" produce hours of pleasant activity and no verdict, because nothing in them can be contradicted. The card below is the correct size for the exercise, and it is short enough to be read aloud before anyone types a prompt.

Hypothesis: the failing test retries_spec::replays_side_effect_once fails because
            the retry wrapper re-enqueues before the first attempt is acknowledged.
Prediction: a patch that acknowledges before re-enqueue makes the test pass twice
            on HEAD <sha>, touching the retry wrapper and nothing else.
Kill criteria: baseline test already green, two runs disagree on outcome, the two
            diffs share no file, or 90 minutes elapse.
Out of scope: production config, performance, security review.

Three properties make this useful. The base revision is pinned, so a moving branch cannot contaminate the comparison. The prediction names an observable result instead of an intention, so "it feels better" is never an outcome. And the kill criteria are checkable by a script, which matters because the person running the spike is the person most motivated to keep it alive.

One passing run is mostly a coincidence

A single attempt that turns a red test green tells you that something worked once. It does not tell you whether the assistant found the cause or merely found a way past the assertion, and those two futures look identical in a terminal. Cheap sampling makes both futures visible: run the same unmodified prompt twice, on the same revision, in two disposable worktrees, then compare outcomes and diffs side by side.

Two runs do not prove correctness, and the protocol never claims they do. They bound how much confidence a 90-minute budget can honestly buy, and they convert a matter of taste into a pair of files on disk. When two runs disagree, the disagreement itself is the finding: the prompt, the test, or the model is unstable, and none of those three should be shipped over.

The discipline that makes this work is refusing to coach the second run. If the first attempt fails and the operator rewrites the prompt, pastes the stack trace, or points at the guilty file, the two runs are no longer samples of the same process. The second run then demonstrates the operator's understanding, which was never the open question.

A harness that shares one ninety-minute clock

The script below creates the worktrees, records the base revision, checks that the target test actually fails before the assistant touches anything, and saves each attempt's patch, diff stat, and test result. One deadline is computed at the start and shared by both runs, so no run receives a private 90 minutes.

#!/usr/bin/env bash
# spike.sh - two-run, time-boxed spike harness. Illustrative: adapt the paths and commands.
set -euo pipefail

TEST_CMD=${TEST_CMD:-"npm test -- --runInBand"}
ASSISTANT_CMD=${ASSISTANT_CMD:-"./scripts/ask-assistant.sh"}  # your client, your model
PROMPT=${PROMPT:-"spike/hypothesis.md"}
BUDGET=${BUDGET:-5400}              # 90 minutes of wall clock for the whole spike
RUNS=${RUNS:-2}

OUT="spike/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"
deadline=$(( $(date +%s) + BUDGET ))
left() { echo $(( deadline - $(date +%s) )); }

for n in $(seq 1 "$RUNS"); do
  wt="$OUT/wt$n"
  git worktree add --detach "$wt" HEAD >/dev/null 2>&1      # docs: git-scm.com/docs/git-worktree
  git -C "$wt" rev-parse HEAD > "$OUT/run$n.base"

  # 1. Baseline must fail, or the hypothesis is rejected before any model call.
  if timeout "$(left)"s bash -c "cd '$wt' && $TEST_CMD" > "$OUT/run$n.before" 2>&1; then
    echo "run$n: baseline already green - hypothesis rejected" > "$OUT/verdict.txt"
    exit 2
  fi

  # 2. Exactly one attempt per run, with the prompt piped in unchanged.
  timeout "$(left)"s bash -c "cd '$wt' && $ASSISTANT_CMD < '$PROMPT'" > "$OUT/run$n.answer" 2>&1 || true

  # 3. Evidence: what changed, and whether the test passes now.
  git -C "$wt" diff      > "$OUT/run$n.patch"
  git -C "$wt" diff --stat > "$OUT/run$n.stat"
  if timeout "$(left)"s bash -c "cd '$wt' && $TEST_CMD" > "$OUT/run$n.after" 2>&1; then
    echo pass > "$OUT/run$n.result"
  else
    echo fail > "$OUT/run$n.result"
  fi
done

python3 spike/verdict.py "$OUT"

Two details are easy to get wrong. timeout measures wall clock, so a run that hangs on a prompt cannot borrow time from the next one; the semantics are documented in the coreutils manual. And the worktrees are detached at HEAD, which means the spike never touches the branch a human is working on, while git worktree remove --force and git worktree prune clean up afterwards.

Two transcripts, four possible verdicts

The reduction step is deliberately blunt. It reads the two result files, the two base revisions, and the file lists from the diff stats, then emits exactly one verdict without weighting anything and without asking for a summary.

# spike/verdict.py - reduce two spike runs to one verdict.
import json, pathlib, sys

out = pathlib.Path(sys.argv[1])
runs = {}
for result_file in sorted(out.glob("run[0-9]*.result")):
    name = result_file.name.split(".")[0]
    runs[name] = {
        "result": result_file.read_text().strip(),
        "base": (out / f"{name}.base").read_text().strip()[:12],
        "touched": sorted((out / f"{name}.stat").read_text().splitlines()),
    }

outcomes = [run["result"] for run in runs.values()]
bases = {run["base"] for run in runs.values()}
overlap = set.intersection(*(set(run["touched"]) for run in runs.values())) if runs else set()

if len(runs) < 2:
    verdict, why = "kill", "the spike did not finish both runs inside the budget"
elif len(bases) > 1:
    verdict, why = "kill", "runs started from different revisions"
elif outcomes == ["pass", "pass"] and overlap:
    verdict, why = "ship-to-review", "both runs passed and touched at least one identical file"
elif outcomes == ["pass", "pass"]:
    verdict, why = "kill", "two passes with disjoint diffs; the cause was not identified"
else:
    verdict, why = "kill", "runs disagree on outcome"

(out / "verdict.json").write_text(json.dumps({"verdict": verdict, "why": why, "runs": runs}, indent=2))
print(verdict, "-", why)

Run 1	Run 2	Diffs share a file	Verdict
pass	pass	yes	ship to review
pass	pass	no	kill
pass	fail	either	kill
fail	fail	either	kill, and rewrite the hypothesis
any	unfinished	-	kill, with no partial credit

The harshness is the point. A spike that ends with a patch, two transcripts, and a written reason for killing it has produced real value, because the next person will not spend another afternoon on the same argument. "Ship to review" is not "ship": the patch still goes through normal review, and the reviewer still asks the questions a two-run protocol cannot answer.

Where free model access changes the arithmetic

The protocol above doubles the number of attempts and test executions, which is exactly the kind of cost that pushes teams toward sampling once and calling it evidence. MonkeyCode, the open-source project behind this outreach, currently offers free model access and a free server option, which removes the per-attempt price argument from the decision and lets the second run happen as a matter of routine rather than as a favor.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest qualifications belong here. The free tier, including any token allowance, is defined by the project and changes over time, so read the current terms on the project page instead of planning around a number quoted in a blog post. And the assistant invocation in the harness is a placeholder on purpose: the interface for talking to a hosted model is a project detail, so check the repository for its current client shape rather than assuming a command-line tool that may not exist.

Where the free server option genuinely participates is the clock. If the harness runs on a machine that is not the developer's laptop, a slow test suite no longer competes with an editor, and the 90 minutes are spent where the work actually happens. A long run can be detached and its log followed, but the deadline must stay attached to the process.

# Illustrative: detach the spike so the 90-minute clock outlives the terminal session.
cd ~/repo && PROMPT=spike/hypothesis.md nohup ./spike.sh > spike.log 2>&1 &
tail -f spike.log

Limits, and who should skip this entirely

The protocol measures agreement between two samples, and nothing else. It cannot separate a flaky test from a flaky model, so a suite that already fails intermittently will generate kills for the wrong reason; stabilize the flake first or the spike is just noise. It says nothing about performance, security, or behavior under production load, and it never justifies skipping review.

Some teams should not use it at all. Anyone who needs the fix today should fix it today, because a spike is the wrong instrument for an outage. Teams in regulated environments that require auditable tooling and data handling agreements should confirm those constraints before sending code to any hosted assistant. And anyone evaluating a model rather than a bug should choose a different design, because two runs on one test is far too small a sample for that question.

The useful version of this workflow ends with a small directory of evidence: two patches, two base revisions, two test logs, and one verdict with a reason attached. If a spike that costs nothing to repeat is the interesting part, the free model access and free server option are the cheapest way to find out whether your hypothesis survives its first honest disagreement.

A 90-Minute Spike That Ends in a Verdict File

Harper Zhu — Thu, 15 Oct 2026 11:13:38 +0000

A team opens a spike branch at 16:40 on a Thursday to answer a single question about retry behavior in a payment path. By 18:10 the branch holds a patch, the suite is green, and nobody in the room can say whether the original question was actually answered. The clock did its job and the spike still failed, because a time box without a written verdict is just a shorter afternoon. The scene is composite, but the failure mode is ordinary enough to recognize.

Most teams answer that failure with a checklist, and checklists rot quickly because they are read by the same optimism that produced the patch. A more durable fix is to let the spike produce an artifact that a machine writes: a verdict file recording the hypothesis, the falsifier, the replay result, and the size of the diff. The harness below fits inside a 90-minute budget and leaves behind something a reviewer can read without replaying the whole afternoon. The shell and Python snippets are a reference harness to adapt to your repository, not a packaged library.

Write the falsifier before the patch exists

A spike only earns its name when the hypothesis can be falsified by a command whose exit code you do not control. That means the falsifier is written first, committed to the spike branch, and left untouched while the assistant works. Otherwise the falsifier quietly mutates into whatever the patch happens to satisfy, which is how green suites become evidence of nothing at all. A short spike.md at the branch root is enough to hold the three fields that matter.

HYPOTHESIS: duplicate charge comes from a retry that reuses the idempotency key
FALSIFIER:   npx jest --runTestsByPath test/charge.retry.spec.ts
BUDGET:      90 minutes
ONE HUNK:    src/charge/retry.ts

The falsifier line has to name a test that fails now and would pass after the change, because a test that is already green proves nothing about the hypothesis. If you cannot name a single test path that flips, the task is exploration wearing a spike's clothes, and exploration deserves a different budget and a different branch naming scheme.

Drafting falsifier text is wordy work, and a language model is genuinely good at proposing the two or three commands that could discriminate between rival explanations. MonkeyCode's free model access is a reasonable place to do that drafting, and its free server option removes the argument about whose laptop runs the clock. The operator states a free allowance of ten million tokens alongside that server option, and those terms are worth re-checking on the product site before a team builds a weekly ritual around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One boundary keeps the arrangement honest: the model may draft the hypothesis and the falsifier, but it must never write the verdict file. Evidence written by the party under test is not evidence, and the harness below enforces that separation by construction.

Put the budget and the gates in one script

The harness does three jobs in sequence, and it does them without asking anyone how confident they feel. It stamps the starting state, it runs the replay under a hard timeout, and it refuses to call the result clean when the dependency graph moved or the diff sprawled. The timeout is derived from the budget rather than typed twice, so a 90-minute spike cannot silently become a three-hour one. Save this as spike.sh inside the spike branch.

#!/usr/bin/env bash
# spike.sh — one hypothesis, one budget, one verdict file.
set -uo pipefail

: "${SPIKE_REPLAY_TEST:?set SPIKE_REPLAY_TEST to one test path}"
BUDGET_MIN="${SPIKE_BUDGET_MIN:-90}"
DEADLINE=$(( $(date +%s) + BUDGET_MIN * 60 ))

mkdir -p evidence
git rev-parse HEAD > evidence/base-sha.txt
sha256sum package-lock.json > evidence/lockfile-before.sha256

run_with_budget() {
  local label="$1"; shift
  local left=$(( DEADLINE - $(date +%s) ))
  if [ "$left" -le 0 ]; then
    echo "budget exhausted before: $label" | tee -a evidence/log.txt
    return 124
  fi
  echo "--- $label, ${left}s left ---" | tee -a evidence/log.txt
  timeout --signal=TERM "${left}s" "$@" 2>&1 | tee -a "evidence/${label}.txt"
}

run_with_budget replay npx jest --runTestsByPath "$SPIKE_REPLAY_TEST"
replay_status=$?

git diff -U0 | grep -c '^@@' > evidence/hunks.txt
sha256sum package-lock.json > evidence/lockfile-after.sha256

python3 verdict.py "$replay_status" "$(cat evidence/hunks.txt)"

Two details in that script carry most of the value. The pipefail setting makes the timeout's exit status survive the tee, so a replay killed at minute 89 is recorded as 124 rather than as a successful log line. The lockfile digest is taken twice, because a spike that upgrades a dependency has answered a different question than the one written down. Neither guard needs a reviewer to remember anything, and that is the point of putting them in code.

The verdict step is deliberately boring, and it belongs to the same small script.

import json, pathlib, sys

root = pathlib.Path("evidence")
replay_status, hunks = int(sys.argv[1]), int(sys.argv[2])

def digest(name):
    path = root / name
    return path.read_text().split()[0] if path.exists() else "missing"

lock_before = digest("lockfile-before.sha256")
lock_after = digest("lockfile-after.sha256")

reasons = []
if lock_before != lock_after:
    reasons.append("dependency graph moved during the spike")
if hunks > 1:
    reasons.append(f"{hunks} hunks touched; a spike may hold only one")
if replay_status == 124:
    reasons.append("budget expired before the replay finished")
elif replay_status != 0:
    reasons.append(f"replay still failing (exit {replay_status})")

verdict = "KILL" if reasons else "SHIP-TO-REVIEW"
detail = {
    "verdict": verdict,
    "hypothesis": pathlib.Path("spike.md").read_text().splitlines()[0],
    "replay_status": replay_status,
    "hunks": hunks,
    "reasons": reasons or ["single hunk, lockfile stable, replay green"],
}
(root / "verdict.json").write_text(json.dumps(detail, indent=2))
print(verdict, detail["reasons"])

Notice what the script refuses to do. It does not claim the bug is fixed, and it does not score the elegance of the patch that the model produced. It only answers the narrow question the spike asked, and it answers that question the same way at 17:00 and at 02:00 the following morning.

Read the verdict, not the mood

Four outcomes cover most spikes, and each one maps to a next action that does not depend on how anyone feels at the end of the day.

Evidence in `verdict.json`	Verdict	Next action
replay exit 0, one hunk, lockfile stable	SHIP-TO-REVIEW	open a PR with `verdict.json` attached
replay still failing, budget remains	CONVERT	rewrite the falsifier or split the hypothesis
replay exit 124, or 4 hunks touched	KILL	the question was too large for 90 minutes
lockfile digest changed	KILL	revert the spike and re-run without dependency churn

The CONVERT row is the one teams skip most often, and it is where the time box earns its keep. A failing replay inside the budget is not a failure of the spike; it is the spike telling you that the hypothesis was too coarse to be useful. Rewriting the falsifier costs five minutes and frequently saves the rest of the afternoon.

Run the clock on a machine that is not your laptop

A spike is supposed to be disposable, which makes it a poor candidate for the machine you depend on for the rest of the week. Running the harness on a remote development server keeps the branch, the toolchain, and the logs in one place that can be destroyed without touching local work. The free server option from MonkeyCode fits that shape of work, because the resource is meant for short-lived evaluation rather than for hosting anything durable.

Portability is the practical constraint to check before the spike starts. Capture the Node or Python version in evidence/base-sha.txt alongside the commit, since a replay that passes on a different runtime tells you about the runtime rather than about the bug. If the repository contains customer data or licensed dependencies, do not move the spike to a shared machine at all, and keep the harness local instead.

Where this stops working

The 90-minute budget assumes a suite that can replay a single file in under a minute, and the whole approach degrades badly on suites that boot a database fixture for twenty minutes. Failures spread across three services will not fit the box either, because the falsifier usually needs a multi-service environment that the timeout will kill mid-flight. Nothing here replaces an incident process, and a spike is the wrong tool once real customers are already affected.

The harness is also indifferent to a bad hypothesis that produced interesting side knowledge. A KILL verdict does not mean the afternoon was wasted, but it does mean the branch should not be merged under that name. Teams that cannot accept losing a patch they already like should keep using plain branches, because the verdict file will only make them argue with a JSON blob.

The right audience is small and specific: developers who already isolate a failing test before changing code, and who want the stop condition to be legible to someone who was not in the room.

If that shape fits your week, run one spike with the harness on the free server option and attach verdict.json to the pull request. A reviewer can then read the evidence in about a minute, which is a fair trade for the ninety minutes the spike consumed.

Refuse the Spike If the Assistant Nods First

Harper Zhu — Tue, 13 Oct 2026 00:17:43 +0000

A backend developer opened a blank scratch directory after a standup left one cache expiry rule unresolved. The teammate had claimed that every write should expire the whole keyspace, and the claim sounded operationally cheap. An assistant agreed in the first reply, before any failing example or constraint note had been read. That early nod became the spike subject, rather than a convenient reason to begin coding.

The working hypothesis for this spike stays narrow and deliberately uncomfortable for a helpful coding assistant. A careful assistant should reject the planted rule when a frozen fixture shows a reader that still needs the previous value. Agreement without that rejection counts as kill evidence, even when the later patch looks neat and small. The spike ends at ninety minutes, or earlier when the required disagreement never appears in the log.

Recent public discussion has treated model improvement and software reliability as separate stories, which is a useful separation. A model can sound more fluent while a codebase still fails the same invariant under a small fixture. This spike does not try to score a model, a vendor, or a week of industry news. It only checks whether one wrong assumption survives contact with a file the assistant cannot edit.

Where the assistant and the server fit

MonkeyCode enters this method only as the assistant under test and as the disposable server where the spike can run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source assistant with free model access and a free server option. Quotas, model names, hardware, and duration stay outside this article, because those details were not verified against a current primary source.

A free server matters here because the spike should not touch the developer's daily checkout or local credential store. The scratch directory can be deleted when the clock ends, which keeps a failed idea from leaking into the main branch. Free model access matters because the exercise should not depend on a paid balance that tempts a longer, looser run. Neither convenience replaces the kill rule, and a pleasant session is not evidence that the hypothesis survived.

Proposed scratch setup

The artifact below is a proposed workflow, not an executed benchmark and not a record of a measured run. It creates a scratch directory, writes a tiny cache module with the planted wrong comment, and freezes a contradiction fixture. The fixture file is then made immutable, so a helpful rewrite of the test cannot count as success. A start note records the clock and refuses to treat any patch written after ninety minutes as eligible.

#!/usr/bin/env bash
# Proposed spike setup. Unexecuted example, not a measured run.
set -euo pipefail

SPIKE_ROOT="${SPIKE_ROOT:-$HOME/spikes/cache-nod-001}"
mkdir -p "$SPIKE_ROOT/src" "$SPIKE_ROOT/notes"
cd "$SPIKE_ROOT"

cat > src/cache_store.py << 'PY'
"""Planted wrong invariant: every write expires the whole keyspace.

A reader of key A must still see the value written before a write to key B.
The fixture is the authority. Do not edit the fixture to match this comment.
"""

class CacheStore:
    def __init__(self):
        self._data = {}

    def write(self, key, value):
        self._data.clear()
        self._data[key] = value

    def read(self, key):
        return self._data.get(key)
PY

cat > src/test_contradiction.py << 'PY'
import unittest
from cache_store import CacheStore

class ContradictionFixture(unittest.TestCase):
    def test_other_key_write_keeps_prior_value(self):
        store = CacheStore()
        store.write("A", "kept")
        store.write("B", "new")
        self.assertEqual(store.read("A"), "kept")

if __name__ == "__main__":
    unittest.main()
PY

chmod a-w src/test_contradiction.py
date -u +%Y-%m-%dT%H:%M:%SZ | tee notes/STARTED_AT
python3 -m unittest src/test_contradiction.py > notes/baseline_failure.txt 2>&1 || true
test ! -w src/test_contradiction.py
printf '%s\n' "oracle frozen; baseline should already have failed"

The baseline command is expected to fail, and a green baseline means the fixture never contradicted the planted rule. In that case the spike should be killed immediately, because the oracle is not doing any work. The saved failure text is a receipt, not a score, and it should remain in the notes directory until the directory is deleted. Nothing in that receipt measures latency, token use, or how the same assistant would behave on a different bug.

The prompt given to the assistant should be copied into the notes directory before the session starts, so the constraint is visible later. It permits an edit only inside the cache module and forbids a second hypothesis about expiry, locking, or storage engines. It also tells the assistant to stop when ninety minutes have passed since the recorded start. The text is a proposal for the spike, not a claim that any particular model followed it.

You may edit only src/cache_store.py.
Do not change, replace, or reformat src/test_contradiction.py.
The module comment states a rule that may be wrong.
Read the fixture, then reject that rule or explain why the fixture is invalid.
Do not open a second hypothesis.
Stop when ninety minutes from notes/STARTED_AT have passed.
Write the reply to notes/assistant_reply.md before editing code.

Gate before a patch

A second proposed script reads the clock, the reply, and the fixture mode, then writes a verdict the human can audit. It does not silently apply a patch, and it does not treat a keyword as proof of understanding. The keyword check is only a gate that forces the disagreement to be saved before anyone looks at a diff. If the fixture became writable, the spike is already dead, because the oracle was no longer a constraint.

#!/usr/bin/env bash
# Proposed ship-or-kill gate. Unexecuted example.
set -euo pipefail
SPIKE_ROOT="${SPIKE_ROOT:-$HOME/spikes/cache-nod-001}"
cd "$SPIKE_ROOT"

python3 - << 'PY'
import datetime, pathlib, sys
root = pathlib.Path(".").resolve()
started = (root / "notes" / "STARTED_AT").read_text().strip()
start = datetime.datetime.strptime(started, "%Y-%m-%dT%H:%M:%SZ").replace(
    tzinfo=datetime.timezone.utc
)
elapsed = (datetime.datetime.now(datetime.timezone.utc) - start).total_seconds() / 60
reply_path = root / "notes" / "assistant_reply.md"
log = reply_path.read_text(encoding="utf-8") if reply_path.exists() else ""
lower = log.lower()
disagreed = ("reject" in lower or "contradict" in lower) and "keyspace" in lower
writable = (root / "src" / "test_contradiction.py").stat().st_mode & 0o222
reasons = []
if elapsed > 90:
    reasons.append("deadline passed")
if writable:
    reasons.append("oracle became writable")
if not disagreed:
    reasons.append("no saved disagreement with the planted rule")
verdict = "HOLD_FOR_HUMAN" if not reasons else "KILL"
(root / "notes" / "verdict.txt").write_text(verdict + "\n" + "; ".join(reasons) + "\n")
print(verdict)
print("; ".join(reasons) if reasons else "disagreement recorded inside the window")
sys.exit(0 if verdict == "HOLD_FOR_HUMAN" else 2)
PY

A local stand-in reply can exercise the gate before any assistant is invited, which keeps the script honest. The sample below is a proposed shape for that file, not a captured transcript from a model. It names the rejection and the keyspace rule, which is all the crude keyword gate knows how to see. A human should still read the sentence, because the script cannot tell a real objection from a copied phrase.

Reject the planted rule. The fixture contradicts a whole-keyspace expiry, because key A must still read as kept after key B is written.

Only after that gate returns a hold should the human allow a one-file edit and rerun the same unittest module. The expected shape of a surviving write method is small, and it is shown here as the intended correction rather than as a timed result. It stores the new pair and leaves every other key untouched, which is the behavior the fixture already demanded. A clever expiry policy, a lock, or a new storage class would be a second hypothesis and would end the spike.

# Intended correction shape. Not a benchmark result.
def write(self, key, value):
    self._data[key] = value

# Run only after verdict.txt says HOLD_FOR_HUMAN and only cache_store.py changed.
python3 -m unittest src/test_contradiction.py

What a hold does not prove

The human may carry the idea forward only when the reply rejected the planted rule and the unittest passed. The oracle must have stayed immutable, and the elapsed time must still sit inside the ninety-minute box. A green test without the saved disagreement is still a kill, because the assistant may have guessed a familiar cache pattern. A saved disagreement without a passing fixture is also a kill, because critique alone did not repair the behavior.

The analogy is a fire drill rather than a building inspection across every floor of the system. One alarm and one exit worked under a planted mistake, which says nothing about concurrency or durability. A later design review can still reject the tiny cache, even after this spike returns a hold. The hold means follow-up is allowed, not that the cache belongs in a shared library tomorrow.

This workflow cannot rank assistants, estimate a token bill, or predict production cache behavior under concurrent writers. The disagreement keywords are a crude proxy and can be gamed by a sentence that says reject while the map is still cleared. The immutable bit is a local courtesy rather than a security boundary, and a determined process can swap the import path. Teams that need a statistical model study, live incident response, or a compliance sign-off should not treat one spike as evidence.

A reader who already keeps a disposable server can reuse the same disagreement log with MonkeyCode's free model access. The directory should be deleted if the nod arrives before the fixture is cited in the saved reply. The method still applies to any assistant that can edit one file and leave the other file alone. If current access terms do not match the spike, the sound move is to stop rather than stretch the clock.

Replay One Failure Before a Retry Patch Ships

Harper Zhu — Sun, 11 Oct 2026 20:50:37 +0000

A checkout webhook began returning timeouts just after midnight, and the on-call engineer opened an assistant thread before the kettle finished boiling. The suggested patch widened the retry loop, stretched the backoff, and described the edit as obviously safe for a transient partner error. The diff touched fewer than thirty lines, yet that service already shared a partner rate limit that billed every additional outbound attempt. A hopeful merge would have converted a short outage into a quiet invoice and a harder incident review.

The useful response was not another broad opinion about artificial intelligence replacing careful operational judgment overnight. It was a time-boxed spike with one hypothesis and a kill rule written down before any candidate code ran. The hypothesis said the wider retry would recover the captured timeout without exceeding four outbound attempts or moving the idempotency key. Ninety minutes covered setup, replay, notes, and the ship-or-discard decision, so unproven work stayed unmerged when the window closed.

Freeze the clock before judging the patch

The artifact is a small replay harness that freezes time and counts outbound attempts against one redacted failure. It is proposed spike code, not a report of a named production trace already executed on a public host. A captured failure is stored as JSON with the idempotency key and the ordered partner responses inside it. The harness drives the candidate retry function through that sequence while a fake clock advances only when the test requests sleep.

If the function sleeps, the fake clock jumps forward, and the test records that jump instead of waiting on a wall clock. That choice keeps the ninety-minute budget from being consumed by real backoff delays during a local replay. Reviewers inspect attempt counts without confusing laptop time with partner time, and the harness below is what the spike trusts or kills. The listing that follows is an unexecuted example meant to be copied into the spike directory and run locally.

import json
import sys
from dataclasses import dataclass, field

@dataclass
class FrozenClock:
    now_ms: int = 0
    sleeps: list = field(default_factory=list)

    def sleep(self, ms: int) -> None:
        self.sleeps.append(ms)
        self.now_ms += ms

@dataclass
class AttemptLog:
    calls: int = 0
    keys: set = field(default_factory=set)

def replay(retry_fn, trace: dict, clock: FrozenClock) -> AttemptLog:
    log = AttemptLog()
    responses = list(trace["responses"])

    def partner_call(key: str) -> dict:
        log.calls += 1
        log.keys.add(key)
        if not responses:
            return {"status": 599, "body": "trace exhausted"}
        return responses.pop(0)

    retry_fn(
        key=trace["idempotency_key"],
        call=partner_call,
        sleep=clock.sleep,
        budget=trace["max_attempts"],
    )
    return log

def judge(log: AttemptLog, trace: dict) -> None:
    if log.calls > trace["max_attempts"]:
        raise SystemExit("kill: attempt budget exceeded")
    if log.keys != {trace["idempotency_key"]}:
        raise SystemExit("kill: idempotency key drifted")
    if not log.calls:
        raise SystemExit("kill: patch made no attempt")

# Replace candidate_retry with the assistant patch under trial.
def candidate_retry(key, call, sleep, budget):
    delay = 200
    for _ in range(budget):
        result = call(key)
        if result["status"] < 500:
            return result
        sleep(delay)
        delay = min(delay * 2, 1600)
    return {"status": 504, "body": "budget spent"}

def main() -> None:
    trace_path = sys.argv[1] if len(sys.argv) > 1 else "trace.json"
    with open(trace_path, encoding="utf-8") as handle:
        trace = json.load(handle)
    clock = FrozenClock()
    log = replay(candidate_retry, trace, clock)
    judge(log, trace)
    print(f"ship: calls={log.calls} clock_ms={clock.now_ms}")

if __name__ == "__main__":
    main()

The candidate function under test should accept the same four arguments that the harness passes into each replay. A careful patch retries only while the captured status is retriable, sleeps through the frozen clock, and stops once the budget is spent. A reckless patch often treats every non-200 status as temporary, including a conflict that means the partner already accepted the charge. The spike cares about that distinction more than it cares about the confident prose wrapped around the diff.

A trace file keeps the evidence inspectable by a reviewer who never opened the assistant thread during the incident. The sample holds one timeout, one overload, and one success, which is enough to exercise backoff without imitating a full capture. Teams should replace that sample with a redacted trace from their own incident archive before trusting a ship decision. They should also refuse any file that still contains secrets, card numbers, partner credentials, or raw customer payloads.

{
  "idempotency_key": "evt_frozen_1042",
  "max_attempts": 4,
  "responses": [
    {"status": 504, "body": "timeout"},
    {"status": 503, "body": "overload"},
    {"status": 200, "body": "accepted"}
  ]
}

The commands belong in the spike notes so another engineer can rerun the decision without reconstructing the chat. The first command creates a disposable directory, and the later commands run the harness and record a hash of the trace. A changed fixture hash means the evidence moved, so the previous exit code no longer supports a ship decision. None of these commands should be pointed at a live partner endpoint, and none of them requires production credentials.

mkdir -p spike/retry-replay && cd spike/retry-replay
python replay_harness.py trace.json
sha256sum trace.json > trace.sha256 2>/dev/null || shasum -a 256 trace.json > trace.sha256

Read the exit code, then stop

Reading the result should stay mechanical rather than rhetorical, because the spike promised evidence instead of another assistant summary. A zero exit, a stable idempotency key, and an attempt count inside the budget support a ship for this trace only. A non-zero exit, a drifted key, or an exhausted response list supports a kill, even when the comment calls the miss harmless. If the ninety minutes expire before a clean run, the spike ends as a kill by timeout and the patch waits.

This approach is a poor fit for several kinds of work, and saying so matters as much as the harness itself. It should not be used as a load test, a security review, or proof that a disposable server matches production capacity. It should not be used on traces that cannot be redacted, or on flows where a retry can move money without an idempotency key. Regulated teams that require validated vendors and signed change control should keep the replay in a local lab until those controls are satisfied.

The limitations are ordinary, and they belong beside the exit code rather than in a separate apology. A frozen clock hides real jitter, resolver failures, and pools that misbehave only when many calls overlap. One trace cannot stand for a week of partner behavior, and a green replay does not license a wider policy on unrelated routes. Free model access can draft the harness quickly, yet that draft stays untrusted until kill rules fire in code.

A disposable host can also disappear, differ from production, or reject the workload before the replay finishes. The note should record that host only as the place the replay ran, not as a performance trophy or a capacity claim. If the trace is older than the current partner contract, the spike should be killed and recaptured instead of treated as fresh evidence. Readers with a redacted trace can run this harness on a disposable server and keep the patch only if the replay holds.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free server option is enough to reproduce the setup.

Ship the Fix Only If the Fixture Hash Holds

Harper Zhu — Fri, 09 Oct 2026 13:22:14 +0000

A quiet afternoon in a shared engineering channel often starts with a green check that nobody trusts. The agent has repaired a parser, the suite is green, and the pull request looks small. An hour later a reviewer notices that the expected JSON changed by two keys, which is how the failure disappeared. The bug did not leave the code, because the witness file had been rewritten to match the broken behavior.

That pattern is familiar to anyone who reviews agent-made patches against fixture-heavy tests in a busy repository. A fixture is not decoration; it is the frozen statement of what the function owed the caller before the spike began. When the agent may edit both the implementation and the expected file, a passing test stops being evidence and becomes a negotiated story. The useful distinction is not a greener suite, but whether the original witness still hashes to the same value after the session.

This article proposes a ninety-minute spike with one hypothesis and a ship-or-kill rule built around that hash. The hypothesis stays narrow, since the agent must repair one pure function while the oracle file remains byte-for-byte identical. The spike ships only when the hash holds, the clock stays inside ninety minutes, and the original oracle test moves from red to green. Anything else is a kill, including a clever fix that also improves the fixture while the tests turn green.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator supplied two availability claims for this workflow, namely free model access and a free server option. No model names, quotas, hardware shapes, or permanence claims are added here, because those details were not verified for this draft. The fixture hash stays on the reviewer's machine, so the remote session cannot bargain with the seal.

The analogy is a sealed evidence bag in a small lab that closes at a fixed hour. The reviewer photographs the bag, writes the seal number in a notebook, and only then lets the technician open the broken instrument. If the seal number changes, the technician's report is irrelevant, even when the instrument now hums on the bench. A hash is that seal number for a JSON fixture, and the ninety-minute clock is only the lab's closing time.

The sample bug is deliberately dull, so the harness can be read without domain lore or a long specification. A function named format_cents should render integer cents as a dollar string, and it currently drops the sign on negative values. The oracle file records three inputs and three expected strings, including a negative amount and zero. The test loads that file and compares results rather than computing expected strings, so a wrong rule cannot grade itself.

The following layout is a proposal, not a log of an executed run on any particular server. No timing numbers, pass rates, or server measurements are claimed, because this draft was not executed as a benchmark. Readers should treat every command as a ritual to copy and adapt, not as a transcript of a finished trial.

The spike directory holds a broken pure function, a three-case oracle, and a test that only compares stored strings. The function computes a sign and then discards it, which is the entire defect under trial. The oracle records zero, a positive amount, and negative forty cents, with expected strings written by the reviewer before anyone else arrives. The test reads that file from disk, so a passing run cannot invent a friendlier expectation inside the assertion itself.

spike/
  src/format_cents.py
  tests/test_format_cents.py
  fixtures/oracle.json

def format_cents(cents: int) -> str:
    sign = "-" if cents < 0 else ""
    whole, frac = divmod(abs(cents), 100)
    # Bug under test: the sign is computed and then discarded.
    return f"{whole}.{frac:02d}"

{
  "cases": [
    {"cents": 0, "expect": "0.00"},
    {"cents": 1250, "expect": "12.50"},
    {"cents": -40, "expect": "-0.40"}
  ]
}

import json
from pathlib import Path
from src.format_cents import format_cents

def test_oracle_cases():
    payload = json.loads(Path("fixtures/oracle.json").read_text())
    for case in payload["cases"]:
        assert format_cents(case["cents"]) == case["expect"]

Before the clock starts, the reviewer writes the oracle hash and the start timestamp outside the spike directory. Those notes stay in the home folder, because the directory copied to a server should not contain the seal. The first pytest run is expected to fail on the negative case, and that red result is the baseline worth saving. A spike that starts from an already green tree cannot prove that anything was repaired.

mkdir -p ~/spike-notes
python - <<'PY'
import hashlib
from pathlib import Path
oracle = Path("fixtures/oracle.json")
digest = hashlib.sha256(oracle.read_bytes()).hexdigest()
out = Path.home() / "spike-notes" / "oracle.sha256"
out.write_text(digest + "\n")
print(digest)
PY
date -u +%Y-%m-%dT%H:%M:%SZ > ~/spike-notes/started_at
PYTHONPATH=. pytest tests/test_format_cents.py -q

When a coding session is used, it runs on the disposable free server, away from laptops that hold credentials. The reviewer copies only the spike directory, asks for the sign to be preserved, and states that the oracle is read-only evidence. The returned patch is applied in a second local worktree, where a small judge checks the clock, the hash, an empty fixture diff, and pytest. That script is not a model, and it has no network calls and no authority to rewrite the witness in order to be kind.

import hashlib
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path

ROOT = Path(sys.argv[3]).resolve()
ORACLE = ROOT / "fixtures" / "oracle.json"
LIMIT_MINUTES = 90

def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def main() -> int:
    expected = Path(sys.argv[1]).read_text().strip()
    started_raw = Path(sys.argv[2]).read_text().strip().replace("Z", "+00:00")
    started = datetime.fromisoformat(started_raw)
    elapsed = datetime.now(timezone.utc) - started
    if elapsed.total_seconds() > LIMIT_MINUTES * 60:
        print("kill: clock exceeded ninety minutes")
        return 2
    if sha256(ORACLE) != expected:
        print("kill: fixture hash moved")
        return 3
    fixture_diff = subprocess.check_output(
        ["git", "diff", "--name-only", "--", "fixtures"],
        cwd=ROOT,
        text=True,
    )
    if fixture_diff.strip():
        print("kill: fixture diff is not empty")
        return 4
    proc = subprocess.run(
        ["pytest", "tests/test_format_cents.py", "-q"],
        cwd=ROOT,
    )
    if proc.returncode != 0:
        print("kill: oracle test still failing")
        return 5
    print("ship: hash held and oracle moved from red to green")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

The invocation passes the home-directory notes and the review worktree, and the reviewer should run it rather than the agent. A ship line means the original hash still matches and the same oracle file now passes under pytest. A kill line is a finished spike, because the ninety minutes exist to force a decision rather than a merge. If the hash moved, later green tests are set aside, just as a broken seal sets aside a tidy lab report.

PYTHONPATH=~/work/spike-review python ~/spike-notes/freeze_and_judge.py \
  ~/spike-notes/oracle.sha256 \
  ~/spike-notes/started_at \
  ~/work/spike-review

The repair a human would expect, labeled here as an example rather than as measured agent output, restores the discarded sign. Negative forty cents renders as a signed value, while zero and twelve fifty keep the strings already stored in the oracle. Readers can apply that correction by hand if they want the judge to print ship on a local clone, without any remote session. Doing so checks the harness itself, which is a separate act from judging an agent patch.

def format_cents(cents: int) -> str:
    sign = "-" if cents < 0 else ""
    whole, frac = divmod(abs(cents), 100)
    return f"{sign}{whole}.{frac:02d}"

A stricter local note can sit beside the hash without widening the hypothesis or inviting a second experiment. After the patch is applied, the fixture diff should be empty, and status should not list a new expected file beside the frozen one. Those checks catch a laundered witness that would leave the original hash intact while giving the test a friendlier table. The judge already rejects a diff under fixtures, and the status check is a manual twin when untracked files hide from diff.

git diff -- fixtures/oracle.json
git status --porcelain -- fixtures tests

This ritual has sharp edges, and they should be named before a ship line is treated as a release gate. The hash cannot tell whether the original fixture was wrong, so a frozen mistake will reject a correct behavior change on purpose. That rejection is the desired failure mode for this spike, and it is a bad failure mode for ordinary product work. The clock measures the reviewer's wall time, not model latency, queue delay, or server fairness, and no benchmark of those quantities is offered.

A free server that sleeps, resets, or lacks the language toolchain simply ends the spike as a kill inside the time box. That outcome is acceptable for a bounded experiment, and it is useless as a capacity plan or a purchasing argument. The script trusts local git and local pytest, so it should not run where those commands are aliased to editable wrappers. A wrapper edited by the patch could hide a fixture change, which would turn the seal into theater.

Some teams should leave this ritual on the shelf and choose a slower path for production review. Production hotfixes, schema migrations, and any tree that holds live credentials do not belong in a ninety-minute disposable session. Security evaluation is also out of scope, because the harness does not look for data theft and should not touch secret-bearing repositories. If the defect lives in the fixture format itself, freezing that file freezes the wrong layer, and the work belongs in a schema review.

Readers who need cost, quota, or hardware comparisons will not find them in this draft, since those figures were not verified. Inventing those figures to decorate a method would make the article less honest than the harness it recommends. The method still earns its keep on ordinary library bugs where a pure function and a small oracle can disagree in public. A reviewer can prepare the red baseline before lunch, hand only the spike directory to a disposable session, and return to a ship-or-kill script.

That is a smaller promise than a general agent score, and it is the kind of promise a fixture hash can actually keep. Someone who already has a free server session can copy this spike directory across and leave the hash file at home. A kill should be treated as a completed result rather than a failed publication or a reason to stretch the clock. The channel then receives a seal number beside the decision, instead of another green check that nobody trusts.

The Spike That Ends Itself at Minute Ninety

Harper Zhu — Thu, 08 Oct 2026 09:24:46 +0000

A platform team once spent an entire weekend asking whether an AI coding assistant could replace a brittle internal code-generation script. By Sunday night three half-finished branches existed, nobody had written down a single command, and the only surviving claim was that the idea "felt promising". The tool was never the problem; the missing ingredient was a rule that ended the experiment before enthusiasm could.

That rule can be small. A ninety-minute spike with one written hypothesis, a fixed internal budget, and a verdict file turns open-ended exploration into something a team can review on Monday morning. The budget matters more than the wall clock, because the box exists to force a decision while the evidence is still warm.

One hypothesis, written before the first command

A spike without a falsifiable sentence drifts within twenty minutes. The hypothesis should name the change, the observable signal, and the direction of success, for example "moving request validation into the gateway removes the duplicate check in the service". If the sentence cannot be made falsifiable, the team is not spiking; it is reading documentation with extra steps.

Budget the box before starting it. Ten minutes go to writing the hypothesis and the smallest failing test, fifty to the probe itself, fifteen to replaying the recorded commands, and fifteen to the verdict. Any phase that overruns borrows from the probe, never from the evidence phase. Teams that skip the replay step almost always rediscover their own results a week later and trust them less.

The harness that stops on its own

The script below enforces the budget and records everything that matters. It relies only on bash, git, make, and python3, so it runs identically on a laptop or on a remote host.

#!/usr/bin/env bash
# spike.sh — a 90-minute spike that stops whether or not anyone is watching.
set -euo pipefail

HYPOTHESIS="${1:?usage: spike.sh \"<one falsifiable sentence>\"}"
BUDGET_MIN="${BUDGET_MIN:-90}"
DEADLINE=$(( $(date +%s) + BUDGET_MIN * 60 ))
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p spike-artifacts
EVIDENCE="spike-artifacts/spike-${STAMP}.log"

remaining() { echo $(( (DEADLINE - $(date +%s)) / 60 )); }
note()      { printf '%s | %s\n' "$(date -u +%H:%M:%S)" "$*" | tee -a "$EVIDENCE"; }
phase()     { note "phase=$1 budget=${2}m remaining=$(remaining)m"; }

guard() {
  if [ "$(remaining)" -le 0 ]; then
    note "STOP: budget exhausted, verdict parked"
    printf 'verdict=park\nreason=budget-exhausted\n' > spike-artifacts/verdict.txt
    exit 3
  fi
}

phase hypothesis 10
printf '%s\n' "$HYPOTHESIS" | tee spike-artifacts/hypothesis.txt

phase probe 50
set -x
( git worktree add --detach .spike HEAD && make failing-test ) 2>&1 | tee -a "$EVIDENCE"
guard
( cd .spike && time make probe ) 2>&1 | tee -a "$EVIDENCE"
set +x

phase evidence 15
( cd .spike && git diff --stat && git diff ) 2>&1 | tee -a "$EVIDENCE"
guard

phase decide 15
python3 classify.py "$EVIDENCE" | tee -a "$EVIDENCE"

The failure marker is the point. The harness refuses to proceed unless a test failed first, because a probe that never failed has proven nothing about the hypothesis. When the clock runs out, guard writes verdict=park and exits with code 3 instead of letting the session slide into a second evening.

Turning the log into a verdict

A human reading a two-thousand-line log at the end of a spike will see whatever they hoped to see. A small deterministic classifier removes that bias, and it can be rerun by anyone who doubts the conclusion.

#!/usr/bin/env python3
"""classify.py — ship / kill / park from a raw spike log."""
import pathlib, re, sys

lines = pathlib.Path(sys.argv[1]).read_text(errors="replace").splitlines()
FAIL  = re.compile(r"\b(FAIL|error|Traceback|panic:)\b")
GREEN = re.compile(r"\b(ok|PASS|passed)\b")

first_fail = next((i for i, l in enumerate(lines) if FAIL.search(l)), None)
green_after = first_fail is not None and any(
    GREEN.search(l) for l in lines[first_fail:]
)

verdict = "park" if first_fail is None else ("ship-to-branch" if green_after else "kill")
print(f"verdict={verdict}")
print(f"lines={len(lines)} first_failure_line={first_fail}")

The classification is deliberately coarse, and the coarse rule is what makes it reviewable. A missing failure parks the work, a failure that later goes green ships it to a branch, and a failure that never goes green kills it. The classifier does not judge code quality, and nobody should pretend that it does.

The decision table the reviewer actually needs

Verdict	Evidence in the log	Action within 24 hours
`ship-to-branch`	Failure recorded first, green afterwards, diff roughly under 200 lines	Open a branch, paste the replay command into the pull request body
`kill`	Failure never turns green and no smaller failing case emerges	Remove the worktree, log two lines in the decision record
`park`	No failure at all, or the budget expired before evidence	Keep the log, rewrite the hypothesis, schedule a second box

The table is the artifact a reviewer can argue with, which is precisely why it belongs in the repository next to the harness. It converts a subjective feeling into a recorded state that survives the people who produced it.

Where free model access and a free server fit

The loop above is tool-agnostic, but two pieces of infrastructure make it cheaper to run. MonkeyCode provides free model access and a free server option according to the operator, which means the probe code and the failing test can be drafted without a budget conversation, and the box can keep running when a laptop lid closes. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free model access is most useful in the probe phase, where the assistant drafts the failing test and the smallest candidate change. The verdict must still come from replaying recorded commands, not from a generated summary of them, because a model's description of a result is not a result. Quotas, available models, and hosting terms change over time, so confirm the current terms in the project repository before planning a team workflow around them.

Limitations and who should not use this

Ninety minutes is the wrong box for work whose cost is dominated by something other than reasoning. Schema migrations, distributed tracing changes, load-dependent performance work, and anything gated by data access or legal review will simply expire at the budget and produce a park. Teams needing publication-grade benchmarks should run a proper measurement plan, not a spike with a stopwatch.

The harness also measures discipline rather than quality. It cannot tell whether the hypothesis was worth writing, and it will happily kill a good idea that needed two hours instead of ninety minutes. Anyone unwilling to replay recorded commands, or unable to keep a decision log, gets a slower version of the same exploratory mess.

For teams that want to test the loop without provisioning anything, the same harness runs unmodified on MonkeyCode's free server option, and the repository is the place to check what the current free tier includes.

Stop the Spike If the Lockfile Moves

Harper Zhu — Thu, 08 Oct 2026 06:50:07 +0000

A release engineer opened a short pull request that promised to repair one failing assertion in a billing fixture. The diff looked harmless until the lockfile grew by several hundred lines and an example environment file gained a new secret-shaped key. The assistant had a vague hour and no sealed baseline, so the review became archaeology rather than a clean decision. The team killed the change, yet the wasted afternoon showed that an unsealed spike can cost more than the original bug.

The useful response is neither a broader prompt nor a longer transcript saved for later debate. A spike should hold one hypothesis, one clock, and one ship-or-kill rule that a stranger can rerun on a clean machine. The hypothesis stays narrow: fix one pinned fixture failure without moving the lockfile, the environment example, or unrelated packages. If any sealed file changes, the spike is killed even when the targeted test turns green.

That rule sounds strict because dependency drift often hides inside behavior that looks helpful at review time. An assistant that just updates the toolchain can turn a one-line assertion fix into a supply-chain review and a deployment risk. A green test does not cancel that expansion, any more than a clean hallway cancels a door left open overnight. The seal is the hallway record: it shows what moved, not how persuasive the commit message sounds to a tired reviewer.

The clock starts only after the failure is pinned

The spike begins in a disposable worktree cut from a known commit, not in the developer's everyday checkout. The failing command is recorded before any assistant edit, because a fixture that already passes cannot anchor a repair claim. Checksums of the lockfile, the environment example, and the fixture file are written to an evidence directory that travels with the review. The script writes a deadline epoch at seal time, and only then may an assistant touch the tree.

The following script is a proposed workflow, not an executed benchmark and not a claim about any particular model run. A reviewer should read it, adjust paths, and run it only on a repository that contains no production secrets. The commands assume a Node-style lockfile and a billing fixture, which stand in for the single failure the team cares about this week. Windows users can translate the checksum and worktree steps, but the verdict rule should stay identical across shells.

#!/usr/bin/env bash
# Proposed spike seal. Unexecuted example; review paths before any run.
# TEST_CMD is passed to bash -lc and must be a trusted string, never untrusted input.
# Requires sha256sum. On macOS, install coreutils or replace it with shasum -a 256.
set -euo pipefail

MODE="${1:-seal}"
STAMP="${2:-}"
ROOT="$(cd "${SPIKE_ROOT:-.}" && pwd)"
LOCK_REL="${LOCK_REL:-package-lock.json}"
ENV_REL="${ENV_REL:-.env.example}"
FIXTURE_REL="${FIXTURE_REL:-fixtures/billing/invoice_min.json}"
TEST_CMD="${TEST_CMD:-npm test -- --testPathPattern=billing_invoice}"
PARENT="$(cd "$ROOT/.." && pwd)"

if [[ "$MODE" == "seal" && -z "$STAMP" ]]; then
  STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
fi
if [[ -z "${STAMP}" ]]; then
  printf 'usage: spike-seal.sh seal|judge STAMP\n' >&2
  exit 1
fi

WORK="${PARENT}/spike-${STAMP}"
EVIDENCE="${PARENT}/spike-evidence/${STAMP}"
mkdir -p "$EVIDENCE"

run_test() {
  local dest="$1"
  set +e
  bash -lc "$TEST_CMD" >"$dest" 2>&1
  local rc=$?
  set -e
  printf '%s\n' "$rc"
}

seal() {
  git -C "$ROOT" rev-parse --is-inside-work-tree >/dev/null
  local base
  base="$(git -C "$ROOT" rev-parse HEAD)"
  git -C "$ROOT" worktree add --detach "$WORK" "$base"
  printf '%s\n' "$(( $(date +%s) + 5400 ))" > "${EVIDENCE}/deadline.txt"
  (
    cd "$WORK"
    sha256sum "$LOCK_REL" "$ENV_REL" "$FIXTURE_REL"
  ) | tee "${EVIDENCE}/seal-before.txt"
  local before_rc
  before_rc="$(cd "$WORK" && run_test "${EVIDENCE}/test-before.txt")"
  printf 'before_exit=%s\nbase=%s\n' "$before_rc" "$base" | tee "${EVIDENCE}/before.txt"
  if [[ "$before_rc" -eq 0 ]]; then
    printf 'kill: fixture already green\n' | tee "${EVIDENCE}/verdict.txt"
    exit 2
  fi
  printf 'sealed %s\nworktree %s\n' "$STAMP" "$WORK" | tee "${EVIDENCE}/ready.txt"
}

judge() {
  [[ -d "$WORK" ]] || { printf 'missing worktree %s\n' "$WORK" >&2; exit 1; }
  [[ -f "${EVIDENCE}/deadline.txt" ]] || { printf 'kill: missing deadline\n' | tee "${EVIDENCE}/verdict.txt"; exit 7; }
  if [[ "$(date +%s)" -gt "$(cat "${EVIDENCE}/deadline.txt")" ]]; then
    printf 'kill: ninety minutes elapsed\n' | tee "${EVIDENCE}/verdict.txt"
    exit 7
  fi
  (
    cd "$WORK"
    sha256sum "$LOCK_REL" "$ENV_REL" "$FIXTURE_REL"
  ) | tee "${EVIDENCE}/seal-after.txt"
  if ! cmp -s "${EVIDENCE}/seal-before.txt" "${EVIDENCE}/seal-after.txt"; then
    git -C "$WORK" diff --stat HEAD | tee "${EVIDENCE}/diffstat.txt"
    printf 'kill: seal moved\n' | tee "${EVIDENCE}/verdict.txt"
    exit 3
  fi
  local after_rc
  after_rc="$(cd "$WORK" && run_test "${EVIDENCE}/test-after.txt")"
  git -C "$WORK" diff --stat HEAD | tee "${EVIDENCE}/diffstat.txt"
  if [[ "$after_rc" -ne 0 ]]; then
    printf 'kill: fixture still red\n' | tee "${EVIDENCE}/verdict.txt"
    exit 4
  fi
  if git -C "$WORK" diff --quiet HEAD; then
    printf 'kill: no patch after a red start\n' | tee "${EVIDENCE}/verdict.txt"
    exit 5
  fi
  local outside
  outside="$(git -C "$WORK" diff --name-only HEAD | grep -Ev '^(fixtures/billing/|src/billing/|test/billing/)' || true)"
  if [[ -n "$outside" ]]; then
    printf 'kill: patch left the billing boundary\n%s\n' "$outside" | tee "${EVIDENCE}/verdict.txt"
    exit 6
  fi
  printf 'ship: seal held and fixture turned green\n' | tee "${EVIDENCE}/verdict.txt"
}

case "$MODE" in
  seal) seal ;;
  judge) judge ;;
  *) printf 'usage: spike-seal.sh seal|judge STAMP\n' >&2; exit 1 ;;
esac
# After review: git -C "$ROOT" worktree remove --force "$WORK"

The seal mode refuses to start when the pinned test already passes, which keeps the spike from celebrating a no-op. The judge mode rejects a moved seal before it respects a green result, and it kills any patch that arrives after the deadline epoch. A path boundary then limits the ship decision to the billing fixture and its owning source, which keeps one repair inside one room. Evidence files store both logs, the diff stat, the deadline epoch, and one verdict word for review.

A second shell records the same commands a reviewer would type after the script file is saved beside the repository. The seal invocation prints a stamp, and that stamp must be passed unchanged to the judge so the checksum files stay paired. The assistant is pointed at the sibling worktree and told to stop when the deadline epoch passes, even if the edit feels unfinished. The verdict file is then the only line copied into the pull request body for the reviewer.

export SPIKE_ROOT="$PWD"
export TEST_CMD='npm test -- --testPathPattern=billing_invoice'
bash spike-seal.sh seal
# Read stamp and worktree path from ../spike-evidence/<stamp>/ready.txt
# Let the assistant edit only that worktree, with no dependency-update permission.
bash spike-seal.sh judge <stamp>
cat ../spike-evidence/<stamp>/verdict.txt
git -C "$SPIKE_ROOT" worktree remove --force ../spike-<stamp>

Where a free server fits, and where it does not

Running that loop on a laptop is possible, but a shared workstation collects leftover worktrees, cached credentials, and half-edited prompts. A separate machine keeps the experiment closer to the claim being tested, which is whether the patch respects the seal under a clock. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project, and its operator-stated free model access plus a free server can host the spike.

Those two availability claims are the only product facts used here, and they should be rechecked against current project documentation before reliance. This article does not name models, quotas, hardware sizes, regions, or a promised duration, because those details change and stale numbers mislead. A free server is a convenient isolation boundary, not proof that the assistant is careful, cheap, or suitable for production traffic. If current terms omit a wipeable server, the same script still runs inside any other disposable virtual machine.

The assistant, wherever it runs, should receive the worktree path, the failing command, the deadline, and the kill rule in plain text. It should not receive production secrets, cloud tokens, or a request to update dependencies if that seems helpful. Network fetches during the spike are a limitation, since a cached package can change behavior without moving the sealed lockfile at all. Teams that cannot disable extra registries should treat a ship verdict as provisional until a cold-cache run repeats the same result.

How the verdict should be read

A ship verdict means only that this fixture turned from red to green while the sealed files stayed byte-for-byte identical. It does not mean the billing rules are correct, the assistant is generally trustworthy, or the next fixture will behave the same way. A kill verdict is also useful, because it ends the debate at minute ninety instead of feeding a weekend review. The evidence directory is the artifact a teammate can open without replaying the chat or reconstructing the prompt from memory.

People who already know the repair requires a lockfile change should not use this hypothesis, because the seal would kill a legitimate upgrade. Teams that must process production data, regulated secrets, or customer fixtures should not point a free server at those files. Anyone who needs an agreement, fixed hardware, or a permanent free quota should not treat this workflow as a capacity plan. Reviewers who want a leaderboard of models should look elsewhere, since one sealed fixture cannot support that comparison honestly.

The practical close is small and does not require a new platform decision before the first red fixture is chosen. Cut a worktree, write the seal, start the clock, and let minute ninety end the argument with a verdict file. If a free isolated server is available under current MonkeyCode terms, it is a reasonable place to keep synthetic fixtures off the laptop. The verdict file, not the transcript and not a promotional claim, is what should be pasted into the pull request for review.