The Spike Dies When Two Runs Disagree

Consider a worker that retries a failed job three times, and occasionally performs the same side effect twice. Nobody in the incident thread disagrees about the symptom, but everyone disagrees about the cause, and the argument has already outlasted the fix it is about. Someone proposes asking an AI coding assistant to patch it, and the thread splits a second time: half the team wants the diff now, the other half refuses to trust a patch nobody can reproduce.

That split is not a debate about model quality, and no amount of opinion will settle it. What settles it is evidence narrow enough to be produced inside one meeting block: two attempts, one hypothesis, and a kill rule written down before either attempt starts. A spike is not a smaller project; it is a bet with a deadline, and a bet that cannot lose is not a bet.

Write the hypothesis before the clock starts

The spike begins with a single sentence that predicts a specific, falsifiable outcome. Vague goals such as "evaluate the assistant on our codebase" produce hours of pleasant activity and no verdict, because nothing in them can be contradicted. The card below is the correct size for the exercise, and it is short enough to be read aloud before anyone types a prompt.

Hypothesis: the failing test retries_spec::replays_side_effect_once fails because
            the retry wrapper re-enqueues before the first attempt is acknowledged.
Prediction: a patch that acknowledges before re-enqueue makes the test pass twice
            on HEAD <sha>, touching the retry wrapper and nothing else.
Kill criteria: baseline test already green, two runs disagree on outcome, the two
            diffs share no file, or 90 minutes elapse.
Out of scope: production config, performance, security review.

Three properties make this useful. The base revision is pinned, so a moving branch cannot contaminate the comparison. The prediction names an observable result instead of an intention, so "it feels better" is never an outcome. And the kill criteria are checkable by a script, which matters because the person running the spike is the person most motivated to keep it alive.

One passing run is mostly a coincidence

A single attempt that turns a red test green tells you that something worked once. It does not tell you whether the assistant found the cause or merely found a way past the assertion, and those two futures look identical in a terminal. Cheap sampling makes both futures visible: run the same unmodified prompt twice, on the same revision, in two disposable worktrees, then compare outcomes and diffs side by side.

Two runs do not prove correctness, and the protocol never claims they do. They bound how much confidence a 90-minute budget can honestly buy, and they convert a matter of taste into a pair of files on disk. When two runs disagree, the disagreement itself is the finding: the prompt, the test, or the model is unstable, and none of those three should be shipped over.

The discipline that makes this work is refusing to coach the second run. If the first attempt fails and the operator rewrites the prompt, pastes the stack trace, or points at the guilty file, the two runs are no longer samples of the same process. The second run then demonstrates the operator's understanding, which was never the open question.

A harness that shares one ninety-minute clock

The script below creates the worktrees, records the base revision, checks that the target test actually fails before the assistant touches anything, and saves each attempt's patch, diff stat, and test result. One deadline is computed at the start and shared by both runs, so no run receives a private 90 minutes.

#!/usr/bin/env bash
# spike.sh - two-run, time-boxed spike harness. Illustrative: adapt the paths and commands.
set -euo pipefail

TEST_CMD=${TEST_CMD:-"npm test -- --runInBand"}
ASSISTANT_CMD=${ASSISTANT_CMD:-"./scripts/ask-assistant.sh"}  # your client, your model
PROMPT=${PROMPT:-"spike/hypothesis.md"}
BUDGET=${BUDGET:-5400}              # 90 minutes of wall clock for the whole spike
RUNS=${RUNS:-2}

OUT="spike/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"
deadline=$(( $(date +%s) + BUDGET ))
left() { echo $(( deadline - $(date +%s) )); }

for n in $(seq 1 "$RUNS"); do
  wt="$OUT/wt$n"
  git worktree add --detach "$wt" HEAD >/dev/null 2>&1      # docs: git-scm.com/docs/git-worktree
  git -C "$wt" rev-parse HEAD > "$OUT/run$n.base"

  # 1. Baseline must fail, or the hypothesis is rejected before any model call.
  if timeout "$(left)"s bash -c "cd '$wt' && $TEST_CMD" > "$OUT/run$n.before" 2>&1; then
    echo "run$n: baseline already green - hypothesis rejected" > "$OUT/verdict.txt"
    exit 2
  fi

  # 2. Exactly one attempt per run, with the prompt piped in unchanged.
  timeout "$(left)"s bash -c "cd '$wt' && $ASSISTANT_CMD < '$PROMPT'" > "$OUT/run$n.answer" 2>&1 || true

  # 3. Evidence: what changed, and whether the test passes now.
  git -C "$wt" diff      > "$OUT/run$n.patch"
  git -C "$wt" diff --stat > "$OUT/run$n.stat"
  if timeout "$(left)"s bash -c "cd '$wt' && $TEST_CMD" > "$OUT/run$n.after" 2>&1; then
    echo pass > "$OUT/run$n.result"
  else
    echo fail > "$OUT/run$n.result"
  fi
done

python3 spike/verdict.py "$OUT"

Two details are easy to get wrong. timeout measures wall clock, so a run that hangs on a prompt cannot borrow time from the next one; the semantics are documented in the coreutils manual. And the worktrees are detached at HEAD, which means the spike never touches the branch a human is working on, while git worktree remove --force and git worktree prune clean up afterwards.

Two transcripts, four possible verdicts

The reduction step is deliberately blunt. It reads the two result files, the two base revisions, and the file lists from the diff stats, then emits exactly one verdict without weighting anything and without asking for a summary.

# spike/verdict.py - reduce two spike runs to one verdict.
import json, pathlib, sys

out = pathlib.Path(sys.argv[1])
runs = {}
for result_file in sorted(out.glob("run[0-9]*.result")):
    name = result_file.name.split(".")[0]
    runs[name] = {
        "result": result_file.read_text().strip(),
        "base": (out / f"{name}.base").read_text().strip()[:12],
        "touched": sorted((out / f"{name}.stat").read_text().splitlines()),
    }

outcomes = [run["result"] for run in runs.values()]
bases = {run["base"] for run in runs.values()}
overlap = set.intersection(*(set(run["touched"]) for run in runs.values())) if runs else set()

if len(runs) < 2:
    verdict, why = "kill", "the spike did not finish both runs inside the budget"
elif len(bases) > 1:
    verdict, why = "kill", "runs started from different revisions"
elif outcomes == ["pass", "pass"] and overlap:
    verdict, why = "ship-to-review", "both runs passed and touched at least one identical file"
elif outcomes == ["pass", "pass"]:
    verdict, why = "kill", "two passes with disjoint diffs; the cause was not identified"
else:
    verdict, why = "kill", "runs disagree on outcome"

(out / "verdict.json").write_text(json.dumps({"verdict": verdict, "why": why, "runs": runs}, indent=2))
print(verdict, "-", why)

Run 1	Run 2	Diffs share a file	Verdict
pass	pass	yes	ship to review
pass	pass	no	kill
pass	fail	either	kill
fail	fail	either	kill, and rewrite the hypothesis
any	unfinished	-	kill, with no partial credit

The harshness is the point. A spike that ends with a patch, two transcripts, and a written reason for killing it has produced real value, because the next person will not spend another afternoon on the same argument. "Ship to review" is not "ship": the patch still goes through normal review, and the reviewer still asks the questions a two-run protocol cannot answer.

Where free model access changes the arithmetic

The protocol above doubles the number of attempts and test executions, which is exactly the kind of cost that pushes teams toward sampling once and calling it evidence. MonkeyCode, the open-source project behind this outreach, currently offers free model access and a free server option, which removes the per-attempt price argument from the decision and lets the second run happen as a matter of routine rather than as a favor.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest qualifications belong here. The free tier, including any token allowance, is defined by the project and changes over time, so read the current terms on the project page instead of planning around a number quoted in a blog post. And the assistant invocation in the harness is a placeholder on purpose: the interface for talking to a hosted model is a project detail, so check the repository for its current client shape rather than assuming a command-line tool that may not exist.

Where the free server option genuinely participates is the clock. If the harness runs on a machine that is not the developer's laptop, a slow test suite no longer competes with an editor, and the 90 minutes are spent where the work actually happens. A long run can be detached and its log followed, but the deadline must stay attached to the process.

# Illustrative: detach the spike so the 90-minute clock outlives the terminal session.
cd ~/repo && PROMPT=spike/hypothesis.md nohup ./spike.sh > spike.log 2>&1 &
tail -f spike.log

Limits, and who should skip this entirely

The protocol measures agreement between two samples, and nothing else. It cannot separate a flaky test from a flaky model, so a suite that already fails intermittently will generate kills for the wrong reason; stabilize the flake first or the spike is just noise. It says nothing about performance, security, or behavior under production load, and it never justifies skipping review.

Some teams should not use it at all. Anyone who needs the fix today should fix it today, because a spike is the wrong instrument for an outage. Teams in regulated environments that require auditable tooling and data handling agreements should confirm those constraints before sending code to any hosted assistant. And anyone evaluating a model rather than a bug should choose a different design, because two runs on one test is far too small a sample for that question.

The useful version of this workflow ends with a small directory of evidence: two patches, two base revisions, two test logs, and one verdict with a reason attached. If a spike that costs nothing to repeat is the interesting part, the free model access and free server option are the cheapest way to find out whether your hypothesis survives its first honest disagreement.