Split the Clock and Repeat the Spike

On a gray Tuesday a payments team let an agent chase a flaky CSV importer while a wall clock sat beside the keyboard. Forty minutes later the unit suite was green and the chat log sounded like a victory lap around the office. A reviewer merged the branch because the spike had a hypothesis, a deadline, and a passing test command. Production broke that night when a fixture filename leaked into a parser branch that nobody had frozen.

Single-run spikes still dominate agent evaluation because a green bar feels like evidence after a tense hour. The bar is a snapshot, not a decision, and snapshots lie when the environment drifts between attempts. A ninety-minute spike needs a ship-or-kill rule that survives luck, cached files, and an agent that memorizes fixtures.

Public conversation this month keeps returning to whether coding agents already outrun the tests used to grade them. Other threads warn that treating a fluent session as engineering confuses a demo with a change that can be replayed. Those arguments stay theatrical until a team records one hypothesis and demands two agreeing runs inside one clock.

The workflow below keeps a ninety-minute box and adds a variance gate that a single green run cannot satisfy. Before the clock starts, the public interface of the change is written by a human and hashed on disk. The agent may implement behind that interface, yet any edit to the contract file kills the spike immediately. The same hypothesis then runs twice in freshly unpacked directories, and disagreement between those runs is also a kill.

Both confirming runs must finish inside the original ninety minutes, including package install time and the test command. If the first attempt consumes the whole budget, the spike fails because no second sample exists to compare. That rule sounds harsh until a team remembers that irreproducible speed is how fixture leaks reach production.

A copyable template uses three artifacts: a frozen contract module, tests that import only that module, and a gate script. The script is an unexecuted example for operators to adapt, not a measured claim about any particular model. Replace the placeholder agent command with the runner the team already trusts, then keep the comparison logic intact.

# contract.py — freeze and hash this file before the clock starts.
from typing import Protocol


class CsvBatchImporter(Protocol):
    def parse_rows(self, payload: bytes) -> list[dict[str, str]]:
        """Return row dicts keyed by the header names in the first line."""
        ...

    def reject_reason(self, payload: bytes) -> str | None:
        """Return a stable error token, or None when parse_rows would succeed."""
        ...

The protocol is intentionally small so the spike cannot hide a redesign inside a passing suite. Header keys and a stable reject token are ordinary product rules rather than a published research benchmark. A human writes this file, commits it, and copies its hash into the evidence directory before any agent process starts.

# test_importer.py — bind only to the frozen protocol and a stable reject token.
from importer import DefaultImporter

SAMPLE = b"id,amount\n1,10.00\n2,bad\n"


def test_parse_rows_keeps_header_keys():
    rows = DefaultImporter().parse_rows(b"id,amount\n1,10.00\n")
    assert rows == [{"id": "1", "amount": "10.00"}]


def test_reject_reason_is_stable_token():
    reason = DefaultImporter().reject_reason(SAMPLE)
    assert reason == "invalid_amount"

These tests refuse to import helper modules the agent might invent for convenience during a lucky first run. They also pin the reject token so a second run cannot rename errors and still look successful. If DefaultImporter is missing, the suite fails, and that failure is a valid kill rather than a reason to extend the clock.

#!/usr/bin/env bash
# spike_gate.sh — unexecuted template for a 90-minute two-run spike.
set -euo pipefail

SPIKE_MINUTES="${SPIKE_MINUTES:-90}"
HYPOTHESIS="${HYPOTHESIS:-DefaultImporter satisfies CsvBatchImporter and reject tokens stay stable}"
CONTRACT="${CONTRACT:-contract.py}"
AGENT_CMD="${AGENT_CMD:-echo replace-me-with-the-team-agent-runner}"
TEST_CMD="${TEST_CMD:-python3 -m pytest -q test_importer.py}"
ROOT="$(pwd)"
START_EPOCH="$(date +%s)"
DEADLINE="$((START_EPOCH + SPIKE_MINUTES * 60))"
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
EVIDENCE="$ROOT/spike_evidence_$STAMP"
mkdir -p "$EVIDENCE"

remaining() {
  echo "$((DEADLINE - $(date +%s)))"
}

if [[ ! -f "$CONTRACT" ]]; then
  echo "kill: missing $CONTRACT" >&2
  exit 2
fi

CONTRACT_SHA="$(sha256sum "$CONTRACT" | awk '{print $1}')"
printf '%s\n' "$HYPOTHESIS" > "$EVIDENCE/hypothesis.txt"
printf '%s\n' "$CONTRACT_SHA" > "$EVIDENCE/contract.sha256"
git archive --format=tar HEAD > "$EVIDENCE/clean_tree.tar"

run_one() {
  local name="$1"
  local dir="$EVIDENCE/$name"
  mkdir -p "$dir"
  tar -xf "$EVIDENCE/clean_tree.tar" -C "$dir"

  local budget
  budget="$(remaining)"
  if (( budget < 120 )); then
    printf '%s\n' "{\"name\":\"${name}\",\"status\":\"timeout_before_start\",\"test_rc\":1,\"contract_sha256\":\"${CONTRACT_SHA}\",\"requirements_sha256\":\"\"}" > "$dir/result.json"
    return 0
  fi

  if [[ "$name" == "run_a" ]]; then
    budget="$((budget / 2))"
  fi

  local test_rc new_sha req_sha
  pushd "$dir" >/dev/null
  set +e
  timeout --preserve-status "$budget" bash -lc "$AGENT_CMD" >agent.log 2>&1
  timeout --preserve-status 120 bash -lc "$TEST_CMD" >test.log 2>&1
  test_rc=$?
  set -e
  new_sha="$(sha256sum "$CONTRACT" | awk '{print $1}')"
  req_sha=""
  if [[ -f requirements.txt ]]; then
    req_sha="$(sha256sum requirements.txt | awk '{print $1}')"
  fi
  python3 - "$name" "$test_rc" "$new_sha" "$req_sha" "$budget" >result.json <<'PY'
import json, sys
name, test_rc, new_sha, req_sha, budget = sys.argv[1:6]
print(json.dumps({
    "name": name,
    "status": "finished",
    "test_rc": int(test_rc),
    "contract_sha256": new_sha,
    "requirements_sha256": req_sha,
    "budget_seconds": int(budget),
}))
PY
  popd >/dev/null
}

run_one run_a
run_one run_b
python3 "$ROOT/compare_runs.py" \
  "$EVIDENCE/run_a/result.json" \
  "$EVIDENCE/run_b/result.json" \
  "$CONTRACT_SHA"

The archive step matters more than the timeout arithmetic, because a dirty working tree is how laptop caches impersonate skill. Using git archive exports HEAD without uncommitted helper files the agent dropped during a previous attempt. Each run unpacks that tarball into its own directory so log files cannot leak from run_a into run_b. Half of the remaining seconds go to the first run so the second run is not leftover theater.

The template assumes GNU timeout from coreutils, which treats the budget as a hard stop rather than a hint. Operators who dislike splitting the clock can keep two directories, yet they no longer have a ninety-minute spike. A longer box is a different experiment, and mixing those experiments is how lucky sessions get relabeled as process.

chmod +x spike_gate.sh compare_runs.py
git add contract.py test_importer.py spike_gate.sh compare_runs.py
git commit -m "Freeze importer contract before the spike clock"
SPIKE_MINUTES=90 ./spike_gate.sh
echo "gate_exit=$?"

Those commands are the whole ritual: freeze the contract, commit the gate, start the clock, and inspect the exit status. Nothing in the chat log can replace the evidence directory that the script leaves beside the repository root. If the exit status is nonzero, the branch stays unmerged even when one of the agent logs reads like a confident summary.

#!/usr/bin/env python3
"""compare_runs.py — unexecuted template. Ship only when both isolated runs agree."""
from __future__ import annotations

import json
import sys
from pathlib import Path


def load(path: str) -> dict:
    return json.loads(Path(path).read_text(encoding="utf-8"))


def main() -> int:
    if len(sys.argv) != 4:
        sys.stderr.write("usage: compare_runs.py run_a.json run_b.json expected_contract_sha\n")
        return 2
    run_a = load(sys.argv[1])
    run_b = load(sys.argv[2])
    expected = sys.argv[3]
    reasons: list[str] = []
    for label, row in ("run_a", run_a), ("run_b", run_b):
        if row.get("status") == "timeout_before_start":
            reasons.append(f"{label} never started")
        if int(row.get("test_rc", 1)) != 0:
            reasons.append(f"{label} tests exited {row.get('test_rc')}")
        if row.get("contract_sha256") != expected:
            reasons.append(f"{label} mutated the contract")
    if run_a.get("contract_sha256") != run_b.get("contract_sha256"):
        reasons.append("contract hashes diverged")
    if run_a.get("requirements_sha256") != run_b.get("requirements_sha256"):
        reasons.append("requirements hashes diverged")
    if run_a.get("test_rc") != run_b.get("test_rc"):
        reasons.append("test exit codes diverged")
    if reasons:
        print("kill")
        for item in reasons:
            print(item)
        return 1
    print("ship")
    print("both runs passed on the frozen contract")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

compare_runs.py is the entire decision procedure, written as code so a chat summary cannot override it. Ship requires matching contract hashes, matching requirement hashes, and matching zero test statuses from both directories. Any other combination prints kill and a reason; the spike ends without a planning meeting attached to the exit code.

Teams that lack a spare empty machine often run both directories on a short-lived remote shell so a laptop cache cannot contaminate the tarball. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option that can host those two isolated runs while the contract hash stays under human control. The gate still belongs to the compared JSON files, and it remains useful if that product mention is ignored.

Reading the evidence directory should feel like reading a receipt rather than a blog post about the session. hypothesis.txt states the only claim under test, and result.json files show whether both clocks produced the same numbers. If reviewers need a narrative, they can add one after the gate, never before it, and never as a substitute for the second run.

The approach is a poor fit for production incidents, where the clock is an outage rather than a learning box. It also fails when the hypothesis requires browsing unknown public APIs, because a frozen contract cannot describe a surface the team has not seen. Visual work without a stable interface, and refactors that must rewrite the public types, should use a different review path than a ninety-minute spike.

Two agreeing runs can still share a systematic hole if both tests encode the same fixture mistake. Matching requirement hashes still allow both runs to add the same unjustified dependency, which isolation will not catch. Isolation cuts environmental luck, and it does not invent missing assertions or turn a weak protocol into a strong one. Teams that cannot control network egress or package mirrors should treat a green gate as a weak signal rather than a merge stamp.

A spike that cannot be repeated inside the same clock is a demo, and demos can teach without shipping. The payments team did not need a longer meeting; they needed a second empty directory and a hash that the agent was forbidden to touch. Freeze the contract, split the remaining minutes, and let disagreement kill the branch before a fixture name reaches production.