The Verdict Compiler: Pre-Registering a 90-Minute Spike

A payments team finishes a ninety-minute spike, the assistant turns three failing tests green on the projector, and the room agrees it felt useful. Three weeks later nobody can reconstruct what the spike actually decided, because the decision was never written down in a form a stranger could check. What survived was a demo, a warm feeling, and a slide.

The failure is not that the team cheated; it is that the grading rule lived in their heads and moved with the mood of the demo. A spike that ends in a feeling cannot be audited, repeated, or honestly killed, and the same team will run the same ninety minutes again next quarter with new enthusiasm and the same missing evidence. The fix is unglamorous: freeze the verdict before the first trial, and make a machine read it back to you.

Why a leaderboard cannot answer a repository question

Public benchmarking work, including the recent wave of Kaggle-style challenges that DEV has been amplifying this week, produces something genuinely useful: a signal about how models behave on shared tasks under shared scoring. That signal is about the model, not about your repository, your module boundaries, your flaky fixtures, or the one test that always fails after a timezone change. A high leaderboard position is a reason to run a spike, never a reason to skip one.

So the question a spike should answer is deliberately narrow and local. For one repository, one failure class, one attempt per failure, and one afternoon, does the assistant produce a test-green patch that stays inside the module it was asked to touch. That sentence contains everything a grader needs, which means it can be compiled rather than remembered.

Writing the registration first

Pre-registration in clinical research means publishing the analysis plan before collecting data, so that a disappointing result cannot quietly become a different hypothesis. The engineering version costs about twelve minutes and lives in a single JSON file committed before the first prompt is sent. The fields are chosen so that a stranger could grade the spike without talking to anyone on the team.

{
  "hypothesis": "A coding agent turns a failing regression test green on the first attempt for at least half of the sampled failures in this repo, without editing files outside the failing module.",
  "metric": "first_attempt_pass_rate",
  "ship_if": { "min_pass_rate": 0.5, "max_scope_violations": 1 },
  "kill_if": { "max_pass_rate": 0.34 },
  "min_trials": 12,
  "registered_at": "2026-11-07T09:00:00Z",
  "repo_commit": "REPLACE_WITH_HEAD_SHA"
}

Two details carry most of the integrity. The min_trials floor stops a team from stopping early on a lucky streak, and the gap between the ship bar and the kill bar creates an honest middle zone where the answer is genuinely unknown. registered_at is an ISO-8601 UTC timestamp, and it is the field that makes the whole exercise falsifiable rather than decorative.

The compiler that grades the spike

The grader is short enough to read in full during the spike itself, which matters more than elegance. It refuses to grade anything if the first trial started before the registration was written, and it returns three exit codes so that shell scripts and humans can both consume the answer.

#!/usr/bin/env python3
"""Grade a 90-minute spike against a rule that was frozen before it ran."""
import json
import sys
from datetime import datetime
from pathlib import Path


def parse_ts(value: str) -> datetime:
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def main() -> int:
    reg = json.loads(Path("preregistration.json").read_text())
    runs = [json.loads(line) for line in Path("runs.jsonl").read_text().splitlines() if line.strip()]

    registered_at = parse_ts(reg["registered_at"])
    if runs and parse_ts(runs[0]["started_at"]) < registered_at:
        print("INCONCLUSIVE: first trial predates the registration")
        return 2

    if len(runs) < reg["min_trials"]:
        print(f"INCONCLUSIVE: {len(runs)} of {reg['min_trials']} required trials")
        return 2

    passed = sum(1 for r in runs if r["test_green"])
    rate = passed / len(runs)
    scope = sum(1 for r in runs if r["scope_violation"])
    print(f"trials={len(runs)} pass_rate={rate:.2f} scope_violations={scope}")

    if rate >= reg["ship_if"]["min_pass_rate"] and scope <= reg["ship_if"]["max_scope_violations"]:
        print("SHIP: the hypothesis survived")
        return 0
    if rate <= reg["kill_if"]["max_pass_rate"]:
        print("KILL: the hypothesis did not survive")
        return 1
    print("INCONCLUSIVE: between the two bars; do not round up")
    return 2


if __name__ == "__main__":
    sys.exit(main())

Exit code 0 means ship, 1 means kill, and 2 means the evidence is insufficient. That third code does the quiet work, because it gives a team permission to say the spike did not settle the question without anyone losing face. A 2 is a real result and should be reported as one.

The loop that produces graded evidence

The runner below is a skeleton rather than a polished tool, and the point is the shape of the loop: detach a disposable worktree, ask once, apply the patch, run the test command, record a line. Every trial is independent, so a failure cannot contaminate the next one through leftover files or stale build output.

import datetime
import json
import os
import subprocess
import tempfile
import urllib.request
import uuid
from pathlib import Path

BASE = os.environ["SPIKE_BASE_URL"].rstrip("/")
KEY = os.environ["SPIKE_API_KEY"]
MODEL = os.environ["SPIKE_MODEL"]
REPO = Path(os.environ.get("SPIKE_REPO", ".")).resolve()
MODULE = os.environ["SPIKE_MODULE"]
TEST_CMD = os.environ.get("SPIKE_TEST_CMD", "python -m pytest -x -q")


def chat(prompt: str) -> str:
    payload = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json", "Authorization": f"Bearer {KEY}"},
    )
    with urllib.request.urlopen(req, timeout=120) as resp:
        return json.load(resp)["choices"][0]["message"]["content"]


def run_trial(test_id: str, failure_log: str) -> dict:
    started = datetime.datetime.now(datetime.timezone.utc).isoformat()
    with tempfile.TemporaryDirectory() as tmp:
        subprocess.run(["git", "worktree", "add", "--detach", tmp, "HEAD"], cwd=REPO, check=True)
        try:
            diff = chat(f"Fix the failing test {test_id}. Return a unified diff only.\n\n{failure_log}")
            patch = Path(tmp, "agent.patch")
            patch.write_text(diff)
            applied = subprocess.run(["git", "apply", "--whitespace=nowarn", str(patch)], cwd=tmp)
            green = applied.returncode == 0 and subprocess.run(
                TEST_CMD, shell=True, cwd=tmp, capture_output=True
            ).returncode == 0
            changed = subprocess.run(
                ["git", "diff", "--name-only"], cwd=tmp, capture_output=True, text=True
            ).stdout.split()
            return {
                "trial": str(uuid.uuid4()),
                "test_id": test_id,
                "started_at": started,
                "test_green": green,
                "scope_violation": any(not f.startswith(MODULE) for f in changed),
            }
        finally:
            subprocess.run(["git", "worktree", "remove", "--force", tmp], cwd=REPO)

The endpoint shape above assumes an OpenAI-compatible chat completions route, and teams whose provider differs should swap the adapter and leave the rest untouched. Appending each returned dict as one line to runs.jsonl keeps the ledger append-only, so nobody can delete an embarrassing trial without leaving a visible gap in the timestamp sequence.

The harness needs two things: a model endpoint and somewhere to run it that is not somebody's laptop. That second requirement is where a free server option is worth testing rather than assuming, and this is the one place where the choice of provider actually changes the spike's design. MonkeyCode's free model access and free server option are the arrangement this account has been evaluating for exactly this kind of time-boxed experiment, so the runner can point at those credentials while the verdict logic stays provider-agnostic. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Ninety minutes, spent honestly

The first fifteen minutes go to reading the failure class, writing the registration, committing it, and capturing the repository commit hash. The next fifty-five are the loop, run without deciding mid-flight that the sample is big enough, and the last twenty are the grader plus a short exit note recording what was and was not measured. That is a boring allocation, and boredom is the property you want, because a spike that keeps getting exciting is usually a spike that keeps changing its question.

What this method cannot tell you

Twelve trials on one repository is a smoke test, not a benchmark, and the confidence interval around a small pass rate is wide enough to swallow most commercial claims. The method never measures review cost, which is the real budget line, and it says nothing about whether a green patch is a good patch that a maintainer would merge. Sample failures chosen by a human also carry selection bias, so the sampling rule should be written down alongside the hypothesis.

Teams that already run a real evaluation harness should not adopt this, because a pre-registration file will look like a toy next to their infrastructure. Teams making a decision that is expensive to reverse, or one touching regulated or safety-critical code, need far more than ninety minutes and a dozen trials. And anyone who wants a spike to produce a demo for a stakeholder should run a demo instead and label it honestly, because this format is built to end conversations, not to start applause.

Run it once on a question you genuinely do not know the answer to, and let the exit code tell you something you did not want to hear. If you would rather not provision anything first, the free model access and free server option are one low-friction way to get the harness running this afternoon, though any published allowance should be confirmed at signup rather than trusted from a blog post, including this one.