The Twin-Run Test: A 90-Minute Spike That Proves a Task Is Agent-Shaped

Most AI spikes test the model. They should test the task. One run tells you a patch exists. Two runs tell you the task is stable enough to hand off. This post is a 90-minute protocol for the second question.

The core claim: if two independent agent runs on the same task disagree wildly, the problem is your task description, not the model. Ship-or-kill evidence should measure that disagreement, not how good one diff looks.

The hypothesis, written before you touch anything

One sentence. One falsifiable claim. Written in a file, committed before the first prompt.

spike/hypothesis.md

Hypothesis: "Add retry with backoff to the S3 uploader" is agent-shaped.
Prediction: two independent runs from the same base commit
            change overlapping code regions and pass the same tests.
Kill if: overlap < 0.15, or either run fails the existing suite.

If you cannot phrase the kill condition as a number, you are not spiking. You are browsing.

Why a single run is weak evidence

A passing patch from run A has three possible causes:

• The task is genuinely well-specified.
• The model got lucky on one sample.
• The task is easy and any approach works.

Only the third case is harmless. The other two mislead you into adopting a workflow that will not survive the next ticket.

Contamination makes it worse. If run B starts from run A's leftover branch, caches, or a partially edited tree, run B is not independent. It is a continuation. Most "the second run agreed with me" reports are this failure in disguise.

The 90-minute budget

Window	Minutes	Output
Pre-register	0–5	hypothesis.md committed
Run A	5–40	Branch spike/run-a, tests green or red
Run B	40–75	Branch spike/run-b, from the same base commit
Compare	75–90	Overlap number + decision table row

Two hard rules. Stop at 90 minutes regardless of state. Both runs start from the identical base commit.

Free server as a clean room

Independence is the whole experiment. You need two throwaway workspaces, and neither may inherit state from the other.

This is where a free server option earns its place. A disposable hosted workspace means run B starts from a cloned repository, not from your machine's dirty tree, warm caches, or uncommitted fixes you forgot about.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

At the time of writing, the project is described as open source and advertises free model access plus a free server option. Do not take that last sentence as a spec sheet. Open the dashboard and confirm the current terms before you plan around them. Quotas and availability change, and your spike should not depend on a number you read in a blog post.

Local worktrees are an acceptable substitute when you want zero network variables:

git worktree add /tmp/run-a spike/run-a
git worktree add /tmp/run-b spike/run-b

Two paths, two clean checkouts, one base commit. Same guarantee, less convenience.

The artifact: hunk overlap in one file

Save this as spike/hunk_overlap.py. It buckets changed lines into 20-line windows and compares the two runs.

import re, subprocess, sys

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")

def hunks(base: str, tip: str) -> set[tuple[str, int]]:
    diff = subprocess.run(
        ["git", "diff", "-U0", f"{base}...{tip}"],
        capture_output=True, text=True, check=True,
    ).stdout
    found, path = set(), None
    for line in diff.splitlines():
        if line.startswith("diff --git"):
            path = line.split(" b/", 1)[-1]
        m = HUNK.match(line)
        if m and path:
            # 20-line buckets so near-misses still count as overlap
            found.add((path, int(m.group(1)) // 20))
    return found

base, a, b = sys.argv[1], sys.argv[2], sys.argv[3]
ha, hb = hunks(base, a), hunks(base, b)
inter, union = ha & hb, ha | hb
score = len(inter) / len(union) if union else 0.0
print(f"runA={len(ha)} runB={len(hb)} shared={len(inter)}")
print(f"jaccard={score:.2f}")
print("suggestion=narrow" if score < 0.15 else "suggestion=keep")

Run it after both branches exist:

git fetch origin main
git branch -f spike/run-a 9c1f2ab
git branch -f spike/run-b 9c1f2ab
python3 spike/hunk_overlap.py origin/main spike/run-a spike/run-b

Then gate on tests, not vibes. Both branches must pass the suite you already trust:

for wt in /tmp/run-a /tmp/run-b; do
  ( cd "$wt" && ./scripts/test.sh ) && echo "PASS $wt" || echo "FAIL $wt"
done

This script is a worked example, not a benchmark result. No measured numbers are implied by the code above.

The decision table

Run A tests	Run B tests	Overlap	Decision
pass	pass	≥ 0.50	Ship the smaller diff. Task is agent-shaped.
pass	pass	0.15–0.49	Narrow the task and re-run once.
pass	pass	< 0.15	Kill. Your description is doing too little work.
pass	fail	any	Kill for now. Test shape is unstable.
fail	fail	any	Stop. Wrong task for automated patch generation.
fail	pass	any	Investigate the failing run before rerunning anything.

Map every row to an action before you start. Post-hoc interpretation is how spikes turn into opinions.

Token budget as a planning number, not a promise

The operator-supplied figure for the free tier is 10,000,000 tokens. Treat it as a ceiling for planning, and measure your own consumption.

Record two numbers per run: tokens consumed, and hunks produced. The second is the useful one.

Illustrative arithmetic only — these are placeholders, not measurements:

tokens_per_run   = 40,000   # replace with your dashboard reading
runs_in_budget   = 10_000_000 / 40_000 = 250
twin_run_spikes  = 250 / 2   = 125

If your real reading is ten times higher, your spike count drops tenfold. That is the number you need before promising a team trial. Pull it from the dashboard, not from this post.

Failure modes that will fool you

• Shared base drift. Run B branched after run A merged something. Overlap becomes meaningless.
• Overlap without tests. Two runs can agree on a wrong fix. The gate is the suite, not the score.
• Bucket blindness. A 20-line window hides a one-line semantic difference. Read both diffs at the end.
• Prompt edits mid-run. Any change to instructions between runs makes them incomparable.
• Time overrun. Past 90 minutes you start optimizing the artifact instead of answering the question.

Limitations and who should skip this

This protocol needs an automatable test suite. If your acceptance criteria live in a design review, the twin-run test measures nothing useful.

It also fits poorly for:

• Security-sensitive changes, where two plausible patches both need human review anyway.
• Trivial single-line fixes, where overlap is almost guaranteed and uninformative.
• Tasks with unresolved product decisions. Two runs will diverge for the right reason.
• Regulated code paths where a disposable free workspace is not permitted at all.

The overlap metric is deliberately crude. It detects gross specification gaps, not subtle ones. A high score is permission to continue, never proof of correctness.

The habit worth keeping

One hypothesis, two clean runs, one number, one row in a table. That is a spike that produces evidence you can defend in a review.

If you run the twin-run test on your own backlog this week, I would like to know which task failed it first — the underspecified ones are usually the tickets nobody wanted to pick up anyway.