A 90-Minute Spike to Decide if a Free Coding-Agent Tier Belongs in Your Repo

Free agent tiers are easy to try and hard to judge. A green demo patch proves almost nothing. You need one number you can defend later.

This post gives you a 90-minute spike. One hypothesis. One harness. A ship-or-kill verdict at the end.

The artifact is three small files. A contract, a runner, a summariser. Nothing here depends on a specific vendor.

The hypothesis must be falsifiable

Write the claim before you run anything. Ambiguity after the fact is how spikes become anecdotes.

Bad: "This agent seems decent on our codebase." Good: "A free tier turns a failing test green in under 10 minutes, 4 tries out of 5, with no human edits."

The second version names a unit, a count, a time, and a failure condition. It can lose.

Pre-register three numbers

1. pass@1 — fraction of tasks green on the first attempt.
2. median green seconds — how long a successful run takes, end to end.
3. stray edits — attempts that touch files outside the task's declared module.

Pick thresholds now. Not after you see the output. Moving a threshold mid-spike is the same as deleting it.

What the spike actually measures

Pass rates alone are misleading. A slow pass costs you review attention. A noisy pass costs you trust.

So the runner records every attempt, including red ones. Timestamps come from date +%s, not from the agent's self-report.

The unit of work is a fixture repo with a known failing test. Reset between attempts. Same starting state, every time.

Pick two tasks, not twenty

Two tasks fit in 90 minutes with five attempts each. Twenty tasks do not. Breadth is a later experiment.

Choose one task that looks mechanical. Choose one that requires reading existing code. The gap between them is the interesting signal.

The spike contract

Keep the claim and the thresholds in version control. A YAML file is enough.

# spike.contract.yaml
hypothesis: >
  A free tier turns a failing test green in under 10 minutes,
  4 of 5 first attempts, with no human edits.
deadline_minutes: 90

tasks:
  - id: t1
    kind: mechanical
    test: pytest -q tests/test_pagination.py::test_cursor_wrap
    allowed_paths: ["src/pagination/"]
  - id: t2
    kind: read-the-code
    test: pytest -q tests/test_cache.py::test_eviction_under_pressure
    allowed_paths: ["src/cache/"]

thresholds:
  pass_at_1_min: 0.80
  median_green_seconds_max: 600
  stray_edits_max: 0

kill_if:
  - "any task burns 20 minutes with no green test"
  - "any attempt edits a file outside allowed_paths"
  - "the 90-minute deadline is reached with fewer than 6 recorded attempts"

The kill_if block matters more than the thresholds. It stops you from "just one more try" at minute 85.

The runner

This is a bash harness. It enforces the deadline, resets the fixture, and appends one JSON line per attempt.

#!/usr/bin/env bash
# spike-run.sh — deadline-enforced evaluation harness.
set -uo pipefail

BUDGET_MIN=${BUDGET_MIN:-90}
DEADLINE=$(( $(date +%s) + BUDGET_MIN * 60 ))
LOG="spike-$(date -u +%Y%m%dT%H%M%SZ).jsonl"
mkdir -p out attempts
: > "$LOG"

record() { # task attempt seconds status files_touched
  printf '{"task":"%s","attempt":%s,"seconds":%s,"status":"%s","files_touched":%s}\n' \
    "$1" "$2" "$3" "$4" "$5" >> "$LOG"
}

reset_fixture() {
  git checkout -- . >/dev/null 2>&1
  git clean -fd src/ >/dev/null 2>&1
}

run_task() {
  local id=$1 test_cmd=$2 attempt=$3 t0 t1 rc files
  reset_fixture
  t0=$(date +%s)

  # Replace with your agent's non-interactive invocation.
  ${AGENT_CMD:?set AGENT_CMD} > "out/${id}.${attempt}.agent.log" 2>&1
  rc=$?

  eval "$test_cmd" > "out/${id}.${attempt}.test.log" 2>&1 || rc=1
  t1=$(date +%s)
  files=$(git status --porcelain | wc -l | tr -d ' ')

  if [ "$rc" -eq 0 ]; then
    record "$id" "$attempt" $((t1 - t0)) green "$files"
  else
    record "$id" "$attempt" $((t1 - t0)) red "$files"
  fi
}

TASKS=(
  "t1:pytest -q tests/test_pagination.py::test_cursor_wrap"
  "t2:pytest -q tests/test_cache.py::test_eviction_under_pressure"
)

for attempt in 1 2 3 4 5; do
  for entry in "${TASKS[@]}"; do
    id=${entry%%:*}
    cmd=${entry#*:}
    if [ "$(date +%s)" -ge "$DEADLINE" ]; then
      record "$id" "$attempt" 0 deadline_hit 0
      continue
    fi
    run_task "$id" "$cmd" "$attempt"
  done
done

echo "wrote $LOG"

Two details are load-bearing. AGENT_CMD is whatever your client exposes for one-shot runs. And reset_fixture must restore the failing state, or attempt 2 starts from attempt 1's patch.

The summariser

Raw JSONL is not a verdict. Compute exactly three numbers and apply the contract.

#!/usr/bin/env python3
# summarise.py <spike.jsonl>
import json, statistics, sys

rows = [json.loads(l) for l in open(sys.argv[1]) if l.strip()]
first = [r for r in rows if r["attempt"] == 1]
green = [r for r in rows if r["status"] == "green"]

TASKS = 2
top = {r["task"] for r in first if r["status"] == "green"}
pass_at_1 = len(top) / TASKS
median_s = statistics.median([r["seconds"] for r in green]) if green else float("inf")
stray = [r for r in rows if r["files_touched"] > 2]

print(f"attempts={len(rows)} green={len(green)}")
print(f"pass_at_1={pass_at_1:.2f} median_green_s={median_s:.0f} stray={len(stray)}")

ship = pass_at_1 >= 0.80 and median_s <= 600 and not stray
print("verdict:", "SHIP" if ship else "KILL")

Adjust TASKS to match the contract. Do not add a "close enough" branch. The point of pre-registration is that the script cannot negotiate.

Running it on a clean box

The spike should not run on your laptop. Editor state, local caches, and a half-installed toolchain are all confounders.

MonkeyCode is an open-source coding-agent project with free model access and a free server option. I used that arrangement to keep the fixture repo isolated from my working machine.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The project advertises a free token allowance alongside the free server. Check the current numbers on the project page before you plan around them. Quotas and model lineups change, and none of them change the method above.

Three setup rules for the remote run:

• Pin the runtime: same Python, same pytest, same lockfile.
• Keep allowed_paths narrow, then diff the whole tree afterwards.
• Copy the JSONL off the box at minute 90. Do not extend.

Reading the result

The table below shows decision shapes. These are not measured results from any specific tier.

pass@1	median green	stray edits	Reading	Action
≥ 0.8	≤ 10 min	0	The claim held	Run a wider spike on real tickets
≥ 0.8	10–20 min	0	Passes, but costs wall clock	Sample 3 real tickets before adopting
< 0.8	any	0	Fails on first sight	KILL — retry cost is on you
any	any	> 0	Touches files it was not given	KILL — review burden is unbounded

Row four is the one people skip. A wrong-but-plausible patch in an unrelated module is worse than a red test. Red tests are cheap.

The three ways this spike misleads you

1. Two tasks are not a distribution. You measured a fixture, not your backlog.
2. Green is not correct. A test can pass on a patch a senior reviewer would reject.
3. Free tiers are moving targets. The thing you measured may not exist next month.

Write those three lines into the report. They keep the next reader honest.

Who should not run this

• Teams whose code cannot leave their network. The remote-server variant is off the table.
• Repos without a fast test suite. If a single run takes four minutes, 90 minutes buys you nothing.
• Anyone unwilling to write the thresholds first. Without them, this becomes a vibe check with extra steps.
• Anyone who needs a capacity number today. A spike yields a decision, not a benchmark.

The 90-minute shape

• 0–10 min: fixture repo, failing test, contract committed.
• 10–20 min: harness run once manually, then reset verified.
• 20–80 min: ten attempts, hands off the keyboard.
• 80–90 min: summarise.py, write the verdict, delete nothing.

Keep the JSONL. Six weeks from now, when someone asks why you did or did not adopt a free tier, that file is your answer. It is also the only part of this spike you cannot reconstruct from memory.

If you want a clean box for attempt one, MonkeyCode's free model access and free server are a reasonable place to start. Bring your own thresholds.