Receipts for Your Agent Spike: A 90-Minute Harness That Records Every Patch

Most agent evaluations end as a feeling. "It seemed faster." Feelings do not survive a code review.

A 90-minute spike should end with a file you can read. This post builds that file: one JSON line per attempt, containing the diff size, the test result, and — most important — what the test does when you remove the patch.

The failure mode: tests that pass without the fix

An agent writes a patch. The suite goes green. You call it a win.

Now revert the patch and run the suite again. If it is still green, your test never guarded the bug. The agent got credit for a test that proves nothing.

This single check separates real acceptance from coincidence. Without it, your hit rate is inflated and you will not know by how much.

Write the hypothesis before the clock starts

One sentence. One bug class. One repo. Commit it before the first attempt.

Within 90 minutes and 5 attempts, the agent produces patches for <bug class> that pass <test command> and fail the same command when reverted, in at least 2 of 5 attempts.

Two properties matter here:

1. The guard clause is in the hypothesis. "Passes tests" alone is not a hypothesis. It is a hope.
2. The denominator is fixed. Five attempts, not "until it works." Stop conditions written in advance are the only ones that hold.

Store the hypothesis next to the ledger. Future you will not remember the wording.

Where to run it

The harness rewrites your working tree twice per attempt. Do not run it in CI and do not run it in your main checkout.

Use a disposable environment. That is the practical value of a free server option: the spike can thrash a machine you do not care about, and the only artifact you copy back out is .spike/receipts.jsonl.

MonkeyCode, which its team describes as an open-source coding-agent project, is the tool under test in this workflow. The operator states that the free tier currently includes free model access — an allowance of 10 million tokens — and a free server option, as of 2026-11-09. Free-tier terms move; confirm the current allowance and server limits on the project page before you plan around them, and do not treat this date as a guarantee.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Before attempt one, print the runtime versions inside the free server and next to your CI config. If Node, Python, or the package manager differ, your guard results describe the server, not your repo.

node -v && python3 -V && git --version

The harness: receipts.sh

One script, one attempt, one line of output. It captures the diff, runs the suite, then reruns the suite with the patch stashed.

#!/usr/bin/env bash
# receipts.sh — record one agent attempt as a single JSON line.
# usage: SPIKE_TOKENS=12000 ./receipts.sh run-03 null-guard "npm test --silent"
set -euo pipefail

RUN_ID="${1:?run id}"
LABEL="${2:?label}"
TEST_CMD="${3:-npm test --silent}"

ROOT="$(git rev-parse --show-toplevel)"
OUT="$ROOT/.spike/receipts.jsonl"
mkdir -p "$(dirname "$OUT")"

BASE_REF="$(git rev-parse HEAD)"
START=$(date +%s)

# 1. capture the diff first, before anything mutates the tree
git diff --binary > "$ROOT/.spike/$RUN_ID.diff"
DIFF_LINES=$(wc -l < "$ROOT/.spike/$RUN_ID.diff" | tr -d ' ')
FILES=$(git diff --name-only | grep -c . || true)
TOKENS="${SPIKE_TOKENS:-null}"

# 2. run the suite with the patch applied
set +e
eval "$TEST_CMD" > "$ROOT/.spike/$RUN_ID.test.log" 2>&1
TEST_RC=$?
set -e

# 3. guard run: same command with the patch removed
set +e
git stash push --include-untracked -q
eval "$TEST_CMD" > "$ROOT/.spike/$RUN_ID.guard.log" 2>&1
GUARD_RC=$?
git stash pop -q
set -e

END=$(date +%s)

jq -nc \
  --arg run "$RUN_ID" --arg label "$LABEL" --arg base "$BASE_REF" \
  --argjson diff_lines "$DIFF_LINES" --argjson files "$FILES" \
  --argjson test_rc "$TEST_RC" --argjson guard_rc "$GUARD_RC" \
  --argjson tokens "$TOKENS" --argjson secs "$((END-START))" \
  '{run:$run,label:$label,base:$base,diff_lines:$diff_lines,files:$files,
    test_rc:$test_rc,guard_rc:$guard_rc,tokens:$tokens,seconds:$secs,
    accepted:($test_rc == 0 and $guard_rc != 0)}' >> "$OUT"

tail -n 1 "$OUT"

Four properties make this small enough to trust:

• The diff is captured first. Later steps cannot contaminate the record.
• Both runs are logged, not summarized. Exit codes are cheap to store and impossible to misremember.
• accepted is a single boolean. Pass the test, fail without the patch. No partial credit.
• Token count is optional. Unset it and you get null instead of a wrong number.

Treat this as a template. Adapt the test command, the paths, and the token capture to your stack before the timer starts.

Run the attempts

One loop iteration per attempt, in this order:

1. Ask the agent for the fix. Note the tokens it reports, if it reports them.
2. Review the diff by eye for anything unreviewable — new dependencies, deleted tests, network calls.
3. Run receipts.sh immediately, before you edit anything by hand.
4. git checkout -- . and git clean -fd before the next attempt.

Step 3 is the one people skip. Hand edits after an attempt erase the evidence that attempt produced.

Summarize in one command

jq -rs '
  group_by(.label)[] |
  {label: .[0].label,
   attempts: length,
   accepted: (map(select(.accepted)) | length),
   discarded: (map(select(.test_rc == 0 and .guard_rc == 0)) | length),
   median_secs: (map(.seconds) | sort | .[length/2 | floor]),
   max_diff_lines: (map(.diff_lines) | max)}
' .spike/receipts.jsonl

You get a table at minute 90. Not a memory. Not a Slack thread.

Read the table: ship, narrow, or kill

Accepted / attempts	Guard-clean	Reading	Action
4–5 of 5	Yes	The task is agent-shaped for this bug class	Adopt, then widen the bug class in a new spike
2–3 of 5	Yes	Partially shaped; scope drifted	Narrow the hypothesis to one file or one function
0–1 of 5	Yes	Not shaped, or the tests are too weak	Kill the workflow for this class
Any	No	Tests pass without the patch	Discard the run, fix the test, restart the clock

Two rules keep the table honest:

• Discard before you divide. A run with test_rc == 0 and guard_rc == 0 is a broken test, not a win. Remove it from both numerator and denominator.
• Do not re-run the losing attempts. "One more try" is how a 90-minute spike becomes a two-day detour with no ledger.

What the numbers do not tell you

• Small samples. Five attempts is an anecdote with a timestamp. Fifteen is a weak signal. Neither is a benchmark.
• Flaky suites. A test that fails 5% of the time fakes both directions. Run the suite twice on a clean tree before you start.
• Stash artifacts. If the patch adds untracked files the suite needs, the guard run fails for the wrong reason and looks like a clean win. Check the guard log when a result surprises you.
• Diff size is not review cost. A 12-line change to a config loader can cost more review time than a 200-line rename.
• Token totals are approximate. Agent-reported usage rarely matches billed usage exactly. Use it as a ratio, not an invoice.
• No security review. Your suite catching the bug says nothing about injected commands, new dependencies, or credential handling.

Who should skip this

• Slow suites. If one run takes over three minutes, the guard check doubles every attempt and you will not finish five.
• No single test per bug class. Without a test that should fail, the guard column is meaningless.
• Revert-unsafe repos. Migrations, generated lockfiles, and prod config are not stash material.
• Solo review. A harness does not replace a second pair of eyes on anything touching auth or payments.

The 90 minutes is the product

The timer is not a gimmick. It is what makes a kill cheap enough to accept.

Minute 90 arrives whether or not you have an answer. The only question is whether you read the answer from a file or reconstruct it from memory.

If you want to run this against the free tier, check the current token allowance and server limits on the MonkeyCode project page first — the numbers in this post are dated, and yours may already differ.