Accepted Diffs Per Hour: A 90-Minute Spike on a Free Model Tier

Conclusion first: the useful agent metric is not tokens per second. It is accepted diffs per hour.

A diff only counts if a verifier accepts it. Everything else is rejected work.

This post is one time-boxed spike. One hypothesis. One decision rule written before the run.

The hypothesis

State it as a single falsifiable sentence. Write it down before you touch a terminal.

On a free model tier, at least 40% of agent diffs on a fixed 12-task set pass an automated verifier within 90 minutes.

If that sentence is wrong, you kill the idea. You do not tune prompts for another week.

Two numbers matter during the spike:

1. Accept rate — accepted diffs divided by attempted diffs.
2. Accepted diffs per hour — the rate that survives a verifier, per wall-clock hour.

Tokens are an input cost. Accepted diffs are the output. Measure the output.

Pre-register the decision rule

Write the rule first. Otherwise you will rationalize whatever happens.

Outcome at minute 90	Decision
Accept rate >= 50%	Proceed to a wider task set
Accept rate 25-49%	Narrow the task set, re-run once
Accept rate < 25%	Kill the workflow
Fewer than 8 attempts	Kill — the harness failed, not the model

Note the last row. A broken harness looks identical to a weak model. Cap your own failure first.

The harness

Keep it boring. A git worktree per task, a patch capture, and a verifier exit code.

Task fixture

Each task is one prompt plus one verify command. Nothing else.

{
  "agent_cmd": "your-agent-cli --print --no-interactive",
  "per_task_timeout_s": 120,
  "tasks": [
    {
      "name": "tz-offset",
      "prompt": "Fix the off-by-one in parse_offset; do not change public signatures.",
      "verify": "python -m pytest tests/test_offsets.py -q"
    },
    {
      "name": "retry-backoff",
      "prompt": "Add capped exponential backoff to fetch_page; keep the retry count unchanged.",
      "verify": "python -m pytest tests/test_fetch.py -q"
    }
  ]
}

Fixture rules that keep the spike honest:

• The verifier must fail on HEAD before the agent runs. Prove it.
• The verifier must be cheap, under 10 seconds per task.
• One task, one intent. No "refactor the module".
• Cap the diff scope in code, not in your judgment. Three files maximum here.

The runner

#!/usr/bin/env python3
"""spike.py - measure accepted diffs per hour for one agent command."""
import json, pathlib, shlex, subprocess, time

ROOT = pathlib.Path(__file__).parent
LOG = ROOT / "results.jsonl"
MAX_FILES = 3

def sh(cmd, cwd, timeout=None):
    try:
        p = subprocess.run(cmd, cwd=cwd, shell=True, capture_output=True,
                           text=True, timeout=timeout)
        return p.returncode, p.stdout, p.stderr
    except subprocess.TimeoutExpired:
        return 124, "", "timeout"

def one_task(task, agent_cmd, budget_s):
    wt = ROOT / "worktrees" / task["name"]
    sh(f"git worktree add -f {wt} HEAD", ROOT)
    t0 = time.time()
    rc, out, err = sh(f"{agent_cmd} {shlex.quote(task['prompt'])}", wt, timeout=budget_s)
    wall = round(time.time() - t0, 1)
    _, diff, _ = sh("git diff --no-color", wt)
    touched = [l for l in diff.splitlines() if l.startswith("diff --git")]
    if rc == 124:
        verdict, why = "reject", "timeout"
    elif not diff.strip():
        verdict, why = "reject", "empty_diff"
    elif len(touched) > MAX_FILES:
        verdict, why = "reject", f"scope_{len(touched)}_files"
    else:
        rc_v, _, _ = sh(task["verify"], wt, timeout=120)
        verdict = "accept" if rc_v == 0 else "reject"
        why = "verifier_pass" if rc_v == 0 else "verifier_fail"
    return {"task": task["name"], "verdict": verdict, "reason": why,
            "wall_s": wall, "files": len(touched), "agent_rc": rc}

def main():
    cfg = json.loads((ROOT / "spike.json").read_text())
    deadline = time.time() + 90 * 60
    for task in cfg["tasks"]:
        if time.time() > deadline:
            break
        rec = one_task(task, cfg["agent_cmd"], cfg["per_task_timeout_s"])
        with LOG.open("a") as f:
            f.write(json.dumps(rec) + "\n")
        print(rec["task"], rec["verdict"], rec["reason"])

if __name__ == "__main__":
    main()

Run it, then read the log:

git worktree list
jq -s '{attempts: length, accepted: map(select(.verdict=="accept")) | length}' results.jsonl
jq -r 'select(.verdict=="reject") | .reason' results.jsonl | sort | uniq -c | sort -rn

Cleanup is part of the spike:

git worktree remove --force worktrees/*

The log schema is fixed, so a record looks like this — illustrative shape, not a measurement:

{"task":"tz-offset","verdict":"reject","reason":"verifier_fail","wall_s":41.2,"files":2,"agent_rc":0}

Do not paste invented numbers into a write-up. Run the harness and publish your own log.

Where the free tier fits

Not every spike deserves a spend approval. A free entry point removes that friction.

The operator states two things about MonkeyCode: free model access, and a free server option. The same operator described a free allowance of 10M tokens. I have not independently audited those numbers, and I am not quoting model names or throughput here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

How I use those two pieces in this spike:

• Free model access answers the hypothesis above. If the accept rate is bad, I have lost 90 minutes, not a budget line.
• The free server option is where I would run the verifier pass, so the worktree churn stays off my laptop. Treat machine specs and uptime as unverified until you test them yourself.

If your accept rate clears 50%, that is the moment to consider a paid tier. Not before.

Kill criteria in practice

Most spikes fail on the harness, not the model. Check these before you blame the agent.

1. Did the verifier fail on clean HEAD? If not, every accept is fake.
2. Did any task exceed the 120-second timeout? Count timeouts as rejects, always.
3. Did the agent touch files outside the intent? The file cap catches this.
4. Is the accept rate under 25% with zero timeouts? That is a real kill signal.

Report the reason histogram next to the accept rate. empty_diff and verifier_fail mean different things.

Limitations

• Twelve tasks on one repository is a signal, not a benchmark.
• The verifier is the bottleneck. A weak test suite inflates the accept rate.
• Free-tier throughput and stability are environment-dependent. Rate limits can end a run early.
• This spike says nothing about long-horizon work, multi-file refactors, or production safety.

Who should not use this

• Teams that cannot write a fast, deterministic verifier. Skip the spike; fix the tests first.
• Anyone planning to point an agent at a production-connected repo during a measurement run.
• Anyone who needs an audit trail, signed artifacts, or compliance evidence from day one.
• Anyone who will treat one 90-minute run as proof about a whole product.

The next 90 minutes

Write the hypothesis. Pre-register the kill rule. Ship or kill at minute 90 with a log file as evidence.

The free tier is one cheap way to get those 90 minutes. The harness is the part that transfers.