Time to First Verifiable Diff

The onboarding number I actually trust is not how long the install takes, but how long it takes before a tool hands me a diff I can verify. Total install time tells you that something went wrong somewhere in the chain; it never tells you which link to cut, and that gap is why so many evaluations end in vibes. I keep running into conversations about AI fatigue, and I think a large share of it is not model quality at all, but onboarding friction that nobody bothered to localize. So I built a small instrument that localizes it, and the number it produces changed how I trial tools.

Earlier I wrote about clocking the first fifteen minutes, and I still believe the clock is worth holding. The problem is that a duration without a per-step ledger is a verdict, not a diagnosis. If the whole run took eleven minutes, do I fix the signup, the runtime, the docs, or my own prompt? You cannot answer that from a single stopwatch reading, and guessing wrong is how people conclude a tool is slow when it was actually their network.

Record the steps, then rank them

The harness below is a template, not a product wrapper, and I am not going to pretend I measured anything for you here. Every number it prints comes from your own first run, on your own machine, against your own network, which is the only measurement that transfers. It writes one JSON line per phase transition, so the ledger is append-only and replayable after the fact. Here is the recorder, with placeholder commands where the vendor-specific work would go.

#!/usr/bin/env bash
# step-ledger.sh — illustrative harness, not tied to any vendor CLI.
set -euo pipefail
LEDGER="${LEDGER:-./onboarding.jsonl}"
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)"

stamp() { # stamp <step> <phase> [note]
  printf '{"run":"%s","step":"%s","phase":"%s","ts":"%s","note":"%s"}\n' \
    "$RUN_ID" "$1" "$2" "$(date -u +%Y-%m-%dT%H:%M:%S.%3NZ)" "${3:-}" >> "$LEDGER"
}
begin() { stamp "$1" begin "${2:-}"; }
end()   { stamp "$1" end   "${2:-}"; }

begin signup  "no card, real email"
# ... whatever the tool asks for ...
end   signup

begin install "docs command, copy-pasted verbatim"
# ... paste the installer line from the docs ...
end   install

begin firsttask "ask for a one-line fix, nothing else"
# ... send the request and wait for output ...
end   firsttask

The rules matter more than the code. Copy the documented command verbatim, because paraphrasing from memory hides the moment the docs failed. Tag each step with a short note about what happened, since the note is what turns a bar on a chart into an actionable complaint. Never edit the ledger by hand, because a hand-edited measurement is a story, and stories are exactly what this exercise exists to replace.

Turn the log into a ranked complaint

A raw log is still just a log, so the second half of the instrument ranks the steps by cost and prints a total. Sorting descending is deliberate: the top line is the fix you should try first, and everything below it is noise until that one is addressed. The printer is short enough to read in one pass, which matters because you will want to trust it.

#!/usr/bin/env python3
"""ledger.py — turn onboarding.jsonl into a ranked friction ledger."""
import json, sys
from collections import defaultdict
from datetime import datetime

def ts(s): return datetime.strptime(s, "%Y-%m-%dT%H:%M:%S.%fZ")

rows = [json.loads(l) for l in open(sys.argv[1]) if l.strip()]
runs = defaultdict(dict)
for r in rows:
    runs[r["run"]].setdefault(r["step"], {})[r["phase"]] = (ts(r["ts"]), r["note"])

run = sys.argv[2] if len(sys.argv) > 2 else next(iter(runs))
total, ranked = 0.0, []
for step, phases in runs[run].items():
    if "begin" in phases and "end" in phases:
        d = (phases["end"][0] - phases["begin"][0]).total_seconds()
        ranked.append((d, step, phases["begin"][1]))
        total += d

for d, step, note in sorted(ranked, reverse=True):
    print(f"{d:7.1f}s  {step:<12} {note}")
print(f"{total:7.1f}s  TOTAL")

Run it and you get a shape like signup 3.9s, install 214.0s, firsttask 41.2s, and those figures are illustrative of the format rather than a claim about any tool. The interesting part is how often the largest line is not the model, not the signup, but the local runtime install. It is also the step people skip when they retry, which is why they conclude the second attempt was faster when the machine was simply warm.

The one fix that mattered

Here is the change that moved the number more than any other, and it has nothing to do with which model answered. Replace the first task with a task that must end in a verifiable diff, using a two-file fixture you already know the correct answer to. A chat reply is unfalsifiable in the first fifteen minutes; a diff against a failing test is not. So I start every trial like this.

mkdir -p fixture && cd fixture
printf 'def add(a, b):\n    return a - b\n' > calc.py
printf 'from calc import add\nassert add(2, 3) == 5\n' > test_calc.py
python -m pytest -q   # fails: this failure is your baseline

Now the first prompt is boring on purpose: fix the failing assertion and return a patch, not an explanation. You apply the patch, you rerun pytest, and the run either passed or it did not, which means the onboarding measurement ends with a boolean instead of an impression. I label this metric time to first verifiable diff, and it collapses a dozen vague first-run complaints into one number you can compare across tools without arguing about style. Should you trust a tool that needs forty minutes to reach a green test more than one that needs four, when the second one also answers every unrelated question beautifully?

Where a zero-cost tier changes the ledger

Two of the steps in that ledger are pure overhead, and neither of them teaches you anything about the tool. One is billing setup, and the other is provisioning a machine before you can see whether the edit loop even works for you. MonkeyCode, which I encountered as an open-source coding agent project, addresses both by being reachable without a payment step and by offering a free server option, so the first pass of the ledger can run without either of those entries. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access is the part that keeps an evaluation cheap enough to abandon, which honestly is the property I care about most.

I am not going to quote a token allowance, a model list, or a server specification here, because I cannot verify those from where I sit and free tiers change quietly. The project's own pages are the only honest source for current limits and terms, so read them before you build anything that depends on them. Treat the free path as a way to get a first data point, never as the foundation of a pipeline whose reproducibility you actually need. A ledger you can rerun tomorrow is worth more than a discount you cannot predict.

Who should skip this approach

If your code cannot leave your network, the free-server path is irrelevant and the harness only helps you measure friction you were never going to pay. If your team needs SSO, audit logs, and deterministic model pinning, optimizing first-run minutes is optimizing the wrong axis, and the ledger will just document the wrong constraint. And if you already have a working loop you trust, the instrument is unnecessary, because you are no longer onboarding and there is nothing here to localize.

For everyone else, the transferable artifact is about sixty lines of shell and Python, and it survives whichever tool you abandon next quarter. If a zero-cost first pass makes that instrument cheap to run, start there, keep your own copy of the numbers, and let the top line of your ledger decide what you fix tomorrow.