The Token Ledger: Catching Cost Drift in AI-Generated Patches

A team that runs the same AI-assisted patch task every week usually notices the problem only when a bill arrives. The prompt file has not changed, the repository has not grown, and the generated diff looks smaller than last month, yet the token count has drifted upward across five consecutive runs. This pattern behaves like a slow memory leak: invisible in code review, painful in production, and almost never caught by a test suite.

Cost drift has three common causes, and none of them appear in a diff. Agent memory is the first, because an assistant that accumulates conversation history, tool outputs, and earlier attempts carries a heavier context prefix on every new request, so the same task burns more input tokens each time it runs. The second is model-side change, when a provider replaces or tunes the model behind a stable label and silently alters tokenization and behavior. The third is silent retries, where a tool re-attempts a failed call and multiplies the price of a single task. Recent conversations about how AI agents remember everything and trust all of it framed this as an architecture concern; a token ledger turns it into a measured number.

The 90-minute spike

A token ledger is a one-file script that records what one fixed task costs and compares the result against a stored baseline. The spike keeps the familiar 90-minute shape: one hypothesis, a repeatable measurement, and a ship-or-kill decision at the end. A useful hypothesis reads: "With the current free model tier, the bug-fix task completes within H tokens and L milliseconds of wall time on the free server option, across five runs." The numbers H and L come from the budget a team can actually afford, not from a benchmark someone else published.

#!/usr/bin/env bash
# ledger.sh — measure what one AI coding task really costs
set -euo pipefail

TASK_ID="${1:?usage: ledger.sh <task-id> <prompt-file>}"
PROMPT_FILE="${2:?usage: ledger.sh <task-id> <prompt-file>}"
API_URL="${API_URL:?set API_URL to your gateway /chat/completions endpoint}"
API_KEY="${API_KEY:?set API_KEY}"
MODEL="${MODEL:-free-default}"
LEDGER="ledger.jsonl"
BASELINE="baseline.${TASK_ID}.json"
DRIFT_LIMIT="${DRIFT_LIMIT:-1.20}"

payload=$(jq -n --rawfile prompt "$PROMPT_FILE" --arg model "$MODEL" \
  '{model:$model, messages:[{role:"user", content:$prompt}], stream:false}')

start=$(date +%s%3N)
response=$(curl -sS --fail "$API_URL" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d "$payload")
end=$(date +%s%3N)

tokens_in=$(jq -r '.usage.prompt_tokens // 0' <<<"$response")
tokens_out=$(jq -r '.usage.completion_tokens // 0' <<<"$response")
total=$((tokens_in + tokens_out))
wall_ms=$((end - start))

if [[ "$total" -eq 0 ]]; then
  echo "no usage object in response; the adapter must expose token counts" >&2
  exit 2
fi

if [[ -f "$BASELINE" ]]; then
  baseline=$(jq -r '.tokens_total' "$BASELINE")
  drift=$(awk -v t="$total" -v b="$baseline" 'BEGIN{printf "%.2f", t/b}')
  verdict=$(awk -v d="$drift" -v l="$DRIFT_LIMIT" \
    'BEGIN{print (d > l) ? "FAIL" : "PASS"}')
else
  drift="1.00"
  verdict="BASELINE"
  jq -n --arg task "$TASK_ID" --argjson total "$total" \
    '{task:$task, tokens_total:$total}' > "$BASELINE"
fi

ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
jq -n --arg task "$TASK_ID" --arg model "$MODEL" --arg ts "$ts" \
  --argjson tin "$tokens_in" --argjson tout "$tokens_out" \
  --argjson total "$total" --argjson wall "$wall_ms" \
  --arg drift "$drift" --arg verdict "$verdict" \
  '{task:$task, model:$model, ts:$ts, tokens_in:$tin, tokens_out:$tout,
    tokens_total:$total, wall_ms:$wall, drift:$drift, verdict:$verdict}' \
  >> "$LEDGER"

echo "$ts $TASK_ID tokens=$total wall=${wall_ms}ms drift=$drift verdict=$verdict"
[[ "$verdict" == "FAIL" ]] && exit 1

The script assumes an OpenAI-compatible /chat/completions response with a usage object; most coding-agent gateways expose one, and pointing API_URL at whatever gateway the team runs is the only required setup. On macOS the %3N timer needs gdate, so the pragmatic move is to run the spike inside a small Linux container, which also keeps the measurement close to a real server environment.

The protocol is deliberately short. Pick one representative task from the team's queue and freeze its prompt file. Run ledger.sh three times so the first run writes baseline.<task>.json and the following runs append rows to ledger.jsonl. If the totals stay within twenty percent of one another, run two more times and record the verdicts; a drift failure twice in a row on unchanged inputs is evidence that something inside the toolchain changed, which is exactly the signal a ship-or-kill decision needs. A finished ledger row looks like this, and it is small enough to commit next to the prompt file:

{"task":"fix-orphan-fk","model":"free-default","ts":"2026-09-01T08:12:33Z","tokens_in":4120,"tokens_out":860,"tokens_total":4980,"wall_ms":14320,"drift":"1.04","verdict":"PASS"}

The verdict meanings stay simple enough to explain in one sentence each.

Verdict	Meaning	Action
BASELINE	First measured run for this task	Repeat the run; if totals are stable, keep the baseline
PASS	Total within 1.2x of the baseline	Continue, then review the patch as usual
FAIL	Total over 1.2x with unchanged inputs	Investigate context, retries, and the model label before shipping

Why a free tier is the honest place to measure

A paid account hides the very margin a team needs to see, because the ceiling is information. A free model tier such as the one bundled with MonkeyCode's open-source agent exposes that constraint directly: if the task cannot complete within the free allowance across repeated runs, the team has concrete budget evidence before spending anything. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The runtime half of the hypothesis deserves the same honesty, and MonkeyCode also ships a free server option, so the spike can measure latency, cold starts, and disk behavior on the same class of machine the feature would actually use; measuring a server workload from a laptop is the fastest way to build a confident but wrong estimate.

Limits and who should skip this

The ledger measures quantity, not quality, and a task that consumes fewer tokens can still produce a broken patch, so the normal human review gate stays in place and the ledger remains a cost metric rather than an acceptance test. Quotas and free-tier policies change over time, which is why every row records a timestamp and a model label; a later re-run can then distinguish a product change from a policy change. The method suits teams with a stable, representative task, and it will not help a team whose assignments change every day, nor a team that already tracks cost per task inside a mature evaluation harness; skipping it is the right call for teams that ship AI-generated patches without any human review.

For everyone else, a ninety-minute ledger replaces a guess with a number, and that number belongs in the README next to the build badges, because cost drift is a test failure like any other. The script above, the free model tier, and the free server option are enough to run the whole experiment before any budget discussion starts.