Free Tokens Are the Wrong Metric for AI Coding Assistants

A developer receives a ten-million-token grant, connects an AI coding agent to a weekend project, and watches the terminal fill with confident diffs for three hours. The token counter climbs, the agent declares every issue resolved, and the repository still fails its test suite at the end of the day. The grant was generous, the server was free, and the output was worthless. Generosity and usefulness are different currencies, and a free tier only helps when someone bothers to exchange one for the other.

The recent DEV discussion about the AI badge makes the same point in a different arena: a label describes how something was produced, not whether it is any good. A badge on a post, like a token count on a dashboard, is a proxy that people mistake for the thing itself. The proxy becomes dangerous when it drives decisions, because a developer who watches tokens burn feels productive while the repository quietly rots.

Free model access and free server options are valuable precisely because they remove the financial excuse for skipping measurement. The mistake is treating the grant itself as progress, as if spending tokens were the same as shipping fixes. Tokens measure what a provider is willing to give away; verified patches measure what a developer actually gains. The gap between those two numbers is the real cost of any AI coding assistant.

MonkeyCode, an open-source coding-assistant project, currently offers free model access and a free server option, with a ten-million-token grant in its published free tier as of this writing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The interesting question is not whether the grant is large enough, but whether it survives contact with a real test suite. A free tier that produces unverified diffs is a tax on attention, not a gift.

The standard remedy is an evaluation harness, and most harnesses measure what the agent claims rather than what the repository confirms. A cleaner approach is a patch ledger: run the agent against a list of real issues, capture the token usage from each response, apply the diff to a disposable worktree, and let the test suite cast the only vote that matters. The ledger below is deliberately small because it is meant to be read, modified, and pointed at whatever free tier is available this quarter.

#!/usr/bin/env bash
# patch-ledger.sh — tokens spent per verified patch
# Usage: AGENT_CMD="..." ./patch-ledger.sh <repo-url> <issues-file>
set -euo pipefail

REPO_URL="${1:?repo url required}"
ISSUES_FILE="${2:?issues file required}"
WORK_ROOT="$(mktemp -d)"
TOTAL_TOKENS=0
VERIFIED=0
ATTEMPTED=0

while IFS= read -r issue; do
  [ -z "$issue" ] && continue
  ATTEMPTED=$((ATTEMPTED + 1))
  worktree="$WORK_ROOT/issue-$ATTEMPTED"
  git clone -q "$REPO_URL" "$worktree"
  git -C "$worktree" checkout -qb "fix/$ATTEMPTED"

  response=$(cd "$worktree" && eval "$AGENT_CMD" --prompt "$issue" 2>/dev/null || true)
  tokens=$(printf '%s' "$response" | jq -r '.usage.total_tokens // 0')
  TOTAL_TOKENS=$((TOTAL_TOKENS + tokens))

  if git -C "$worktree" diff --quiet; then
    printf 'no patch: %s\n' "$issue"
    continue
  fi

  if git -C "$worktree" diff --check --exit-code && (cd "$worktree" && npm test --silent); then
    VERIFIED=$((VERIFIED + 1))
    printf 'verified: %s (%s tokens)\n' "$issue" "$tokens"
  else
    printf 'rejected: %s (%s tokens)\n' "$issue" "$tokens"
  fi
done < "$ISSUES_FILE"

printf '\nattempted: %s\nverified: %s\ntotal tokens: %s\n' "$ATTEMPTED" "$VERIFIED" "$TOTAL_TOKENS"
if [ "$VERIFIED" -gt 0 ]; then
  printf 'tokens per verified patch: %s\n' "$((TOTAL_TOKENS / VERIFIED))"
fi
rm -rf "$WORK_ROOT"

The script clones the repository once per issue, asks the agent to fix it, and extracts the total token count from the JSON response using jq. It then rejects any diff that fails git diff --check or the project's test suite, and prints a running account of tokens spent per verified patch. The AGENT_CMD environment variable keeps the harness provider-agnostic, so it works with MonkeyCode's free model access, a local model, or any CLI that emits an OpenAI-style usage object. Adjust the jq path when a provider reports token counts under a different key.

export AGENT_CMD="your-agent-cli --model free-tier"
printf 'fix the flaky login test\nupdate the cache invalidation logic\n' > issues.txt
./patch-ledger.sh https://github.com/example/repo.git issues.txt

The output reads like a receipt for the agent's labor: a line for every issue, a token count for every attempt, and a final ratio that is hard to argue with. A run that reports three verified patches from nine attempts at sixty thousand tokens each is a run that spent half a million tokens for three working fixes. The same run against a different free tier might cost nothing in cash but everything in debugging time, and the ledger makes that trade visible before the merge request is opened.

A healthy small repository should land well under one hundred thousand tokens per verified patch, while a number above two hundred thousand usually means the agent is burning the free tier on noise. Those thresholds are starting points for a small codebase, not benchmarks, and they will shift with repository size and test coverage. The point of the ledger is the trend: if tokens per verified patch climbs across a week of issues, the model, the prompts, or the task list needs attention. A free server option makes this experiment costless to run, which is exactly why it should be run continuously.

The approach has real limits. Token accounting varies by provider, some agent CLIs hide usage data entirely, and flaky tests will punish a good patch as readily as a bad one. The ledger also assumes every issue can be verified by an automated test, which rules out documentation changes, configuration migrations, and architectural decisions. The ten-million-token figure is current as of August 2026, and free tiers change without ceremony, so the script should be re-run against whatever terms are published when a new grant arrives.

Developers who are exploring a greenfield prototype, where the agent's wrong turns are part of the discovery process, do not need this ledger. Teams without a test suite will find the script mostly silent, and anyone seeking benchmark-grade model comparisons needs controlled seeds and repeated runs, not a weekend script. For everyone else, the ledger turns a marketing number into an engineering number, which is the only conversion that matters. The next time a free grant lands in an inbox, spend the first hundred tokens on a ledger like this one and let the test suite do the arguing.