Most teams instrument their agents down to the token. Almost nobody instruments the one bit that decides the bill: whether the output was accepted.
Quick disclosure before anything else: I work on KarmaBox, which sells hardware that runs agents locally — so I have an obvious interest in you believing that tokens aren't the whole story. The schema below is deliberately vendor-neutral. It works on any stack, cloud or local, and you can check every claim in it against your own data. Judge it on that.
Why per-token metrics lie to you
The per-token arithmetic is public and precise. Take a standard task shape — 4,000 input tokens, 800 output, 10% retries, 70% cache hit — and published prices put one task between $0.00086 and $0.0603 depending on the model, roughly a seventy-fold spread (InventiveHQ, prices verified 13 August 2026, page updated 19 October 2026). A mid-range model lands near $0.0121 per task. Pennies.
Two problems. First, agents loop — every step re-reads context, tool results come back as billable input, failures retry — so "unit price × task count" misses real bills by an order of magnitude. Second, and bigger: when we costed out our own October properly (1,840 accepted deliverables, self-reported, one team), model fees were 18% of total cost. Human time spent checking and redoing output was 41%. The line everyone optimizes was the second-smallest line we had.
Oh, and the token line isn't even stable: Grok users just watched their weekly quotas get maxed out overnight, with the CEO confirming serving costs were "unsustainably high relative to revenue" (TechCrunch, 2 November 2026). You can't forecast a meter someone else can re-read on a Sunday.
So here is the minimal schema that measures the number that actually reaches your P&L.
The three fields
attempt_cost
Everything this attempt burned: the sum of model fees across all calls in the job — or, if you run on your own hardware, an amortized share (monthly hardware cost ÷ jobs that month, recomputed monthly). Don't agonize over precision here. This will turn out to be the small number.
The collection trap: you will undercount by default. An agent job is not one API call, and the misses are systematic. Retries are usually swallowed by a retry decorator and never surface in your logs — they billed you anyway. Tool-call round trips bill twice in disguise: the tool's output comes back as input tokens on the next step, and on long research-style jobs that reflow is most of the spend. Sub-agent calls land on a different code path and get logged under a different name, or not at all. The fix is structural, not disciplinary: meter at the one client wrapper every model call in your codebase already flows through, and make a job_id tag mandatory at that chokepoint. If a call can run without a job_id, your numerator has a hole in it.
review_minutes
Wall-clock minutes a human spent checking the output before deciding its fate. This is the field everyone skips because it's annoying to capture, and it's the one that matters most.
The collection trap: humans won't log this, so don't ask them to. Capture must be a side effect of work the reviewer already does, or it dies within a week. The workable pattern: start the clock when the output is first opened, stop it when the verdict button is clicked, and pause on idle (no scroll, no keystroke for ~90 seconds) so a coffee break doesn't become a 47-minute review. No review UI? Put the timestamps in the verdict action itself — opened_at written on first view, elapsed computed at the accept/reject click. Honest 1-minute-granularity estimates are fine for week one, but treat them as provisional: when I audited my own gut estimates against a timer, I was off by 2–3×, always underestimating. Bad data here flatters the agent.
accepted
Boolean, no middle state. The output either shipped or it didn't. A redo is a new row with its own attempt_cost and review_minutes — that's how rework stops being invisible: it shows up as rows whose cost you paid but whose accepted never contributed to the denominator.
The collection trap: who holds the verdict. The accept/reject call belongs to whoever would be accountable for the deliverable in front of a customer, a boss, or an auditor — not to the engineer who built the agent, and not to the operator who ran it. Self-graded homework drifts optimistic; nobody rejects their own pipeline's output at 6 p.m. And record the verdict at ship time, not in a Friday batch: retrospective grading flatters the agent, because you remember the outputs that worked and shrug at the ones you quietly fixed by hand. The quiet hand-fix is exactly the cost this field exists to catch — if you touched it, it's a reject plus a redo row.
The one query
CREATE TABLE agent_jobs (
id bigserial PRIMARY KEY,
agent text NOT NULL,
finished_at timestamptz NOT NULL,
attempt_cost numeric NOT NULL, -- dollars
review_minutes integer NOT NULL,
accepted boolean NOT NULL
);
-- Cost per accepted deliverable, weekly.
-- 45.0 = loaded hourly cost of a reviewer (illustrative — use yours).
SELECT
date_trunc('week', finished_at) AS week,
round(
SUM(attempt_cost + review_minutes * (45.0 / 60))
/ NULLIF(SUM(CASE WHEN accepted THEN 1 END), 0)
, 2) AS cost_per_accepted,
round(AVG(CASE WHEN NOT accepted THEN 1.0 ELSE 0 END), 3) AS rework_rate
FROM agent_jobs
GROUP BY 1
ORDER BY 1;
The numerator is everything you spent, including rejected attempts and all review time. The denominator is only what shipped. That asymmetry is the whole point.
Two edge cases people hit in week one. First: what counts as a "job"? Use the unit a human would accept or reject — a finished draft, a reconciled ledger, a reviewed document — not an LLM call. If your pipeline chains five agents to produce one deliverable, that's one row, and attempt_cost is the sum across all five. Second: partial acceptance ("I used half of it") — count it as rejected and log the salvage time as review_minutes on the redo row. Strict rules make the metric ungameable; generous rules make it decorative.
The second query: which role is actually expensive
Once the weekly number is boring, split it by agent. This is where the metric starts paying rent, because the expensive role is almost never the one with the biggest API bill.
-- Cost per accepted job, by agent role, last 28 days.
SELECT
agent,
COUNT(*) AS attempts,
round(AVG(CASE WHEN NOT accepted THEN 1.0 ELSE 0 END), 3) AS rework_rate,
round(AVG(review_minutes), 1) AS avg_review_min,
round(
SUM(attempt_cost + review_minutes * (45.0 / 60))
/ NULLIF(SUM(CASE WHEN accepted THEN 1 END), 0)
, 2) AS cost_per_accepted
FROM agent_jobs
WHERE finished_at >= now() - interval '28 days'
GROUP BY agent
HAVING COUNT(*) >= 20 -- below ~20 attempts the number is noise, not signal
ORDER BY cost_per_accepted DESC;
Read the top row with the two middle columns, because they point at two different fixes. If rework_rate is what's high, you have a procedure problem: the task is under-specified, the acceptance bar is fuzzy, or the job is genuinely too hard for automation — fix the playbook, not the infrastructure. If avg_review_min is high while rework_rate is low, you have a boundary problem: the outputs are fine, but the reviewer is spending their minutes auditing what the agent might have done rather than grading what it wrote. In our own October split (self-reported, as above), the roles that topped this query were the outward-facing ones — the ones that could plausibly send, publish, or reply — not the ones doing the hardest cognitive work.
A dashboard that fits on one screen
Three cards, in this order:
cost_per_accepted, weekly line. The only number that goes to management. Trend matters more than level for the first two months.rework_rateby agent, 4-week bars. Where redo work concentrates. This card drives playbook edits and task re-scoping — procedure fixes.Median
review_minutesby agent. The boundary smell test. An agent whose median review is long while its rework is low is costing you perimeter patrol, not quality control — that card is pointing at architecture, and no prompt edit will move it.
Use medians on card 3, not means: one 40-minute incident review shouldn't repaint the whole card.
What the number does once you have it
Run an illustrative model — $0.30 model fees per attempted job, review at $45/hour, all replaceable:
Halve the token price in that model and you save $0.23. Cut rework from 35% to 12% and you save $5.33. If your cost conversations are about token discounts, you're negotiating the sliver.
Why rework rate is an architecture problem
Here's the part that surprised me when the data came in: review_minutes isn't mostly a function of output quality. It's a function of what the agent was capable of doing.
If the agent could have sent the email, deleted the record, or touched production, your reviewer isn't just checking whether the draft is good — they're auditing whether anything irreversible happened or was about to. That's the 6-minute review. If the dangerous capabilities are enforced away at the system layer — the agent's runtime simply has no tool connected for "send" or "delete", or those actions stop at an approval step — the reviewer checks content quality only. That's the 2-minute review.
A rule written in the prompt doesn't get you the second regime, because a prompt rule is text: it can fall out of a truncated context, be argued around, or be overridden by instructions hiding in the data the agent was asked to read. A capability that was never connected can't. Same model, same prompts, different review burden. The longer version of that argument is in the private AI capability paradox; the schema above is how you'd verify it on your own stack instead of taking my word.
What this schema fixes, what it doesn't — and where it lies
Worth being precise, because a measurement scheme is not a fix, and this one has failure modes of its own.
It fixes visibility risk: after two weeks of rows you know your real unit cost, your real rework rate, and which lever is biggest. Most teams discover their 41% is invisible precisely because nothing was writing it down.
It does not fix rework risk itself — logging a 35% rework rate doesn't lower it; that takes boundary architecture, narrower task definitions, or better procedures. And it does not fix quota risk: your attempt_cost column is denominated in someone else's pricing policy unless the hardware is yours. Separate problems, separate fixes.
And three ways the bookkeeping itself will mislead you if you let it:
Small samples jitter. At 15 jobs a week per agent, one botched redo swings
cost_per_acceptedby 30% and someone declares a trend. That's what theHAVING COUNT(*) >= 20line is for; read 4-week windows, not single weeks.review_minutesis a floor, not the truth. It logs wall-clock time, not the cost of the interruption — a 4-minute review that breaks a developer out of deep work costs more than 4 minutes buys back. The real human line is higher than this field reports. (It still came out as our biggest line at 41%, self-reported — which tells you how big the honest number must be.)acceptedstandards drift. As trust in an agent grows, reviewers get generous, and the metric improves while reality doesn't. Re-anchor monthly: pull five random shipped jobs and re-review them against the original acceptance checklist. If two fail on re-review, your rework rate is fiction and the drift is the finding.
One caveat on the self-hosting escape hatch: local hardware only beats cloud per-job when the box actually works most of the day — MindStudio's analysis (8 October 2026) makes the break-even a pure utilization question, and an idle box is pure waste. Measure first. The schema will tell you whether you have the volume.
Start tonight
Three columns. Two queries. Two weeks of data. Then ask yourself the question I couldn't answer a month ago: what's your median review_minutes — and if you don't know, what is that costing you?
Sources
TechCrunch, Elon Musk throttles Grok Bot limits due to "unsustainably high" costs, 2 November 2026.
InventiveHQ, LLM API Pricing Comparison, page updated 19 October 2026.
MindStudio, Local AI Hardware vs Cloud API Tokens: Is It Cheaper?, 8 October 2026 — vendor-affiliated analysis. No publicly verifiable URL at time of writing; cited by title and date.
KarmaBox Team ledger, October 2026.
Top comments (0)