Your token dashboard can't tell you what one successful agent workflow costs

dualdust — Sat, 11 Jul 2026 10:18:58 +0000

Most teams can find last month's model spend. Far fewer can price one agent result that somebody accepted and used.

Consider a support agent handling a refund. It reads the conversation, searches a policy document, calls a billing tool, drafts a reply, and retries after the first answer fails review. A person edits the second answer and sends it.

The provider records several model calls. The tool vendor records a search and a transaction. The tracing system records a chain of steps. The support team sees one request.

Each system is correct, but none of them has the whole job. A token dashboard can be detailed and still leave the product team unable to answer a basic question: how much did the accepted result cost?

The provider bill stops at the call

Provider usage data helps finance reconcile spend. It can group usage by model, project, user, or API key. That is useful, but the provider cannot see the complete business workflow inside your application.

An agent may call tools, delegate work, retry a failed step, switch models, and wait for human review before it produces anything useful. If each model call becomes a separate unit, retries and workers lose their connection to the request that caused them. If you only report token cost, tool charges and review work disappear.

This distorts model comparisons. A cheap first call can trigger repairs and a premium fallback. A route with a higher call price can cost less when a reviewer accepts its first result.

The comparison needs a record that follows the work from the original request to the final outcome.

Give the requested job one parent run

Create a parent run when the user request or scheduled job begins. Give it a stable run_id. Every planner, worker, model call, tool call, retry, fallback, and review step becomes a child event linked through parent_run_id.

A compact event can look like this:

{
  "run_id": "run_refund_8421",
  "parent_run_id": "run_refund_8421",
  "event_type": "model_call",
  "attempt": 2,
  "provider": "example-provider",
  "requested_model": "model-small",
  "returned_model": "model-large",
  "input_tokens": 4812,
  "output_tokens": 623,
  "estimated_cost_usd": 0.041,
  "retry_reason": "review_rejected",
  "fallback_reason": "quality_threshold",
  "occurred_at": "2026-07-11T14:32:18Z"
}

The values are illustrative. The shape matters more than the numbers.

Keep requested_model and returned_model separate. A gateway may change the route after a capacity error or policy check. Logging only the requested model can assign cost and quality to the wrong route.

Keep failed attempts too. Updating one trace span with the latest attempt hides the money spent on the first attempt and removes the evidence you need to fix the failure.

The first version of the ledger only needs to answer four questions:

Which requested job caused this work?
Which model and provider served each call?
Which tools, retries, and route changes added cost?
Did the owner accept the final result?

Close the parent run with an outcome

Runtime completion is not a business outcome. An API can return successfully while a reviewer rejects the answer.

Give the parent run a state such as:

accepted
rejected
abandoned
pending_review

The workflow owner should define accepted before the run starts. A support team might require a sent reply that does not reopen the ticket within a set period. A coding team might require a merged patch with passing tests. A research workflow may need a reviewer to approve the sources.

Some outcomes arrive late. Do not pretend that problem has disappeared. Store the current state, keep the time of each decision, and update the outcome when new evidence arrives. Your first report can separate immediate acceptance from later reversal.

Human review creates another awkward field. If you have a defensible labor rate, include the cost. If you do not, report review minutes beside the monetary total. An honest incomplete field is more useful than a precise number built on an invented rate.

Compare cost per accepted outcome

Once every child event belongs to a parent run, add the provider cost, paid tool cost, and any material review cost for that run. Then compare total workflow cost with accepted outcomes.

A basic calculation is:

cost per accepted outcome =
  total cost of completed parent runs / number of accepted parent runs

You will probably want the distribution as well as the average. One runaway loop can hide inside a reasonable mean. Track the median, a high percentile, and the most expensive rejected runs.

This view gives the workflow owner better questions:

Which route produces accepted work with the least retry overhead?
How often does the default route depend on a premium fallback?
Which workflow version creates the most reviewer rework?
Which agent keeps spending money without producing accepted results?

Token counts still help with diagnosis. The parent workflow gives the team a unit for budget and routing decisions.

Start with one workflow

Pick one agent workflow with meaningful spend, volume, or risk. Write down its start event, end event, owner, and acceptance rule before adding instrumentation.

Generate the parent ID at the application edge. Pass it through the agent framework, gateway, and tools. Record each attempt instead of overwriting the previous one. Keep raw provider exports for reconciliation because trace estimates and billed costs can differ.

Review one week of runs. Reconstruct several expensive or rejected examples by hand. You will probably find missing IDs, tool costs that never reached the trace, and acceptance rules that sounded clear until two reviewers disagreed.

Do not start with a company-wide agent cost platform. A broad dashboard built on weak identifiers gives you precise totals that nobody can connect to a result.

Start with one question: how much did one accepted result cost?

Sources

Originally published on the TokenAir Blog.

Disclosure: I used AI tools to help edit this post. I checked the technical claims and sources before publication.