DEV Community

Kunal
Kunal

Posted on • Originally published at kunalganglani.com

Agent Per-Task Cost Calculation [2026]: Retries, Tools, Caching

#aiagents #llmcost #tokenoptimization #productionai

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

Agent Per-Task Cost Calculation [2026]: Retries, Tools, Caching

Agent per task cost calculation is the practice of estimating what an entire agent workflow costs on average, not what a single model call costs. The moment you add tool calls, retries, parallel fanout, and growing context, “$ per call” becomes a lie you tell yourself to feel in control.

Key takeaways

  • A usable model is an expected-cost model: probability-weighted retries, tool fanout, and context growth included.
  • Tool calls don’t just add latency. They add tokens (schemas + tool results reinjected) and sometimes separate provider-billed charges.
  • Caching changes the math. You need cache hit rate plus cache read/write categories, and sometimes storage costs.
  • Budget enforcement is orchestration logic, not dashboards. Decide what gets cut first when the workflow exceeds budget.
  • If your tracked cost doesn’t match the invoice, assume mismatched time windows and token categories (including cache) before you assume fraud.

If you can’t explain your agent’s cost in a spreadsheet, you don’t have cost control. You have vibes.

Why “cost per call” breaks for agents

Single-call apps are simple: cost = (input_tokens * $/input) + (output_tokens * $/output).

Agents are not single-call apps. They’re graphs:

  • Step A: planning call
  • Step B: tool selection call
  • Step C: N tool calls in parallel (fanout)
  • Step D: synthesis call
  • Plus: retries when the tool fails, the schema doesn’t validate, or the model “hallucinates” an argument.

In my experience running this site’s multi-agent blog publishing pipeline (261+ published posts), the “model-per-job-shape” approach is the only thing that kept costs sane: cheaper models for tool-heavy loops, better models for final prose. One-model-everywhere is the expensive way to learn that retries exist.

This post is the missing piece: a spreadsheet-ready expected-cost model you can use to budget and enforce spend per workflow.

The spreadsheet schema: the minimum columns that matter

[YOUTUBE:mTnXWRmZVlQ|Build Reliable AI Agents Part 2: Prompt Consistency, Tracing, & Costs]

Here’s the template I wish more teams started with. You can implement it in Sheets, Excel, or a database table. The point is the same: every agent run becomes a ledger.

Spreadsheet columns (table-snippet bait)

Column What it means Example value
workflow_name Chargeback boundary support_refund_agent
task_name Step name within the workflow extract_policy
step_type llm / tool / guardrail llm
model Model used for this step gpt-*
calls_per_task How many times this step runs per workflow execution 1
fanout Expected parallel calls (map step) 4
retry_prob Probability this step retries at least once 0.15
avg_retry_count Expected number of retries when retry happens 1.2
input_tokens Tokens sent to model (uncached) 2200
output_tokens Tokens generated by model 450
cached_input_tokens Tokens served from cache (if provider reports it) 1600
tool_schema_tokens Tokens spent describing tool schemas (often forgotten) 300
tool_result_tokens Tokens from tool output reinjected into context 1200
tool_result_reinjection_rate Fraction of tool output that gets re-included downstream 0.6
context_compaction_factor Compression ratio after summarization/compaction 0.35
input_$per_million Provider input price from pricing
output_$per_million Provider output price from pricing
tool_provider_cost Non-token costs (search, KB, storage) $0.002

Three “hidden multiplier” columns that make this spreadsheet actually work in production:

  • tool_result_reinjection_rate
  • context_compaction_factor
  • tool_schema_tokens

Most teams track input_tokens/output_tokens and then wonder why their invoice is 2–3x their forecast.

The expected-cost equation (retries + fanout)

You want a single number per workflow run: expected cost.

At the step level:

  • Base LLM cost (tokens):
    • llm_cost = (billable_input_tokens * input_rate) + (output_tokens * output_rate)
  • Billable input tokens should include your hidden multipliers:
    • billable_input_tokens ≈ input_tokens + tool_schema_tokens + (tool_result_tokens * tool_result_reinjection_rate)
    • then apply compaction (if you compact before the step): multiply by context_compaction_factor

Now add retries.

A simple probability-weighted retry model:

  • expected_attempts = 1 + (retry_prob * avg_retry_count)
  • expected_step_cost = expected_attempts * (llm_cost + tool_provider_cost)

Now add fanout.

  • If a step fans out into fanout parallel tool calls, treat it as:
    • expected_step_cost = expected_step_cost * fanout

Finally, per-workflow:

  • expected_workflow_cost = Σ expected_step_cost across all steps.

This is boring math. That’s why it works.

For token pricing, use the canonical provider pages, e.g. OpenAI API pricing.

Tool calls: schema tokens, tool result tokens, and “provider surprise” costs

Tool calls affect your cost in three distinct ways:

  1. Tool schema overhead

    • Tool/function definitions are tokens. If you include a 200-line JSON schema in every call, you’re buying that schema over and over.
    • OpenAI’s tool/function calling docs make it clear you’re sending the schema as part of the request context: Function calling.

  2. Tool results reinjected into context

    • Agents typically do: call tool → get result → paste result into the next LLM call.
    • That tool output is now input tokens, sometimes repeatedly across multiple future steps.

  3. Tool provider costs beyond tokens

    • Some platforms price additional features and tooling categories beyond pure token I/O.
    • For example, the Amazon Bedrock pricing page includes a “Built-In-Tools Pricing” navigation area, which is a strong hint that not everything is covered by model token rates.

Practical advice: when you define your chargeback ledger, give tools their own rows with step_type=tool and include both tool_result_tokens and tool_provider_cost.

Hidden token multipliers (the checklist)

“Hidden token multipliers” are the reasons your agent cost blows up even when every single call looks “reasonable.”

Here’s the checklist I use.

1) Context growth across turns

Every turn adds:

  • prior messages
  • tool schemas
  • tool results
  • intermediate reasoning/scratch (depending on framework)

Even if each step is 2k input tokens, a 10-step workflow can end up sending 10k–30k total input tokens due to repeated prefixes.

If you want a single spreadsheet lever to represent this: add context_growth_rate (e.g., +15% per step) or, better, explicitly track tool_result_tokens and reinjection rate.

2) Retries that are “invisible” in product metrics

Retries often don’t show up in “successful task count.” They show up in cost.

Model them explicitly:

  • retry_prob
  • avg_retry_count

And if you do schema validation, split retry reasons:

  • retry_prob_schema
  • retry_prob_tool_timeout
  • retry_prob_model_refusal

3) Reasoning/internal tokens (where applicable)

Some reasoning-style models can consume additional internal tokens/state beyond what you see as plain output. OpenAI discusses reasoning models and related concepts in their guide: Reasoning models.

I’m not going to pretend every provider bills this the same way. The practical takeaway is simpler: when you’re using reasoning models, budget with a safety factor (I start with 1.2x) until you’ve measured real usage from traces.

How prompt caching works (and how it changes the math)

Prompt caching is the only optimization that can feel like cheating when it’s configured correctly.

Anthropic documents prompt-prefix caching using cache_control, with both automatic caching and explicit cache breakpoints, and TTL options including 5 minutes and 1 hour: Prompt caching.

The cost model impact is straightforward:

  • Without caching: you pay full input tokens for repeated prefixes.
  • With caching: repeated prefixes can be billed differently (often cheaper reads) or not billed the same way, depending on the provider.

Explicit cache breakpoints

The key idea: cache the stable prefix (system prompt, policies, tool schemas, static instructions), then vary the suffix (user input, retrieved docs, tool outputs).

In a spreadsheet, represent caching with two columns:

  • cache_hit_rate (0–1)
  • cached_input_tokens

Then:

  • effective_billable_input_tokens = input_tokens - (cached_input_tokens * cache_hit_rate)

If your provider distinguishes cache reads vs writes, add separate rates:

  • cache_write_$per_million
  • cache_read_$per_million

Don’t overthink it. Start by tracking hit rate and cached tokens. You can refine once you’ve got real traces.

Caching strategies and considerations

  • Cache the stable prefix. Don’t cache volatile retrieved context.
  • Keep tool schemas small. If you must have many tools, split them across steps.
  • TTL matters. A 5-minute TTL is great for bursts. A 1-hour TTL is great for workflows that repeat over a day.
  • Caching can introduce separate storage-related costs on some platforms. Google Cloud’s Vertex pricing page includes a “Context Cache Storage price for Explicit Caching” section, which signals caching may not be free: Vertex AI Generative AI pricing.

How to Track Spend with LiteLLM (and make Finance stop hating you)

LiteLLM’s proxy is one of the more pragmatic ways to get multi-provider cost tracking without writing a pile of glue.

The Spend Tracking docs describe tracking spend for keys/users/teams, along with tags for attribution and endpoints for querying spend: Spend Tracking.

Total spend per user

The obvious move: track spend by user for showback.

The less obvious move: track spend by workflow and task. “Per user” is useful for internal controls, but it won’t tell you why your refund agent costs 5x more this week.

Daily Spend Breakdown API

Daily breakdowns are how you catch regressions fast. You want a chart that answers:

  • Did cost jump on Tuesday because of traffic?
  • Or because retry rate doubled after a tool change?

LiteLLM exposes spend endpoints and daily breakdown concepts in the docs; wire these into whatever you use for dashboards.

Custom Tags

Tags are where chargeback becomes real.

Use tags like:

  • jobID:<uuid>
  • workflow:<name>
  • task:<name>
  • team:<name>
  • env:prod|staging

LiteLLM shows passing tags via metadata in requests: Spend Tracking.

If you do nothing else, do this. Tags let you answer “what is burning money?” in minutes.

Supported Caches (LiteLLM) and how caching affects cost attribution

LiteLLM supports multiple caching backends including in-memory, disk, Redis, and semantic caches, per their caching docs: Caching.

In cost accounting, caching creates two problems:

  1. Your app-level cache hits might bypass the provider entirely (great), but then your “provider usage” undercounts real request volume.
  2. Provider-level prompt caching might show up as separate token categories (cache read/write) depending on the vendor.

Solution: treat caches as first-class components in your ledger.

  • Add a cache_layer column: none | app_response_cache | provider_prompt_cache
  • Track cache_hit boolean per call.
  • Tag cache namespace with workflow/task, so Finance can see savings by feature.

Instrumentation: traces, spans, and the metrics per step

You cannot budget what you can’t measure, and agents are dynamic.

OpenAI’s Agents SDK has two docs worth reading end-to-end:

The usage docs explicitly list tracked metrics like:

  • requests
  • input_tokens
  • output_tokens
  • total_tokens
  • per-request breakdown entries, including cached token details where available (the docs show input_tokens_details.cached_tokens).

The tracing docs describe spans for LLM generations, tool calls, guardrails, and custom events.

What metrics should be tracked per span/step?

Minimum viable per span:

  • span_id, trace_id
  • workflow_name, task_name, job_id, user_id, team
  • step_type (llm / tool)
  • model
  • input_tokens, output_tokens
  • cached token breakdown where available
  • retry_attempt
  • tool_name and tool_duration_ms
  • error_type (timeout, schema_validation, tool_5xx, model_refusal)

If you already run OpenTelemetry, you’re 80% of the way there.

As Arize AI’s team puts it, the era of single-turn calls is behind us, and observability needs to be trace- and span-based for multi-step agents: LLM Observability for AI Agents and Applications (Arize AI).

Hard budgets per workflow: what to cut first (a real policy)

Dashboards don’t enforce budgets. Your orchestrator does.

Here’s the decision tree I actually recommend. It’s intentionally brutal.

  1. If budget exceeded during tool fanout: reduce fanout first.

    • Cap fanout to 1–2.
    • Prefer breadth reduction over depth reduction because fanout multiplies everything.

  2. If still over budget: downgrade models on tool-heavy steps.

    • Keep the best model for the final synthesis.

  3. If still over budget: compact/truncate context.

    • Apply summarization and drop tool outputs older than N steps.
    • This is where your context_compaction_factor column becomes a knob, not a retrospective stat.

  4. If still over budget: skip optional tools.

    • Make optionality explicit in step definitions.

  5. If still over budget: stop and return a partial result.

    • Don’t silently keep spending. Fail “loudly” with a reason.

This is one of those things where the boring answer is actually the right one. Budget policy belongs next to your retry policy.

Reconciling tracked cost vs provider invoice (what mismatches to expect)

If your internal tracking doesn’t match the invoice, don’t panic. Assume there’s a dumb explanation first.

LiteLLM’s docs include a debugging workflow that starts with aligning time ranges and comparing token categories (including cache): Spend Tracking.

Common mismatch sources:

  • Different time windows (UTC vs local, ingestion delays)
  • Cache token categories not included in your rollup
  • Model pricing map drift (your internal $/token is stale)
  • Retries counted in provider bill but filtered out of “successful requests” in your app metrics
  • Tool provider charges (search, storage, KB) not captured in token-based accounting

Rule: your invoice is the source of truth for dollars, but your traces are the source of truth for causality.

A concrete anchor from my own work

Running this blog’s agent pipeline taught me a painful SEO lesson that maps directly to “hard budget enforcement”: slug identity is a one-way door. Rewriting slugs on live URLs burned 907K impressions of link equity in one incident.

Cost control is the same kind of one-way door. If you don’t put stop conditions in the orchestrator, you’ll learn about cost when Finance pings you. That’s the wrong feedback loop.

If you want more on budgeting at the workflow level, I wrote a first pass here: AI agents and the broader production context is in AI in production.

Pricing: what to link, what to trust

For token rates, use canonical sources:

And then treat your spreadsheet as a live artifact. Pricing changes. Your architecture decisions shouldn’t.

What to do next

If you’re building agentic AI and you’re still budgeting off “average tokens per call,” stop. Seriously. Take one workflow, break it into spans, and fill in the spreadsheet columns above.

The teams that win in 2026 won’t be the ones with the fanciest prompts. They’ll be the ones who can say, with a straight face: “This workflow costs $0.012 on average, and we can cap it at $0.02 without degrading user experience.”

That’s not hype. That’s engineering.

Suggested inline illustrations (for your editor)

1) At first major section break

  • altDescription: “Spreadsheet template for agent per-task cost calculation showing steps, fanout, retry probability, cached tokens, and per-step expected cost.”

2) Before caching section

  • altDescription: “Diagram of prompt prefix caching with an explicit breakpoint between stable system/tool schema prefix and variable user/tool-output suffix.”

3) Before budgets section

  • altDescription: “Decision tree for enforcing hard LLM workflow budgets: reduce fanout → downgrade model → compact context → skip tools → stop.”

Internal links used (contextual)

Originally published on kunalganglani.com

Top comments (0)