AI agents look simple at first.
You take a model, add a prompt, maybe connect a tool, and it works. It feels like you are just making one API call...
For further actions, you may consider blocking this person and/or reporting abuse
One small thing on the
totalTokenCost = calls * (input + output)model though: it's actually optimistic, because input tokens aren't constant across those calls. Each loop usually carries the growing transcript plus fresh memory hits, so call number three is paying for calls one and two all over again. Closer to a sum of increasing inputs than one number times a count. Doesn't change your point, it just means the real curve is steeper than the formula suggests, which honestly makes the guardrails argument stronger.@nazar-boyko, you’re right—the simple multiplication is optimistic. The more accurate model is a sum over calls, because each call may carry an expanding transcript, new retrieval, and tool output unless caching or compaction intervenes. I’ll tighten that framing; the steeper curve makes per-step context accounting and budgets even more important.
The retry section understates the nastiest failure mode: retries don't just multiply cost linearly, they interact with your context window. When a tool call fails and you retry inside the agent loop, a lot of frameworks append the error output and the failed attempt back into the conversation history. So retry #3 isn't running against the same prompt as attempt #1 — it's running against a bigger, dirtier context that now contains two prior failures, which makes the model more likely to fail again the same way. You get a doom loop where each retry costs more tokens and is more likely to need another retry.
The fix isn't just a retry cap. It's deciding what the retry actually sees. Retrying a transient network 500 should replay the original clean request; retrying a bad-model-output validation failure should include the error so the model can correct — but you almost never want to accumulate every prior failed attempt. Those are two different retry policies that people collapse into one
forloop, and the collapse is where the cost blows up. Worth adding a rule to your guardrails list: retries must be idempotent in context, not just bounded in count.@wrencalloway, “retries must be idempotent in context” is a strong rule. A transport retry should usually replay a clean request, while a semantic repair may include one bounded validation error—but neither should blindly accumulate every failed attempt. The trace should record attempt number, error category, context size, and idempotency key so a retry cap measures the whole policy, not just loop count.
The expanding-prompt point is the sneaky one: once memory retrieval feeds back in, input tokens grow every turn and cost stops being linear within a single run. I've found retries hide the worst of it, since a silent retry doubles a step nobody is counting. Do you cap retries per step and track cost per successful task rather than per call?
@kartik-nvjk, I prefer both limits: a small attempt cap per step and a larger run-level budget across tokens, latency, and tool calls. Cost per successful task is the business metric, while per-call cost is diagnostic. A run that succeeds after repeated hidden retries should therefore be marked functionally successful but operationally degraded.
The retry point is the one that bit us hardest. A failed step doesn't cost one more call — it re-runs the whole sub-loop, memory pulls and tool calls included, so the curve bends up fast. What helped was making "when do we stop retrying" a real rule: a hard budget per unit of work, and when it's blown the run caps and hands back to a human instead of digging a deeper hole.
The memory-tax point is the other sneaky one. Context doesn't just cost tokens once — it costs them every turn after, since it rides along in the window forever. We ended up compressing history as the window fills instead of all at once: summarize the low-value messages first, then actually drop them as pressure climbs, and hard-stop before a request would overflow. Trimming what rides along each turn saved us more than any per-call optimization did.
@distilled, budgeting per unit of work is the crucial distinction. A cap on one model call misses the memory reads, tools, and sub-loop that the failed step triggers, so the budget should apply to the whole subtree and end in an explicit human handoff. Progressive compaction plus a hard preflight check before context overflow is much safer than waiting for one emergency summary.
Great breakdown. The biggest cost of AI agents isn't just tokens it's the cumulative impact of retries, latency, and orchestration in production.
@mariaandrew, exactly—the unit that matters is the whole execution path, not the headline token price of one call. Retries, serial tool latency, memory growth, and fallback branches can turn a cheap model request into an expensive task even when the final answer looks normal.
The retry loop is the one that quietly kills the budget, exactly like Wren said. One lever that rarely comes up though: prompt caching. If your context is mostly stable across retries, caching the prefix cuts the input cost of each re-attempt instead of re-billing the full history every time. It won't fix a bad retry policy, but it takes the sting out of the ones you can't avoid.
@valentin_monteiro, prompt caching is a valuable lever when the prefix is stable. I would still track cached and uncached input separately and make invalidation visible, because a small system or tool-schema change can erase the expected savings. It complements a good retry policy; it cannot compensate for carrying unnecessary failed context forever.