Agent Cost Optimization: Token Accounting, Cache, and Dynamic Dispatching

In March 2026, I paid 6x the expected API bill for the month. The pipeline validator ran on Sonnet because "Sonnet is the default, I'll optimize later." Later never arrived. The validator had a retry loop that re-entered on any malformed JSON. The model, occasionally, would wrap the JSON in a markdown fence (

json ...

), the parser would break, the loop would re-enter. Each item became 4-6 attempts instead of one. Three million tokens per day before Anthropic's billing alert arrived. I had no alert of my own.

Spend caps are not optional. Retry instrumentation is not optional. And "Sonnet is the default" is laziness dressed as engineering.

Agent cost scales asymmetrically. A five-minute session with Sonnet might cost two cents. The same operation run fifty thousand times a month costs $1,000 without you noticing. The individual example doesn't hurt. The tail multiplies.

Four mechanisms: token accounting, prompt caching, batch processing, dynamic dispatching.

Token accounting: input always weighs more

Each call charges two token types: what you sent (input) and what the model returned (output). Output costs 5x more per token than input across Anthropic's entire model line (May 2026 pricing). In practice, the bill comes from accumulated input, not output.

Why: each turn resends the entire conversation history. In a thirty-turn session with Sonnet ($3 input / $15 output per Mtok), output disappears in the bill. What hurts is the input that doubles on every round-trip.

First optimization: shorten persistent input. Lean system prompt, pruned history after N turns, truncated tool results when they return too large.

Tools returning too much inflate the next turn:

const MAX_TOOL_CHARS = 12_000;  // ~4k tokens in English code

function truncateToolResult(output: string): string {
  if (output.length < MAX_TOOL_CHARS) return output;
  const head = output.slice(0, MAX_TOOL_CHARS / 2);
  const tail = output.slice(-MAX_TOOL_CHARS / 2);
  return `${head}\n\n[... ${output.length - MAX_TOOL_CHARS} chars truncated ...]\n\n${tail}`;
}

The model receives beginning plus end, knows there was a cut, can request the tool again with a narrower scope.

Prompt caching

Anthropic offers prompt caching with cache_control: { type: "ephemeral" }: mark a prefix as cacheable and subsequent calls within the TTL reuse that part at a discount.

Actual pricing (Anthropic docs, 2026-05):

Operation	Cost vs normal input	TTL
Cache write (first time)	1.25x	5min
Cache read (subsequent hits)	0.1x	5min
Cache write (extended TTL, beta)	2x	1h
Cache read (extended TTL)	0.1x	1h

Break-even: cache pays off starting from the second hit within the TTL. For single-call flows, write costs more than no cache.

const response = await anthropic.messages.create({
  model: "claude-sonnet-4-6",
  system: [
    {
      type: "text",
      text: STABLE_SYSTEM_PROMPT,
      cache_control: { type: "ephemeral" }  // cached up to here
    },
    { type: "text", text: SESSION_CONTEXT }  // variable, not cached
  ],
  messages: [{ role: "user", content: userPrompt }],
});

The rule that generalizes: order from most stable to most volatile. System prompt and initial history go in cache; user messages and recent tool results don't.

In a multi-agent pipeline, Writer, Editor, and Validator typically share 60-80% of the system prompt (ADRs, style guide, editorial voice). These blocks cache once per run, then cost 10% thereafter. A pipeline costing $0.30 per item drops to $0.10-0.15.

The confusing behavior: cache works by cumulative prefix-match. Changing the global system prompt invalidates the chapter context cache too. Changing the chapter context doesn't affect the system prompt above.

Batch processing

For work that doesn't need an immediate response, batch is a 50% discount traded for up to 24h of latency:

const batch = await anthropic.messages.batches.create({
  requests: chapters.map((ch) => ({
    custom_id: `validate-${ch.slug}`,
    params: {
      model: "claude-sonnet-4-6",
      max_tokens: 1024,
      messages: [{ role: "user", content: validatorPrompt(ch) }]
    }
  }))
});

const results = await anthropic.messages.batches.results(batch.id);

Batch wins for: nightly re-validation of all content, eval suites in CI, overnight drafting to wake up with everything ready. Composes with caching: the batch accepts cache_control on the system prompt.

Batch loses for: interactive flows where the operator waits now, step-dependent pipelines where each call needs the previous result in seconds.

Heuristic: if you can wait an hour, use batch.

Dynamic model dispatching

A cheap classifier decides the tier dynamically:

async function smartDispatch(task: Task): Promise<Result> {
  const classification = await llm.generate({
    model: "claude-haiku-4-5",
    max_tokens: 30,
    prompt: `Classify this task's complexity:\ntrivial | standard | complex\n\nTask: ${task.description}\n\nAnswer (one word):`,
  });

  const model = {
    trivial: "claude-haiku-4-5",
    standard: "claude-sonnet-4-6",
    complex: "claude-opus-4-7",
  }[parseTier(classification.text)];

  return await llm.generate({ model, prompt: task.fullPrompt });
}

The classifier runs in ~200ms at negligible cost. Savings depend entirely on the real load distribution: if 70% of tasks are trivial, you save a lot. If 90% are complex, you only added overhead. Measure before assuming.

Anti-pattern: Sonnet for everything

Operational laziness: leaving Sonnet on everything because it's the technical default. A 200-item nightly pipeline costs $80 when it could cost $12 with proper dispatching.

Diagnosis: you didn't measure. How much of this load would Haiku handle? A Validator with 800-token output running expensively on Sonnet, when Haiku with the right prompt reaches close to the same judgment at 10% of the cost.

Fix: instrument first. Log per call: model, milliseconds, input tokens, output tokens, judge score. In a week you have real distribution. Only then decide dispatching based on data.

A cap being hit is a signal, not an obstacle. If you have MAX_TOKENS_PER_DAY = 1_000_000 and hit it regularly, the dispatching is wrong, the prompt grew, or a retry got stuck in a loop. Bumping the cap without investigating doubles cost without fixing the problem.