Cutting Agent Token Costs Without Breaking Task Quality

I treat an agent run as the unit of cost, not an individual model request. A chatbot may make one call per user message; an agent can make 10, 20, or more before finishing. Every call may reprocess instructions, history, tool results, and intermediate state. Retries, reasoning, and workers add to that bill even when the final answer is short.

My primary metric is cost per successful task: total workflow cost divided by accepted tasks. A cheaper request is irrelevant if it produces more failed runs, repeated tools, or human correction. I would clean up the loop before changing models.

Start With the Cost of Repeated Context

Consider a hypothetical support agent: a 4,000-token stable prefix, 1,500 new tokens added after each step, complete history resent on every request, and 12 model calls.

Input at step n = 4,000 + 1,500 × (n - 1)
Total input = 4,000 × 12 + 1,500 × (0 + 1 + ... + 11)
            = 48,000 + 99,000
            = 147,000 input tokens

The final request contains just 20,500 input tokens. Looking only at that request hides most of the run's input volume. Cache the stable prefix after the first call, then compact history after step six into a 2,500-token state summary, and the accounting changes:

Configuration	Uncached input	Cached input	Total processed input
Full history	147,000	0	147,000
Cached stable prefix	103,000	44,000	147,000
Cached prefix plus compaction	64,000	44,000	108,000

Caching changes the price mix without reducing processed volume. Compaction cuts that volume by 26.5% in this example. This is a planning calculation, not a provider benchmark; selective retrieval and state construction produce different curves. Cached tokens still occupy context. Caching is not a substitute for making the prompt smaller.

Instrument the Run Before Tuning It

I want enough telemetry to reconstruct both the execution tree and its costs. Record run_id, step_id, and parent_step_id; rendered input and cached versus uncached tokens; output and reasoning tokens; raw tool-result size and retained tokens; retry reason and attempt; compaction size before and after; worker IDs and returned tokens; and whether the result was accepted, rejected, or escalated.

Four ratios help explain cost per successful task: context amplification is cumulative input divided by final-step input; tool retention ratio is retained tool-result tokens divided by originally returned tool tokens; retry tax is retry and repair cost divided by workflow cost; reasoning share is reasoning-token cost divided by total model cost. High amplification points to repeated context, while high retention can flag excessive raw evidence. I keep separate baselines for research, coding, browser, and support agents.

Put a Budget Around the Whole Loop

A response limit does not constrain an agent that can keep calling models and tools. Budget total steps, cumulative input and output, tool calls and result sizes, retries by failure type, subagents, and elapsed time or estimated cost. This provider-neutral check covers the model-step and token portion:

from dataclasses import dataclass
from enum import Enum

class Action(str, Enum):
    CONTINUE = "continue"
    COMPACT = "compact"
    STOP = "stop"

@dataclass(frozen=True)
class Budget:
    max_steps: int = 12
    max_input_tokens: int = 120_000
    max_output_tokens: int = 18_000
    compact_at: float = 0.80

@dataclass
class Usage:
    steps: int = 0
    input_tokens: int = 0
    output_tokens: int = 0

def evaluate_budget(usage: Usage, budget: Budget) -> Action:
    if (usage.steps >= budget.max_steps
        or usage.input_tokens >= budget.max_input_tokens
        or usage.output_tokens >= budget.max_output_tokens):
        return Action.STOP
    input_ratio = usage.input_tokens / budget.max_input_tokens
    return Action.COMPACT if input_ratio >= budget.compact_at else Action.CONTINUE

Run it before each model request and update Usage from provider-reported tokens. At 80% of the input budget, compact state or narrow the next query; at 100%, stop with a structured reason. This checks usage already accumulated, so it does not by itself guarantee that the next request stays within the remaining allowance.

Control What Survives Into the Next Prompt

Filter Evidence at the Tool Boundary

I would first inspect search pages, logs, repository trees, database responses, terminal sessions, and API payloads. The next decision usually needs selected evidence, not the entire artifact. A compact search result can contain source_id, title, url, and relevant_passage; keep the complete artifact outside the prompt and retrieve narrower sections when needed. Do not truncate the first 1,000 characters of JSON. Parse it, select fields, limit arrays, and serialize valid JSON. Arbitrary truncation can break structure or discard the relevant records.

Compact State, Not the Story

A useful summary preserves the goal and success criteria, decisions, verified facts and source IDs, changed files or records, failed approaches, open questions, next action, and safety and output constraints. It should not narrate the conversation. OpenAI documents compaction for long-running Responses API interactions; Anthropic offers context-management controls for clearing or summarizing older content. Their implementations differ, so verify current provider fields. Compact before growth hurts cost, latency, or available output space, then check for repeated searches and tool calls. Losing state can cost more than retaining it.

Keep Reusable Input Stable

Order the prompt as system instructions, policies and constraints, tool definitions, stable examples, shared reference material, then request-specific data. Keep timestamps, request IDs, session data, and other changing values away from the beginning. Long, stable, reused prefixes are the useful caching case; short sessions or frequently changing prompts may not save money. Measure cache-write, read, and storage costs, not just hit rate.

Stop Paying for Repeated Failures

Retries need failure-specific handling. For invalid structured output, return the validation error and retry once. For a tool timeout, retry an idempotent operation once, then stop or fall back. Context overflow needs compaction or less evidence, not an unchanged request. Deduplicate repeated tool calls with an operation hash. Rate limits need backoff or a tested fallback route; low-confidence results may need missing information or escalation.

Use idempotency keys for side effects such as payments, emails, deployments, and database writes. Track retry tax by failure type so the largest recurring failure gets fixed first. Repeatedly resending a large context without addressing the failure is not a recovery strategy.

Allocate Reasoning and Workers Deliberately

Extraction, formatting, classification, validation, and routine tool selection can often use lower reasoning effort and compact structured output. Reserve higher effort for complex planning, difficult coding, multi-document synthesis, ambiguous decisions, and recovery from failed execution. The target is the lowest effort that preserves the accepted-task rate.

Workers need a narrow task, task-specific context, a tool allowlist, a token budget, and a compact output schema. The root agent generally needs findings, evidence IDs, confidence, and unresolved issues, not a worker's full transcript. Subagents may improve coverage or elapsed time, but duplicated context and overlapping analysis can increase total tokens. Parallelize independent work without distributing the entire root history.

Evaluate Changes Against Accepted Tasks

My rollout order is telemetry, run limits, tool filtering, measured compaction, stable prefixes, then model-route comparisons. Let observed waste adjust that order: high retry tax calls for failure handling; high reasoning share calls for effort tuning; duplicated evidence calls for narrower worker scopes; low cached input calls for prefix inspection. These are investigation signals, not universal thresholds.

For multi-model evaluations, a unified API such as CometAPI can simplify route comparison and fallback integration. Estimate input, output, cached-token, and reasoning costs before testing. Switching routes must preserve validated state and avoid repeating completed tool calls; budgets, validation, compaction, and acceptance criteria remain application responsibilities.

Change one major variable at a time and replay the same evaluation set. Compare acceptance rate, cost per successful task, cumulative input, tool-call count, retry tax, reasoning share, p50 and p95 latency, and human-review time. I would roll back any token reduction that removes necessary evidence or lowers task quality. The useful optimization is a cheaper accepted outcome, not a smaller prompt in isolation.

---

Originally published at cometapi.com