When building AI agents (Claude Code, Cline, Roo Code, or custom multi-step pipelines), API use can rise as context accumulates. What each request contains depends on the client: it can include the system prompt, available tool schemas, message history, and function results.
To control the budget without losing functionality, establish a baseline for one fixed task, identify the main source of excess tokens, and change the agent environment one variable at a time.
The Anatomy of AI Agent Context: What You Pay for on Each Turn
For measurement, split the context sent to the model into four observable components:
- System instructions and rules: style requirements, safety constraints, and repository context.
- Tool schemas: JSON descriptions of connected functions, parameters, and data types, when the client includes them in a request.
- Message history: earlier user messages and agent replies that accumulate while the task runs.
- Tool outputs: read-file contents, terminal-command logs, and API dumps.
Do not blindly multiply the first request by the number of steps. Export usage for every call: history may grow, a client may truncate data, and a provider may account for cached tokens separately.
Breakdown of Context Components and Optimization Strategies
| Context component | What to measure | Main overspend risk | Change for an isolated test |
|---|---|---|---|
| Tool schemas | Size of the list actually sent | Unused tools in the shared set | Retain only tools needed for the task |
| Tool outputs | Size of every result | Reading whole files instead of targeted slices | Limit line ranges and log volume |
| Step history | Input growth from call to call | Results that are no longer needed accumulate | Check the client-supported history reduction |
| System prompt | Size and stability of the prefix | Repeated instructions | Remove duplication while retaining required rules |
Step-by-Step Guide: Profiling Your Baseline and Cutting Token Waste
Use one-variable measurement to make the optimization objective:
First, record current model rates. Use current BetterToken prices for the baseline rather than values from old examples. View current BetterToken prices
Step 1. Define a Reproducible Benchmark Task
Choose a reproducible engineering scenario, such as finding a validation function in a repository, adding edge-case handling, and running unit tests. The task needs a clear completion criterion, for example exit code 0 from pytest or bun test.
Step 2. Capture the Baseline Metrics (Input, Output, Cache)
Run the task in the standard agent configuration and record the number of steps, total input tokens, total output tokens, cached tokens, and cost at current rates.
Check the selected model's current rate on the BetterToken price page. In Workspace, an accepted request can show its model, time, status, input/output tokens, cached tokens where supported, and call cost. If one agent step creates several requests, do not present a request-level record as a finished per-step breakdown; match records by time and your own client data.
Step 3. Modify One Context Variable at a Time
Execute isolated test runs modifying strictly one parameter per trial:
- Experiment A (tool filtering): Keep only tools required for the control task and measure the input-token difference.
- Experiment B (output truncation): Limit terminal output to the first 50 error lines instead of a full long dump.
- Experiment C (prefix stability): If the model and endpoint support prompt caching, keep the system prompt and tool-schema order unchanged, then check the actual cached-token data. ### Step 4. Evaluate Financial Gain vs Solution Quality
Compare final metrics with the baseline. Keep a change only if the control task still meets the same quality criterion and measured time or spend improves on your set of runs.
Best Practices for Agent Architecture
- Narrow tools by role: a read-only agent does not need write functions; test whether this reduces actual input without harming the result.
- Keep a stable prefix: where caching is supported, do not reorder common rules needlessly; check cached tokens rather than assuming a discount.
- Bound the loop: set a finite attempt count and a clear stopping condition appropriate to the task.
Edge Cases and Pitfalls to Avoid
- Pitfall: Stripping Necessary Schema Definitions. Over-truncating tool schema descriptions causes the model to output malformed JSON, triggering retry overhead.
- Pitfall: Relying on Unverified Social Media Benchmarks. Context optimization outcomes vary heavily based on repository size, language, and file density.
- Pitfall: Opaque telemetry. If the endpoint does not return separate input/cache statistics, do not infer cache use; mark it unknown.
Originally published on the BetterToken blog.
BetterToken provides pay-as-you-go access to AI model APIs through
OpenAI-compatible and Anthropic-compatible endpoints — useful if you are wiring
Claude Code, Codex, or your own tooling to a custom base URL.
See the docs to get started.
Top comments (0)