Every turn of a coding agent can resend the same expensive material: repository conventions, tool schemas, security rules, dependency snapshots, and a long task history. If that stable context is processed from scratch on every tool call, the agent gets slower and costs more precisely when it starts doing useful multi-step work.
Prompt caching for AI coding agents is the practical fix, but it is not a switch you turn on and forget. A cache only helps when you design a stable request prefix, keep changing data out of it, and measure whether real requests are actually hitting it.
This guide shows how to use LLM prompt caching to reduce AI-agent cost and latency without accidentally turning every request into a cache miss.
The useful mental model: stable prefix, volatile tail
Most provider caches reuse a matching prefix of the request. For a coding agent, split each request into two zones:
| Put early in the stable prefix | Put late in the volatile tail |
|---|---|
| System instructions and coding standards | The developer’s current task |
| Versioned tool definitions | Current tool results |
| Repository architecture summary | Timestamps, request IDs, trace IDs |
| Reusable policy and test conventions | Newly retrieved files or diff hunks |
| Stable few-shot examples | The latest conversation turn |
The principle is simple: common, large, slowly changing context first; per-request data last. OpenAI’s documentation describes caching as a matching-prefix mechanism. Google likewise advises putting large common content at the beginning of the prompt to improve implicit cache hits. OpenAI’s prompt-caching guide and Google’s context-caching guide are worth reading before you assume a provider will cache a particular request shape.
Why coding agents are unusually good caching candidates
A chat app may have a short system prompt. A production coding agent often carries much more repeated input:
- a repository map and language conventions
- tool schemas for search, test, git, and issue tracking
- an approval policy for write and shell actions
- examples of the desired plan or patch format
- a task-specific working set that persists across several steps
That creates a repeat workload. The first request may create or warm a cache; later requests can reuse the shared prefix if it stays compatible. This is especially useful for an agent that plans, inspects files, runs tests, and revises a patch in one session.
It is not a replacement for better context selection. Caching irrelevant 100 KB of repository text still makes the model reason over irrelevant material. First reduce the context to what the task needs, then cache the stable portion.
A request shape that protects cache hits
Treat prompt assembly as an interface, not a string concatenation exercise. Version the stable prefix and append dynamic inputs after it.
const stablePrefix = [
'agent-policy:v3',
REPOSITORY_CONVENTIONS,
TOOL_SCHEMAS_V7,
SECURITY_AND_TEST_RULES,
FEW_SHOT_PATCH_EXAMPLES,
].join('\n\n');
const request = {
model: process.env.AGENT_MODEL,
instructions: stablePrefix, // provider-specific field
input: [
{ role: 'user', content: currentTask },
{ role: 'user', content: latestToolResult },
],
};
// Send with your provider SDK, then emit cache telemetry from its usage object.
const result = await runAgentStep(request);
recordMetric('agent.cache_read_tokens', result.cachedInputTokens ?? 0);
recordMetric('agent.cache_write_tokens', result.cacheWriteTokens ?? 0);
The exact field names and controls are provider-specific. Anthropic supports automatic caching and explicit cache breakpoints via cache_control; its docs also document minimum cacheable lengths, TTL choices, pricing, and the limit on explicit breakpoints. See the official Anthropic prompt-caching documentation.
The architecture is portable even when the API is not:
- Build one deterministic stable prefix.
- Put it before task and tool-result data.
- Version it deliberately, for example
tool-schema:v7. - Log cache-read and cache-write usage from the provider response.
- Change one layer at a time when you investigate misses.
Three cache-miss traps in agent harnesses
1. Dynamic metadata at the front
A timestamp, UUID, user-specific greeting, or trace ID before the shared instructions can make otherwise identical requests diverge. Put observability metadata in your application logs or at the request tail, not in the prefix you want reused.
2. Rebuilding tool schemas every turn
If tool definitions are semantically the same but serialized in a different order, a prefix-based cache may not match. Canonicalize the order, pin a schema version, and only regenerate the stable block when the actual tool contract changes.
3. Mixing long-lived policy with rapidly changing retrieval
A repository-wide policy can be stable for days. The top ten retrieved code chunks may change every turn. Keep those as separate layers: cache the policy and durable repository summary, then append fresh retrieval just before the user task. Do not force a large per-turn RAG payload into the stable prefix just because it is expensive.
Measure outcomes, not just cached tokens
A high cache-read count is encouraging, but it is not the business metric. Add cache fields to the same trace you use to evaluate an agent run:
agent_run_id
model
prompt_version
total_input_tokens
cached_input_tokens
cache_write_tokens
latency_ms
cost_usd
completed_task
unsafe_action_blocked
Then compare cache-hit rate, p50/p95 latency, and cost per completed task for a stable workload. This connects naturally to a trace-to-regression workflow: if a prompt refactor increases cache misses or makes task completion worse, catch it before rollout. I covered that evaluation loop in Stop Replaying Coding-Agent Bugs by Hand.
Also keep caching separate from concurrency control. A cheap cached request can still fan out into hundreds of subagents. Pair cache telemetry with budgets and queue limits, such as the approach in Claude Code 2.1.212: Put Hard Budgets on Agent Fan-Out Before Costs Run Away.
When prompt caching does not apply
Prompt caching is a poor fit when:
- each request has little repeated input
- the shared prefix is below a provider’s cacheability threshold
- every request changes the policy, tool list, or retrieved context
- a one-off batch task will not reuse the context enough to recover any cache-write or storage cost
- the security model forbids the relevant retention behavior
Check provider retention, data-handling, TTL, and pricing details rather than treating caching as a universal discount. OpenAI, for example, documents cache retention and cache-write accounting for its newer model families; Anthropic documents distinct write/read pricing and TTL options; Google documents model-specific implicit-cache thresholds and exposes cached-token usage.
What to do now
- Pick one repeat-heavy workflow, such as a PR-fix agent that makes several tool calls per task.
- Log the full request structure for a safe test workload, then label each part stable or volatile.
- Move the stable material to a deterministic prefix and give that prefix a version.
- Turn on or configure the provider’s supported caching mechanism.
- Add cache-read, cache-write, latency, and cost-per-completed-task fields to your traces.
- Run a small canary against representative tasks and compare it with the uncached baseline.
- Add a regression case for a known cache miss, just as you would for a tool failure or unsafe action.
For multi-agent systems, make this contract explicit at the coordinator boundary. Reusing the same policy and tool contract is one reason a well-scoped team can be easier to operate than a pile of ad hoc subagents. See Claude Code Agent Teams: When Shared Context Beats More Subagents.
Sources
What is the most expensive stable context your coding agent currently resends on every step: tool schemas, repository guidance, or something else?
Top comments (0)