You turned on prompt caching expecting your repeated questions to get cheap, and your input tokens did get a discount. But the model still wakes up, still reasons through the task, still calls every tool, and still writes the whole answer from scratch, every single time, even when someone asks the exact same question it answered a minute ago. Prompt caching discounts the input you send again. It never reuses the answer.
Here is the part that stings. Your agent already knows the answer to a lot of what it is asked. Someone asks "what documents do I need to travel to Japan?" in the morning, and by the afternoon three other people have asked the same thing in three different wordings, and your agent pays full price for all four. The real savings do not live in the input tokens. They live in the work you can skip, the answer you already generated, the plan you already figured out, the API you already called. Prompt caching cannot reach any of that, because it never looks at meaning.
That is the layer this series is about. When you cache by meaning instead of by exact text, a repeated question comes back in milliseconds with no generation, and a new-but-similar question skips most of the exploration the agent would otherwise redo. In this first post I map where an AI agent can cache, show you the application-level caches that eliminate work instead of discounting it (the semantic response cache and the reasoning cache), and share the measured results and the traps from a deployment.
This is the first post of a series. All the code is in this repository, on two interchangeable backends. You can start with the local Jupyter notebooks, which cache a Strands agent from your machine with nothing but AWS credentials (no CDK, no VPC), and the production stacks deploy the same pattern with AWS CDK (Cloud Development Kit). The next two posts cover each backend in depth. It's built on Strands Agents.
What makes all of this simple is where the caching lives. It is not a wrapper bolted around the agent; it plugs into the agent's own lifecycle. Strands exposes two capabilities that carry the whole design. Hooks let you subscribe to events across the agent loop and react to them: a hook at the start of a request can answer from cache and stop the model before it runs, a hook before a tool call can hand back a stored result so the real tool never fires, and a hook at the end can capture what happened for next time. Memory gives the agent durable knowledge that persists across sessions, which is where reused plans and trajectories live. The caches are ordinary Strands components; the only call your application makes is still agent(question). The next posts show how; this one is about what and why. The code is here and the capabilities are documented in the Strands hooks and Strands memory guides.
⚠️ This post assumes familiarity with AI agents.
Why isn't prompt caching enough?
Every major model provider ships prompt caching. The processed prefix of your prompt is reused, so you pay less for repeated input tokens. It's valuable, and it never returns a stored response. In the providers' own words, "Prompt caching has no effect on output token generation" (Anthropic), and "Prompt caching does not change how the model generates output tokens" (OpenAI).
Watch what one repeated question costs. Your agent answered "What's the weather in Madrid?" three seconds ago. A second user asks "How's Madrid looking weather-wise?" and the agent runs the full loop again, planning cycles, tool calls, and generation. A third user asks the same thing in Spanish, "¿Qué tiempo hace en Madrid?", and pays full price a third time. Prompt caching discounted the input prefix and nothing else, and a different wording or a different language is a different prefix, so it never matches. Conversation management trims history, but it cannot detect that the question itself is a paraphrase of one already answered. Same answer, full price, three times.
Where an AI agent can cache
An AI agent can cache at five layers. Two you get for free (the model provider gives you prompt caching, your agent framework gives you conversation management); the other three you build. This sample builds those three:
| Layer | What it saves | Who provides it |
|---|---|---|
| Prompt caching | Input-token price on repeated prefixes; the model still generates every response | The model provider |
| Conversation management | History tokens re-sent on every turn | Your agent framework |
| Semantic response cache | The whole generation on a repeated question (0 tokens on a hit) | You (this sample) |
| Reasoning cache | Planning cycles on a new-but-similar question (the model still generates) | You (this sample) |
| Tool-result cache | The external API call itself: its latency, third-party cost, and rate limits | You (this sample) |
Prompt caching comes from the model provider (you enable it, the provider does the caching) and conversation management comes from your agent framework (Strands ships sliding-window and summarizing managers); this sample does not reimplement either. It builds the last three, the application-level caches. They are where the real savings are, because they skip work instead of discounting it: the response cache skips the whole generation, the reasoning cache cuts planning cycles, and the tool-result cache skips the external API call. The decision framework is one question per layer. Does the question repeat (response cache), does the reasoning repeat (reasoning cache), or does the tool call repeat (tool-result cache)?
How does a semantic response cache work?
A semantic response cache matches incoming questions to previously answered ones by meaning, not exact text. Embed the incoming question with an embedding model, run a vector search for the nearest previously answered question, and on a hit above a similarity threshold (0.85 by default in the sample) return the stored answer. Zero generation. On a miss, run the agent and store the new pair with a TTL (Time To Live).
Similarity alone will lie to you, so the sample adds three guards:
- Critical-parameter guard. "Flights on 2026-09-15" and "flights on 2026-12-15" score ~0.97 cosine similarity in the repo's calibration harness, close enough that the embedding treats them as the same question. The wording can still vary freely (that is what the embedding is for); only the dates and numbers extracted from both questions must match exactly. Same dates, different phrasing is a hit; same phrasing, different date is a miss.
- Rewrite mode. On a hit where the cached answer is in a different language than the question, one cheap call re-expresses that already-verified answer in the question's language. It does not re-run the agent or the tools and does not research or add facts, it only translates the stored answer (a Spanish question against an English cached answer cost ~195 tokens for the translation, versus a full agent run). If the answer is already in the right language it is returned unchanged.
- Fail open. If the cache store or the embedding call fails, the agent runs normally. The cache is an optimization, never a dependency.
How does a reasoning cache work?
The response cache fires when the question repeats. The reasoning cache fires when the question is new but similar. The answer changes, yet the trajectory (which tools, in what order) is stable. Weather for Madrid and weather for Rome need different data from the same two tool calls.
The sample builds it with agent lifecycle hooks. When a similar question arrives, the hook injects the known plan and tool trajectory before the first cycle, so the agent goes straight to the right tools instead of rediscovering them. Repeated tool calls are served from the tool-result cache, with freshness policies matched to each tool's volatility. Geocoding can live for weeks, weather for hours, prices for minutes, and a stale result is served on API error rather than failing the run.
What did they save in a demo?
Measured on the deployed sample (Amazon Nova Lite, us-east-1), verified August 2026:
| Metric | Cold run | Warm run | Saved |
|---|---|---|---|
| Reasoning: event-loop cycles | 5 | 2 | 60% |
| Reasoning: total tokens | 7,000 | 2,965 | 58% (4,035 tokens) |
| Reasoning: tool executions | 3 | 0 | 100% |
Across test runs the warm savings ranged from 40% to 85% of tokens and cycles, because cold-run exploration is model-driven; tool-execution savings stayed stable. For an external anchor, AWS's published benchmark for semantic caching reports up to 86% cost savings and 88% latency reduction.
The traps that cost me time
- The first iteration saved nothing. The plan hint was injected in a way the agent ignored, and cold and warm runs cost the same until the hint prompt was fixed. Measure savings from real runs; never assume the hint landed.
- Cold-start measurement mistakes. The first request pays index creation and connection setup. Benchmark hits and misses separately, after warm-up.
- Threshold tuning is data, not folklore. Every hit in the sample reports its similarity score and near-misses are logged, so the threshold is tuned from real traffic instead of guesses.
- A cached answer can be wrong tomorrow. The critical-parameter guard keeps date-specific answers apart, and per-tool TTLs expire volatile data (a flight price lives minutes, a geocode lives weeks) while stable answers stay cached.
- A shared cache is a security surface. One user's cached answer can contain personal data another user's similar question retrieves, and content read from untrusted sources can plant instructions that get cached and replayed. Detect PII (Personally Identifiable Information) at the cache boundary and validate what gets written, the same discipline as validating before an agent writes to memory, keeping poisoned content out of the vector store, and stopping prompt injection from untrusted tool output.
Which backend should you deploy on?
The repository ships the same agent, tools, and web UI on two tracks. They share the response and tool-result caches, and each one reuses reasoning its own way. One hints the agent while it thinks, the other saves the finished plan and reuses it as a template. Pick by workload, not by ranking.
| Track | Best for |
|---|---|
| In-memory (ElastiCache for Valkey vector search) | Sustained hot-path traffic, lowest lookup latency; runs in a VPC (Virtual Private Cloud) |
| Serverless (Amazon DynamoDB vector search, one table) | Spiky traffic, no idle compute cost; no VPC |
"In-memory" and "serverless" are not just labels for the same thing with a different name. In-memory means the vector index and the cached values live in a running node's RAM (ElastiCache for Valkey), so a lookup is a sub-millisecond read and never touches disk. That speed is the point on a hot path, but the node runs and bills whether or not traffic arrives, it sits in a VPC, and its memory is a fixed size you provision.
Serverless (DynamoDB with native vector search) has no node to run: the table scales on demand, you pay per request with no idle floor, there is no VPC, and capacity is not something you size.
The trade is a higher per-lookup latency than a RAM read, though still far below an LLM call. So the real differences are latency floor, idle cost, VPC footprint, and how capacity is managed, not the word on the box. The caching logic, the agent, the tools, and the results are identical on both; only the store underneath changes.
The next post in this series builds the serverless track end to end, and the one after goes deep on the in-memory track with the production guards each backend needs.
FAQ
What is semantic caching for LLMs?
A cache that matches incoming questions to previously answered ones by meaning (vector similarity) instead of exact text. On a match above a similarity threshold, the stored answer is returned and the LLM (Large Language Model) never runs, saving that invocation's tokens and most of its latency.
How is a semantic cache different from prompt caching?
Prompt caching reuses the processed prefix of your input to cut input-token cost; the model still generates every response. A semantic cache skips generation entirely on a hit. They stack; use both.
What's the difference between a response cache and a reasoning cache?
The response cache fires when the question repeats (stored answer, zero tokens). The reasoning cache fires when the reasoning repeats on a new question (known plan and tool trajectory, fewer cycles and tool calls).
Is a shared semantic cache safe for personal data?
Not by default. Validate before writing, detect PII at the cache boundary, and partition per tenant the moment answers depend on who is asking. Treat the sample as a demo.
Deploy the sample, repeat a question, and watch the second one skip the model entirely. Then tell me in the comments: how much of your agent's traffic is questions it already answered?
Resources
- Sample repository: semantic and reasoning caches for AI agents
- Semantic caching with ElastiCache (AWS documentation)
- Amazon DynamoDB vector search (AWS documentation)
- Well-Architected Agentic AI Lens: agent caching layers
- Strands Agents hooks documentation
Gracias!
Top comments (0)