Semantic caching for AI agents fixes something that should embarrass all of us: an agent paying full price to answer a question it already answered an hour ago. The open question is not whether to build the cache. It is where to keep it.
The matching logic is nearly the same wherever you keep it. What changes is how fast a lookup comes back, what you pay while nobody is asking anything, whether the agent has to live inside a private network (a VPC, or Virtual Private Cloud), and what each store makes you work around. I built the same cache on two stores, Amazon DynamoDB vector search and Amazon ElastiCache for Valkey, and what follows is what differs.
This post continues a series: the opener maps all five layers an agent can cache and why prompt caching reaches none of them. Start there if you have not, because this one goes straight to the storage decision.
All the code is in this repository: two tracks, cache-dynamodb and cache-valkey, shipping the same agent, tools and web UI. Each has notebooks that run on nothing but AWS credentials and a production stack on AWS CDK (Cloud Development Kit). Deploy steps, costs and troubleshooting stay in the track READMEs. Built on Strands Agents; the patterns carry over to other frameworks.
⚠️ Assumes familiarity with AI agents, Amazon Bedrock and AWS CDK (Python). The two storage sections are independent: read the one that matches your workload.
What is identical on both stores
Four things happen on every request, none of which depend on where you keep the data:
- Turn the question into numbers. An embedding model turns text into a list of numbers that stands in for its meaning, so questions that mean similar things get similar lists (Amazon Titan Text Embeddings V2 here).
- Look for the closest question you have already answered.
- Hit, only if two checks pass: the two questions are close enough (you set that bar, 0.85 out of 1 by default) and the dates and numbers inside them match exactly. The stored answer comes back and the agent never runs.
- Miss, if either check fails. The agent runs, and the new question and answer go into the store with an expiry (a TTL, Time To Live).
The second check is where a semantic cache usually goes wrong, because it is the one people skip. Closeness tells you two questions are worded alike; only the exact check tells you they are about the same thing.
One trap catches everybody, identically on both stores: the number that comes back is a distance, not a similarity. It says how far apart the two questions are, so 0 means identical and 2 means opposite. Compare it against a bar like 0.85 and nothing ever hits. Flip it around first, or lose an afternoon convinced vector search is broken:
similarity = 1.0 - (score / 2.0)
if similarity < threshold:
return None # miss: run the agent
The cache is code, not model behaviour
Every cache here is a small class of your own, wired to fixed moments in the agent's life. One looks something up before the agent starts, one stores the answer after it finishes, one serves a repeated tool result. In the notebooks all three are Strands hooks; in production the reasoning cache is a hook and the response cache wraps the agent. The model is never told a cache exists.
So reusing an answer is a predictable decision: nothing asks a model whether two questions mean the same thing, and the verdict is the if above plus a comparison of the dates and numbers. The search is approximate, so which candidate comes back can vary; what your code does with it cannot.
It also means the pattern does not care which model you run. The embedding model and the agent's model are both settings, and no cache class knows what they are set to. In production the response cache filters the lookup by model id, so one model's answer is not served as if another had written it.
What a hit costs, on either store
A word-for-word repeat costs 0 LLM (Large Language Model) tokens unless a guard below has to rewrite the stored answer. Cheap, not free: three things still bill on both tracks, one embedding call for every question that arrives, hit or miss, a small fraction of a cent; one lookup; and storage until the entry expires, which makes that expiry a spending control as much as a freshness one.
What goes away is the model call and the agent loop behind it. On repetitive traffic that trade is lopsided in your favour. Where every question is new, you pay the embedding call every time and hit almost nothing, so check how often your users repeat themselves before building either version.
Reusing the thinking, not the answer
A response cache needs the question itself to repeat. A third cache fires when only the thinking repeats, and both tracks ship it two ways. The reasoning cache hints before the first step: a similar question used these tool calls, arguments already resolved, so the agent still thinks, now pointed in the right direction. The plan template cache stores the finished recipe, a tool-call sequence with slots, so a cheap call fills in today's city and date and the planning loop never runs.
Which one fits your workload
Not a ranking. Two things decide it.
Your traffic shape. A node bills around the clock, busy or not. The table has no node to pay for, but idle is not free there either: what you store bills per GB-month, and the vector index bills on top of the table it sits on. Sustained traffic pays for a node without noticing; bursts, or a demo that runs twice a week, do not.
Speed matters on the misses, not on the hits. Any store beats an LLM call, so hits feel fast either way. But the lookup runs on every question, misses included, so a slow store taxes every user to save the occasional one. That is the real reason to pick in-memory.
If neither answer is obvious yet, start serverless: nothing to size and no network to build, so a wrong guess is cheap to undo.
The serverless store is one DynamoDB table
SearchVectors is a DynamoDB API (search_vectors in boto3) that searches a vector column on an ordinary table and hands back the nearest matches. You pass the table, the index, the question's vector, how many matches you want, and a filter that scopes the search. No separate vector database, nothing to keep in sync, and the full call is in the repo.
Read the requirements and limitations before you design around it.
DAX (DynamoDB Accelerator) takes eventually consistent reads "from single-digit milliseconds to microseconds", but does not support SearchVectors: the semantic lookup goes straight to the table. It would still speed up the tool-result cache, a plain key lookup on the same table.
The write side is eventually consistent, which matters when the index is a cache. AWS documents "a brief delay between writing or updating a vector and it appearing in search results", so a repeated question arriving right behind a miss can miss too. Fine for a cache, awkward when you go to demo one. The notebook in this track waits three seconds after a cold run for that reason, and the Valkey one needs no wait.
Three cache patterns in one table
Every item carries an entry_type, declared as a filter on the vector index, so one search scopes itself to one pattern. Items with no vector never show up in vector results.
| Pattern | entry_type |
What it stores | What a hit saves |
|---|---|---|---|
| Semantic response cache | response |
Full answers keyed by question embedding | The entire agent loop |
| Tool-result cache | tool_result |
Tool outputs keyed by hash(tool_name + args)
|
The external API call |
| Reasoning and plan caches |
plan, trajectory
|
Tool-call sequences to reuse | The planning loop, or most of it |
One client, one table, and cdk destroy removes all of it.
The TTL trap that shows up after deployment
Here is how a serverless cache quietly serves stale data: a read can still return an item whose expiry has passed. Not a bug, and not a small window either. AWS deletes expired items "typically within a few days after their expiration". Harmless for an answer that ages slowly. For a flight price with a five-minute expiry, you are serving old prices as fresh and nothing raises an error.
So trust the expiry to clean up, never to be correct. Every read of a cached tool result here compares the stored expiry against the clock, one line, in the notebooks and in the production stack. A stale entry becomes a miss even though the item is sitting right there in the response. A live price, though, does not need a shorter expiry, it needs no cache. Cache what holds still, a city's coordinates for weeks, and keep the moving numbers out of it.
The in-memory store is ElastiCache for Valkey
Vector search on Valkey lives in a search module whose commands all start with FT.. The index is created once, on the first cold start, and it has to be told the same vector size the embedding model produces. Lookups then ask it for the nearest match, filtered by model.
One storage detail matters at scale: the vector and the answer live under two separate keys, so long answers stay out of the index. Both expire together, with a little jitter so a batch cached at the same moment does not all vanish in the same second.
The guards that keep either cache honest
Closeness alone will serve wrong answers, and that is a property of closeness, not of storage, so each guard below is the same code on either store.
The critical-parameter guard earns its keep. "Flights on 2026-09-15" and "flights on 2026-12-15" are nearly the same sentence, and the store rates them ~0.97 alike in the repo's calibration harness, far above any bar you would set. Research on time-sensitive caching calls this the main way semantic caches fail (arXiv:2605.20630). So the guard pulls every date and number out of both questions and demands they match exactly, while the wording stays free to vary: the embedding handles phrasing, the guard handles values. It is blunt in the safe direction, because a false miss costs one agent run and a false hit costs your credibility. It only sees digits, so "flights to Tokyo tomorrow" has nothing to compare and matches the same question asked last week; resolve dates before the cache sees the question.
The other three:
- Prompt-hash self-healing. Every entry remembers which system prompt produced it. Change the prompt and the old entries are not thrown away: the first hit rewrites that answer under the new prompt and stores it back.
- Verified near-miss promotion. A question landing just under the bar is not discarded. The agent answers it, and that answer is compared against the entry it nearly matched. If the two agree, the new question becomes a second way to reach that entry, so the cache widens from real traffic. The paper behind it verifies asynchronously; here the check runs inline after the miss, for two extra embedding calls.
- Fail open, always. Store unreachable, embedding call failed, search module missing (checked, never assumed): the agent runs normally. The cache is an optimization, not a dependency.
All four are in each track's production cache class; the first and last also run in the notebooks.
Is a shared semantic cache safe for personal data?
Not by default. Treat both tracks as demos. One cache serves everybody: right for factual answers, wrong the moment answers depend on who is asking. Personal data in a cached answer can reach the next person whose question is merely similar, and text from an untrusted page can plant instructions that get cached and replayed. Three things before production:
- Check answers before they go into the cache, the same discipline as gating what an agent writes to memory, and screen tool outputs for prompt injection first.
- Look for personal data (PII, Personally Identifiable Information) on the way in, with Amazon Comprehend or Bedrock Guardrails sensitive-information filters, and skip or redact anything flagged.
- Give each tenant its own cache.
Deploy whichever track matches your traffic, ask the same question twice, and watch the second answer skip the model. The store and the embedding call still bill; the model is what you stop paying. On the serverless track, leave a moment between the two.
Then tell me in the comments: is your agent's traffic spiky or sustained, and did that decide it?
Resources
- Sample repository: semantic and reasoning caches for AI agents
- Amazon DynamoDB vector search (AWS documentation)
- Semantic caching with ElastiCache (AWS documentation)
- Well-Architected Agentic AI Lens: agent caching layers
- Strands Agents hooks documentation
Gracias!
🇻🇪 Dev.to Linkedin GitHub Twitter Instagram Youtube
Top comments (0)