DEV Community

Cover image for Semantic Cache on AWS: Cut LLM Costs Without Serving Wrong Answers
Node & Neural
Node & Neural

Posted on Edited on Originally published at nodeneuralai.hashnode.dev AI-assisted

Semantic Cache on AWS: Cut LLM Costs Without Serving Wrong Answers

Your semantic cache just did its job perfectly: a near-identical question, no model call, 23 ms, zero cost. It also told a customer that order 1043 was cancelled. It wasn't. That answer was for order 1042. This is the part most semantic cache tutorials skip.

This article builds a semantic cache on AWS that hits often and never hits wrong, including the five lines of code that stop this exact bug.

Exact match: same intent, new key

Exact match: same intent, new key

First, why semantic at all? Most teams start with an exact-match cache: hash the prompt, look up the hash. "How do I reset my VPN?", "VPN password reset" and "I can't log into the VPN" produce three hashes, three misses and three model calls. People don't repeat text. They repeat intent. So you compare meaning instead.

Turn the question into a vector, find the closest one already answered, and return that answer. Simple, except that "closest" is a trap. Try the four pairs below before reading on.

Same answer?

For each pair, should the second question reuse the cached answer from the first?

Pair Should it reuse? Cosine distance
"reset my VPN?" ↔ "VPN password reset" ✓ yes, same intent 0.06
"cancel order 1042" ↔ "cancel order 1043" ✗ different order 0.02
"what's our refund window?" from Acme ↔ from Globex ✗ different contract 0.00
"export a report" ↔ "delete a report" ✗ opposite action 0.14

Pair one is the hit you want: different words, same intent. Pair two is a different order, yet its distance (0.02) is smaller than pair one's.

Pair three is the exact same question, at distance zero, from two customers with two different contracts. Reusing it isn't a saving; it's a data leak. Pair four, export versus delete, is one word apart with the opposite meaning. The threshold in AWS's own example would serve all four, and three of them would be wrong.

The code below fixes each of these pairs.

Semantic cache on AWS

Semantic cache on AWS

The blueprint: API Gateway handles authentication and per-tenant throttling. A Lambda function runs the cache logic, in the same VPC as an ElastiCache for Valkey cluster. Valkey 8.2 added vector search, so vectors, answers and expiry live in one in-memory store.

For embeddings, Titan Text Embeddings V2 on Bedrock, at 512 dimensions instead of 1024. That is half the memory, and AWS reports about 99% of the retrieval accuracy. Only a miss goes to the LLM, and its answer is written back with a time-to-live.

Step by step

Follow two requests. Someone at Acme asks how to reset their VPN. The embedding takes about 40 ms; the search takes 1 ms and finds nothing yet, so the model answers in almost 3 seconds and the answer is stored for 24 hours. Ten minutes later, a colleague types "VPN password reset". This time the search returns a distance of 0.06: a hit. The request completes in 40 ms, and the model never runs.

→ POST /ask  tenant=acme  "how do I reset my VPN?"
  embed  titan-v2 · 512d ..........    41 ms
  FT.SEARCH  scope=acme ...........     1 ms   no match
  bedrock.invoke ..................  2,870 ms
  HSET + EXPIRE 24h ...............     1 ms   MISS
→ POST /ask  tenant=acme  "VPN password reset"
  embed  titan-v2 · 512d ..........    38 ms
  FT.SEARCH  scope=acme ...........     1 ms   d=0.06  HIT
2,912 ms → 40 ms   ·   $0.021 → $0.000
Enter fullscreen mode Exit fullscreen mode

Index and scope

Now the code, starting with pair three: the leak. The index is created once, with a scope tag and the embedding (HNSW, cosine distance). A cached answer is only valid for the same tenant, the same model, the same system prompt, and the same version of the knowledge behind it. All four go into one tag.

Acme's answers can't even be seen from Globex. And when you change the prompt or re-index the documents, old answers simply stop matching, with no mass delete.

// cache/index.ts
// Once, at deploy time
await valkey.customCommand([
  'FT.CREATE', 'answers', 'ON', 'HASH', 'PREFIX', '1', 'ans:',
  'SCHEMA',
  'scope', 'TAG',
  'embedding', 'VECTOR', 'HNSW', '6',
  'TYPE', 'FLOAT32', 'DIM', '512', 'DISTANCE_METRIC', 'COSINE',
]);

// An answer is only reusable inside the same scope
export const scopeOf = (req: CacheRequest) =>
  sha256([
    req.tenantId,
    MODEL_ID,
    SYSTEM_PROMPT_VERSION,
    req.knowledgeVersion,
  ].join('|'));
Enter fullscreen mode Exit fullscreen mode

The lookup in Lambda

The client is created outside the handler, so warm invocations reuse the connection.

The handler starts with a guard: anything that depends on live or personal data, like "where's my order?", skips the cache in both directions. Then comes a single search. The scope filter goes before the arrow, followed by the single nearest neighbour and its distance.

This is where pair four comes in. AWS's example accepts a similarity of 0.8, which is a distance of 0.2, and that serves all four pairs. Start at 0.1 instead: export versus delete, at 0.14, becomes a miss. But pair two, at 0.02, still passes. No threshold can catch that one.

// cache/handler.ts
const valkey = await GlideClusterClient.createClient(config); // reused while warm

export const handler = async (event: APIGatewayProxyEvent) => {
  const req = parseRequest(event);
  if (!cacheable(req)) return reply(await askModel(req), 'bypass');

  const scope = scopeOf(req);
  const vector = await embed(req.question); // Titan V2, 512 dims

  const hit = firstHit(await valkey.customCommand([
    'FT.SEARCH', 'answers',
    `@scope:{${scope}}=>[KNN 1 @embedding $v AS distance]`,
    'PARAMS', '2', 'v', toBytes(vector),
    'DIALECT', '2',
  ]));

  if (hit && hit.distance < 0.1 && sameEntities(req.question, hit.question)) {
    return reply(hit.answer, 'hit');
  }
  return answerAndStore(req, scope, vector);
};
Enter fullscreen mode Exit fullscreen mode

Store, and the entity check

On a miss, the code calls the model, stores the answer with its scope and vector, and gives it 24 hours to live, so even a bad answer has a limited life. The last five lines of the block below are the ones promised at the start: pull out anything that looks like a number or an ID, and require the same set on both sides.

Order 1042 against order 1043: different sets, no hit. The check throws away some valid hits too, and that's the right trade. A miss costs a few cents. A wrong answer costs a customer.

// cache/store.ts
async function answerAndStore(req: CacheRequest, scope: string, vector: number[]) {
  const answer = await askModel(req);
  const key = `ans:${randomUUID()}`;

  await valkey.hset(key, {
    scope,
    question: req.question,
    answer,
    embedding: toBytes(vector), // raw float32, little-endian
  });
  await valkey.expire(key, 24 * 60 * 60);

  return reply(answer, 'miss');
}

// "Cancel order 1042" must never reuse the answer for order 1043
const sameEntities = (a: string, b: string) =>
  ids(a).join() === ids(b).join();
const ids = (text: string) =>
  (text.match(/\b[\w-]*\d[\w-]*\b/g) ?? []).sort();
Enter fullscreen mode Exit fullscreen mode

Where this breaks

There are four ways this still breaks.

The threshold. A distance of 0.1 is a starting point, not an answer. Label a few hundred real question pairs from your logs, the same exercise as the four pairs above, and pick the distance where wrong hits stay near zero.

Cache poisoning. A cache multiplies whatever it stores, hallucinations included. Log the cache key with every response, and let a thumbs-down evict it.

Low repetition. Helpdesks repeat a lot. Open-ended analysis barely does. Run the cache in shadow mode for a week, looking up but not serving, and measure the hit rate before you switch it on.

Memory. Every entry lives in RAM. Size the cluster for your working set, not your full history.

What the business gets

  • 0 LLM tokens billed on a cache hit.
  • ~0.1 s to answer on a hit, instead of several seconds.
  • 0 answers shared across tenants.

The cost curve also bends the right way: the more people use the assistant, the more questions repeat.

The numbers are targets for an illustrative scenario, not measurements.


▶️ Watch the video: https://www.youtube.com/watch?v=XjfXwFTX_R8

💻 Full code: https://github.com/NodeNeuralAI/examples/tree/main/v4-semantic-cache

🔔 More enterprise AI architecture every week: https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1

Top comments (0)