DEV Community

Srijan Verma
Srijan Verma

Posted on

Stop Wasting Tokens: Building a Sub-50ms Semantic Cache with Redis

#ai #backend #softwareengineering #systemdesign

Slash LLM latency by 90% and cut API bills by caching query intent instead of raw strings.

The Bottleneck in Production

Direct LLM calls are the slowest, most expensive component in modern backend stacks. A typical round-trip to an LLM provider takes between 2,000ms and 4,000ms and costs fractions of a cent per request. At scale, this destroys both user experience and infrastructure budgets.

Traditional caching fails completely for conversational AI and search interfaces. If you hash the raw query string into a standard key-value store, your cache hit rate rarely exceeds 15–20%.

# The Naive Approach: Breaks on minor variations
cache_key = hashlib.sha256(user_query.strip().lower().encode()).hexdigest()
cached_response = redis_client.get(cache_key) # Fails on synonyms or typos
Enter fullscreen mode Exit fullscreen mode

A user asking "How do I reset my password?" and another asking "Forgot my credentials, need a reset link" represent identical user intent. Yet, exact-match string hashing treats them as two distinct misses, triggering two redundant, expensive model evaluations.

The System Architecture & Fix

The solution is Semantic Caching: caching queries based on vector embeddings rather than raw text strings.

Instead of an exact key lookup, we convert incoming queries into dense vector representations and run an approximate nearest neighbor (ANN) search in Redis using cosine distance. If the distance to a cached query is within our confidence threshold (e.g., similarity $\ge$ 0.90), we serve the cached answer immediately.

[ User Query ]
      │
      ▼
[ Embedding Model ] (e.g., bge-small-en-v1.5, ~10ms)
      │
      ▼
[ Redis Vector Search ] ── (Cosine Sim ≥ 0.90?)
      │                               │
    [ YES ]                         [ NO ]
      │                               │
      ▼                               ▼
[ Return Cached ]             [ Call LLM API ] (~3000ms)
    (~50ms)                           │
                                      ▼
                              [ Save Vector + Text ]
Enter fullscreen mode Exit fullscreen mode

This cuts downstream latency from ~3,000ms to under 50ms for cache hits while protecting your upstream rate limits.

The Implementation

Below is a production-ready pattern using a fast local embedding model (BAAI/bge-small-en-v1.5) and Redis Vector Search.

import numpy as np
import redis
from redis.commands.search.query import Query
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("BAAI/bge-small-en-v1.5")
r = redis.Redis(host="localhost", port=6379, decode_responses=False)

def get_semantic_answer(query: str, threshold: float = 0.90) -> str:
    query_vector = encoder.encode(query).astype(np.float32).tobytes()

    # Search for nearest vector within index
    q = Query("(*)=>[KNN 1 @vector $vec AS score]").return_fields("response", "score").dialect(2)
    results = r.ft("idx:cache").search(q, query_params={"vec": query_vector})

    if results.docs and (1 - float(results.docs[0].score)) >= threshold:
        return results.docs[0].response.decode("utf-8")

    # Cache Miss: Fetch from LLM and store vector
    llm_response = call_llm(query)  # Replace with actual LLM provider call
    doc_id = f"cache:{hash(query)}"
    r.hset(doc_id, mapping={"vector": query_vector, "response": llm_response})

    return llm_response
Enter fullscreen mode Exit fullscreen mode

Why This Works in Production:

  • Low Overhead: Running a lightweight embedding model locally takes ~10ms on CPU, keeping total cache-hit round trips under 50ms.
  • Fail-Open Design: If vector search fails or encounters network degradation, the pipeline gracefully falls back to the direct LLM call without crashing the request cycle.

Production Lessons & Takeaways

  • Tune Your Similarity Threshold Conservatively: Start with a strict cosine similarity threshold between 0.88 and 0.92. Anything lower than 0.85 introduces false positives where slightly different questions get wrong answers.
  • Co-locate Embeddings with Workers: Run your embedding model directly on the application runtime container or within the same VPC subnet as Redis. Avoid using external embedding APIs for the cache lookup, as network latency defeats the purpose of the cache.
  • Enforce TTLs on Dynamic Knowledge: Set explicit Time-To-Live (TTL) policies on cached vector keys. LLM answers that depend on changing backend data must expire automatically to avoid serving stale hallucinations.

Top comments (0)