How I Dropped OpenAI Response Latency From 1,800ms to 14ms Using Semantic Caching

erdeveshkumardev — Mon, 09 Nov 2026 12:18:43 +0000

If you run an AI chatbot, customer support bot, or search feature in production, you've probably noticed that 30% to 60% of user queries are repetitive:

"What is your refund policy?"
"How can I get my money back?"
"Return terms please"

Traditional string hashing (like standard Redis keys) fails because users phrase sentences differently. As a result, you pay OpenAI full token fees and your users wait 2 to 3 seconds for answers that the LLM has already generated dozens of times.

To solve this, I built HyperCache-LLM — a lightweight, drop-in semantic caching proxy.

🏛️ How It Works Under the Hood:

Local ONNX Embeddings: It uses a local ONNX model (bge-small-en-v1.5, 384 dimensions) running directly on CPU in <10ms (Zero external API costs).
In-Memory Cosine Similarity: An in-memory matrix checks cosine similarity against cached query vectors.
Smart Interception:
- If similarity >= threshold (default 0.88), it returns the cached response in 14ms (Cache HIT ⚡).
- If it's a Cache MISS, it forwards the query to OpenAI and stores the new vector.

📊 Benchmark Numbers:

Cache MISS (Cold Query): ~1,450 ms | $0.003
Cache HIT (Semantically Rephrased): ~14 ms | $0.000 (99% latency reduction)

🔌 2-Line Integration (No Code Rewrite):

from openai import OpenAI

client = OpenAI(
    api_key="your-openai-key",
    base_url="https://hypercache-semantic-cache-for-openai-and-llms.p.rapidapi.com/v1"
)

The core proxy is completely open-source and auditable:

⚡ RapidAPI (500 free calls): https://rapidapi.com/erdeveshkumardev/api/hypercache-semantic-cache-for-openai-and-llms

Would love to get feedback on the latency numbers and similarity threshold handling!

DEV Community: erdeveshkumardev

How I Dropped OpenAI Response Latency From 1,800ms to 14ms Using Semantic Caching

🏛️ How It Works Under the Hood:

📊 Benchmark Numbers:

🔌 2-Line Integration (No Code Rewrite):