How I Dropped OpenAI Response Latency From 1,800ms to 14ms Using Semantic Caching

If you run an AI chatbot, customer support bot, or search feature in production, you've probably noticed that 30% to 60% of user queries are repetitive:

• "What is your refund policy?"
• "How can I get my money back?"
• "Return terms please"

Traditional string hashing (like standard Redis keys) fails because users phrase sentences differently. As a result, you pay OpenAI full token fees and your users wait 2 to 3 seconds for answers that the LLM has already generated dozens of times.

To solve this, I built HyperCache-LLM — a lightweight, drop-in semantic caching proxy.

🏛️ How It Works Under the Hood:

1. Local ONNX Embeddings: It uses a local ONNX model (bge-small-en-v1.5, 384 dimensions) running directly on CPU in <10ms (Zero external API costs).
2. In-Memory Cosine Similarity: An in-memory matrix checks cosine similarity against cached query vectors.
3. Smart Interception:
   • If similarity >= threshold (default 0.88), it returns the cached response in 14ms (Cache HIT ⚡).
   • If it's a Cache MISS, it forwards the query to OpenAI and stores the new vector.

📊 Benchmark Numbers:

• Cache MISS (Cold Query): ~1,450 ms | $0.003
• Cache HIT (Semantically Rephrased): ~14 ms | $0.000 (99% latency reduction)

🔌 2-Line Integration (No Code Rewrite):

from openai import OpenAI

client = OpenAI(
    api_key="your-openai-key",
    base_url="https://hypercache-semantic-cache-for-openai-and-llms.p.rapidapi.com/v1"
)

The core proxy is completely open-source and auditable:

• ⚡ RapidAPI (500 free calls): https://rapidapi.com/erdeveshkumardev/api/hypercache-semantic-cache-for-openai-and-llms

Would love to get feedback on the latency numbers and similarity threshold handling!