TL;DR: If you are making repeated or similar LLM calls, you are burning money on tokens you have already paid for. Semantic caching catches not just exact duplicates but semantically similar requests and serves cached responses in sub-millisecond time. Bifrost supports both Redis and Weaviate backends with streaming response caching built in.
You Are Probably Paying for the Same Answers Twice
Here is something most teams do not think about until the bill arrives. A large chunk of your LLM calls are either identical or very similar to calls you have already made.
A customer asks "how do I reset my password?" and another asks "how can I change my password?" Different words, same intent, same answer. Without caching, you are paying full token cost for both.
Semantic caching fixes this. And when it is done right, it can meaningfully cut your LLM spend.
How Semantic Caching Works in Bifrost
Bifrost uses a dual-layer caching approach:
- Exact hash matching - If the request is identical to a previous one, it returns the cached response immediately. Zero LLM cost.
- Semantic similarity matching - If the request is similar (but not identical) to a cached request, it uses vector similarity to find the best match. The only cost here is the embedding generation for the similarity check.
For cache misses, you pay the normal model cost plus a small embedding cost for storing the response in the cache for future use.
The Cost Breakdown
| Scenario | What You Pay |
|---|---|
| Cache hit (exact) | Zero |
| Cache hit (semantic) | Embedding cost only |
| Cache miss | Model cost + embedding cost for cache storage |
The cache-aware cost calculation in Bifrost handles this automatically. You do not need to track cache hits and misses manually. The cost dashboard reflects actual costs after caching.
Backend Options
Bifrost supports multiple vector store backends for semantic caching:
Redis (with RediSearch)
If you already run Redis in your infrastructure, this is the easiest path. You need the RediSearch module enabled for vector similarity search. Redis gives you:
- Sub-millisecond cache retrieval (in-memory storage)
- HNSW algorithm for similarity search
- Connection pooling with advanced connection management
- TTL support for automatic cache expiration
Weaviate
If you want a dedicated vector database, Weaviate is supported. It is purpose-built for vector similarity search and works well for larger cache stores.
Custom Implementations
If you are running a different vector store, Bifrost supports custom backend implementations. You can plug in your own storage layer.
Streaming Response Caching
This is the feature that most caching solutions get wrong. When your LLM response is streamed (which it should be for any user-facing application), caching becomes tricky. You need to cache the complete response while maintaining proper chunk ordering.
Bifrost handles streaming cache correctly. Streamed responses are cached with proper chunk ordering, so when a cached response is served, it streams back in the same way the original response did.
Setting It Up
If you are using Bifrost, semantic caching is a configuration option. You pick your vector store backend, configure the connection, and caching starts working across all your providers.
Bifrost supports 19+ providers (OpenAI, Anthropic, Azure, Bedrock, Gemini, Mistral, Cohere, Groq, and others). Caching works uniformly across all of them. A cached OpenAI response can serve a semantically similar request that would have gone to Anthropic.
The gateway itself adds just 11 microseconds of latency overhead and handles 5,000 RPS sustained throughput. Written in Go, so the caching layer does not become a bottleneck.
TTL and Cache Management
You probably do not want cached responses living forever. TTL (Time-To-Live) support lets you set automatic expiration for cached entries.
For fast-changing data, set short TTLs. For stable responses (like documentation queries or FAQ-type questions), longer TTLs make sense. The right TTL depends on your use case, but having automatic expiration prevents stale responses.
When Semantic Caching Makes Sense
Semantic caching is not for every use case. It works best when:
- Customer support bots - High volume of similar questions
- Documentation assistants - Same docs, similar questions
- Code generation - Common patterns get repeated across developers
- Internal tools - Teams asking similar questions about the same codebase
It is less useful for highly unique, one-off requests where semantic similarity is unlikely.
Getting Started
Bifrost is open source. You can deploy with npx or Docker, configure your vector store, and start caching immediately. The docs walk through each backend option.
If you are already running Redis with RediSearch, you can have semantic caching working in under 10 minutes.
GitHub: git.new/bifrost
Docs: getmax.im/bifrostdocs
Website: getmax.im/bifrost-home
Top comments (0)