I analyzed 3 months of API logs for a production app. The results
were embarrassing.
The data
| Type | % of requests | Cost share |
|---|---|---|
| Unique prompts | 28% | 34% |
| Near-duplicates | 41% | 52% |
| Exact duplicates | 31% | 14% |
31% of calls were literally identical — same model, same prompt,
same temperature. Every single one hit the provider and burned
tokens.
41% were "near-duplicates" — same system prompt, slightly different
user input. A 10-word difference on a 500-token context. Wasted.
28%. That's how many of my API calls actually needed to reach the
model.
The fix that took 5 minutes
AIBridge has response caching built in. Not a separate Redis layer.
Not a middleware I had to deploy. Just... on.
curl https://aibridge-api.com/v1/chat/completions
-H "Authorization: Bearer mb-xxx"
-H "Content-Type: application/json"
-H "X-Cache-TTL: 3600"
-d '{"model":"deepseek-chat","messages":[...]}'
Add one header. X-Cache-TTL: 3600. That's it.
Same prompt within the hour? Returned from cache. Zero tokens.
Zero latency. Zero cost.
What changed
| Before | After |
|---|---|
| 100% of requests hit provider | ~30% hit provider |
| $120/month in API costs | $38/month |
| Avg latency 1.2s | 200ms (cached hits) |
| Rate limit anxiety | Gone |
The $82/month I saved isn't life-changing. But the latency
improvement for cached responses is — users notice 200ms vs 1200ms.
Every single time.
When to use it
Not everything should be cached. Here's my rule:
- ✅ Static system prompts ("You are a helpful coding assistant")
- ✅ FAQ-style user prompts ("What's your return policy?")
- ✅ Deterministic generation (temperature=0, JSON mode)
- ❌ Creative generation (temperature > 0.5)
- ❌ Per-user personalized responses
- ❌ Real-time data queries
For most SaaS apps — chatbots, customer support, documentation
search — the cache hit rate should be above 50%.
For API wrappers and middleware that resell AI access? 70%+ is
realistic. Every cached response is pure margin.
The lesson
Before you spend another dollar on API credits, check your logs
for repeats. You're probably paying for the same answer twice.
If you're on AIBridge, add X-Cache-TTL. If you're not, build a
simple in-memory cache. Either way, stop burning tokens on answers
you already have.
→ aibridge-api.com — 14 models, caching built in, free to start.
Top comments (0)