Everyone tells you to just drop a semantic cache in front of OpenAI or Anthropic to cut costs. I tried it. tbh, it's mostly an illusion.
Here is the reality of AI agents: they don't ask the same questions twice. They inject dynamic context, timestamps, user-specific IDs, and multi-step reasoning traces into every single prompt.
I looked at 50,000 requests from a basic RAG setup. The cache hit rate? 4.2%.
A 4% cache hit rate doesn't save a $5k/mo bill. It just adds Redis infrastructure costs and a layer of latency you didn't need. The real problem isn't caching. The real problem is you don't know which user or which agent loop is burning the other 96% of the budget.
If you have a multi-tenant app, your biggest risk isn't missing a cache hit. It's one power user triggering a runaway agent loop that eats your entire API quota while everyone else gets rate-limited.
You need per-user cost attribution, not just a cache.
I built LLMeter (https://llmeter.org?utm_source=devto&utm_medium=article&utm_campaign=devto-caching-illusion-llmeter) exactly for this. It's an open-source (AGPL-3.0) dashboard that tracks costs per model, per user, per day. It supports OpenAI, Anthropic, DeepSeek, and OpenRouter.
Stop trying to cache dynamic agent loops. Start tracking who is actually spending your money.
Top comments (0)