Disclosure: this is my own project (withOhm). Sharing it because the problem is generic enough that if you're building anything agentic or RAG-adjacent, you've probably hit some version of it already.
The problem
Two things kept happening in every agent/RAG project I built:
The same prompt gets sent to the model more than once — retries, self-consistency passes, research loops that re-check the same question — and every single one gets billed at full price, even though the response is byte-for-byte identical to one already paid for.
Anything that fetches web content for context does it raw: no robots.txt check, no PII scrubbing, no protection against the fetch target resolving to an internal IP. Fine in a demo, a liability in production.
Neither problem is hard to solve badly. What's annoying is solving them correctly — structurally exact, not "good enough for a blog post" — while keeping streamed and non-streamed calls indistinguishable to the caller either way.
What I built
withOhm sits in front of the model calls as an OpenAI-compatible endpoint. Point your existing SDK at it, add one header, done.
Two decisions shaped everything:
Exact-match caching, never semantic. Requests are canonicalized — transport noise stripped (line endings, outer whitespace), nothing inside a code block touched — then hashed. Identical hash → replay from Redis. "Almost the same prompt" doesn't count, on purpose. Semantic caching reads great in a README and turns into a refunds feature the first time it serves a near-miss as if it were the real answer.
Streaming replays as streaming. If the original call was SSE, the cached replay is reassembled and re-emitted as SSE too — the calling code can't tell a cache hit from a live call just by watching how the tokens arrive.
For the fetch side, there's a second tool that runs any public URL through a compliance pass before the content reaches the model: robots.txt is consulted at fetch time, obvious PII gets redacted before it hits context, and the fetch itself is SSRF-guarded (no sneaking a request to an internal IP through a redirect). It returns a verdict alongside the content — what got redacted, what robots said — so the caller can reason about what it received instead of trusting it blindly.
Trying it
If you're in Cursor, it's an MCP server:
pip install withohm-mcp
That gives you ohm_chat (cache-replayed chat) and ohm_fetch_web (compliant fetch) as tools, plus a couple of introspection tools (ohm_usage, ohm_savings) so you can see what's actually getting deduped instead of taking it on faith.
Outside Cursor it's just an HTTP endpoint — point any OpenAI-compatible client at it:
curl -s https://api.withohm.dev/v1/chat/completions \
-H "Authorization: Bearer $OHM_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "hi"}]
}'
Send that exact request again and the second response comes back from cache instead of the provider. It's BYOK — you pass your own OpenAI/Anthropic key via a header (X-Ohm-Upstream-Key); withOhm forwards it for that one call and doesn't persist it.
Where I'd like pushback
Has anyone here made semantic caching work in production without a correctness incident? I ruled it out early and would genuinely like to see a counterexample that's survived real traffic — right now I think "almost the same prompt" is a trap dressed up as an optimization.
More at withohm.dev — the quickstart is the fastest way to see the cache-miss/cache-hit difference for yourself.
Top comments (0)