Your CI suite is probably re-paying OpenAI for prompts it already sent

iwasinnam2 — Thu, 13 Aug 2026 15:12:35 +0000

Disclosure: this is my own project (withOhm). Sharing it because the problem is generic enough that if you're building anything agentic or RAG-adjacent, you've probably hit some version of it already.

The problem
Two things kept happening in every agent/RAG project I built:

The same prompt gets sent to the model more than once — retries, self-consistency passes, research loops that re-check the same question — and every single one gets billed at full price, even though the response is byte-for-byte identical to one already paid for.
Anything that fetches web content for context does it raw: no robots.txt check, no PII scrubbing, no protection against the fetch target resolving to an internal IP. Fine in a demo, a liability in production.
Neither problem is hard to solve badly. What's annoying is solving them correctly — structurally exact, not "good enough for a blog post" — while keeping streamed and non-streamed calls indistinguishable to the caller either way.

What I built
withOhm sits in front of the model calls as an OpenAI-compatible endpoint. Point your existing SDK at it, add one header, done.

Two decisions shaped everything:

Exact-match caching, never semantic. Requests are canonicalized — transport noise stripped (line endings, outer whitespace), nothing inside a code block touched — then hashed. Identical hash → replay from Redis. "Almost the same prompt" doesn't count, on purpose. Semantic caching reads great in a README and turns into a refunds feature the first time it serves a near-miss as if it were the real answer.
Streaming replays as streaming. If the original call was SSE, the cached replay is reassembled and re-emitted as SSE too — the calling code can't tell a cache hit from a live call just by watching how the tokens arrive.
For the fetch side, there's a second tool that runs any public URL through a compliance pass before the content reaches the model: robots.txt is consulted at fetch time, obvious PII gets redacted before it hits context, and the fetch itself is SSRF-guarded (no sneaking a request to an internal IP through a redirect). It returns a verdict alongside the content — what got redacted, what robots said — so the caller can reason about what it received instead of trusting it blindly.

Trying it
If you're in Cursor, it's an MCP server:

pip install withohm-mcp

That gives you ohm_chat (cache-replayed chat) and ohm_fetch_web (compliant fetch) as tools, plus a couple of introspection tools (ohm_usage, ohm_savings) so you can see what's actually getting deduped instead of taking it on faith.

Outside Cursor it's just an HTTP endpoint — point any OpenAI-compatible client at it:

curl -s https://api.withohm.dev/v1/chat/completions \
  -H "Authorization: Bearer $OHM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "hi"}]
  }'

Send that exact request again and the second response comes back from cache instead of the provider. It's BYOK — you pass your own OpenAI/Anthropic key via a header (X-Ohm-Upstream-Key); withOhm forwards it for that one call and doesn't persist it.

Where I'd like pushback
Has anyone here made semantic caching work in production without a correctness incident? I ruled it out early and would genuinely like to see a counterexample that's survived real traffic — right now I think "almost the same prompt" is a trap dressed up as an optimization.

More at withohm.dev — the quickstart is the fastest way to see the cache-miss/cache-hit difference for yourself.

DEV Community: iwasinnam2

Your CI suite is probably re-paying OpenAI for prompts it already sent