Why we open-sourced cachegate, the LLM router we built for our own product

sami — Mon, 31 Aug 2026 08:11:00 +0000

cachegate started as a completely unglamorous internal fix. Our own product was burning money calling LLM APIs directly, mostly on requests that were exact or near-duplicates of something we'd already paid for and answered minutes earlier. So we built a small proxy that sat in front of Anthropic and OpenAI: cache what we can, route the rest to whichever provider was actually cheapest and healthy at that moment.

It worked well enough internally that it stopped feeling like plumbing and started feeling like a real, small project on its own - so we pulled it out, cleaned it up, and open-sourced it under MIT as cachegate.

What it is: a self-hosted, OpenAI-compatible proxy. Point an existing client's baseURL at it and nothing else about your integration changes. Behind that one endpoint:

Two kinds of cache, reported honestly. An exact-match cache (same request, same everything - guaranteed-correct hit) and a semantic cache for near-duplicates (a paraphrase, reordered context) that scores above a similarity threshold. We report cache_hit_rate.exact, .semantic, and .combined as three separate numbers in /stats, because blending them into one headline hit-rate is exactly the kind of vendor-marketing figure that doesn't survive contact with real production traffic - a thing we ran into ourselves while shopping for a caching layer before deciding to just build one.

Routing that doesn't pretend to be smarter than it is. You define capability tiers (router:fast-cheap, router:frontier, whatever fits your workload) and pick one of three plain strategies: cheapest healthy candidate, fastest healthy candidate, or cheapest-with-a-latency-guardrail. No blended score inventing a tradeoff on your behalf.

Real savings numbers instead of inflated ones. Caching alone typically saves 20-45% of spend; adding routing on top of well-tuned traffic can reach 47-90%. Some vendors will tell you 90%+ is normal. It isn't, and we didn't want to be the next tool making that claim.

What it deliberately isn't: a hosted service, a 140-provider gateway, or (yet) a vector-indexed semantic cache - the current implementation is a brute-force cosine scan over a capped Redis list per model, which is fine at single-instance self-hosted volume and not built to scale past it. All three limitations are stated plainly in the README, not buried.

Try it:
npx cachegate (zero-clone)
docker run -p 4000:4000 --env-file .env ghcr.io/idebunk/cachegate:latest
or clone it directly: https://github.com/iDebunk/cachegate

It's MIT licensed and has no relationship to any future hosted product we might build on top of it - the engine stays open regardless of what we do next. If you're fighting the same LLM-cost problem we were, we'd like to hear whether this actually helps, and where it doesn't.

DEV Community: sami

Why we open-sourced cachegate, the LLM router we built for our own product