How we cut our LLM bill 40% with multi-provider routing

Running LLM features in production is expensive in ways the pricing page doesn't show. Here's what actually moved the needle for us.

We ship LLM features to production. For the first few months, our bill looked reasonable — until it didn't. The model call price was never the problem. The problem was everything around it.

The real cost isn't the happy path

When a provider starts rate-limiting or flaking, two things happen that the naive cost estimate ignores:

1. Retry storms. The same prompt gets hammered 3–5 times before anything gives up. Each retry is a full-price call.
2. Tail latency. Requests queue behind slow providers, users time out, and you retry again.

In one bad week, our effective cost per successful request was ~2.4× the listed price. Nobody budgets for that.

What we changed

1. Route across providers by latency + price

Instead of pinning one model to one provider, we route each request to the cheapest healthy endpoint that meets our latency budget.

• A request that needs a fast answer goes to the lowest-latency provider currently under quota.
• A batch job goes to the cheapest one.
• Same model family, different provider — the caller doesn't care.

This alone removed most of the "stuck behind a throttled provider" tax.

2. Cache embeddings and deterministic steps

A surprising amount of our traffic re-embeds the same inputs. We added a cache layer:

• Embeddings for repeated prompts → served from cache.
• Deterministic pre/post-processing → computed once, reused.

That cut a meaningful chunk of repeat calls without touching quality.

3. Health-checked failover

We keep a health-checked pool of upstreams. A provider that returns N consecutive 5xx/429 gets pulled from rotation for ~60s. Retries use exponential backoff with jitter.

Result: when one provider degrades, traffic moves before users notice. No dropped requests as long as one upstream is alive.

The numbers

• Bill per successful request: down ~40%.
• Retry calls: down ~60%.
• 2am incidents where we manually switched providers: roughly zero.

None of this required an exotic stack. It's routing, caching, and a health check — the boring infra stuff that nobody writes blog posts about until the bill arrives.

If you're in the same boat

Getting reliable, affordable model access set up for a team has its own headaches — provider quotas, region limits, payment friction. If any of that sounds familiar, I'm happy to compare notes. Find me here or DM me; no pitch, just war stories.