Every team eventually ends up with three API keys, four SDKs, and no consistent way to answer "what did we spend on this feature last month?"
A gateway fixes the accounting. It does not automatically fix the architecture. Here is what actually matters.
What a gateway buys you
One schema. Providers disagree on field names, streaming formats, and error shapes. Normalizing that in one place beats normalizing it in five services.
One bill. Per-token attribution by feature, team, or model. This is the underrated win — you cannot optimize what you cannot see.
One failover path. When a provider degrades, rerouting should be a config change, not a deploy.
What it costs you
A gateway is another hop. That means another place for latency and another component that can be down. If your gateway is a single container on a single host, you have built a SPOF in exchange for convenience.
The mitigations that matter: weight-based routing rather than strict primary/backup (a degraded primary should shed load gradually), and health checks that actually exercise the model endpoint rather than just the HTTP port.
The measurement trap
Aggregate spend looks informative and is not. What you want is per-model, per-feature token counts, split by input/output/cache. Without that split, "costs went up" is unactionable.
I wrote up the full set of trade-offs, including the weighting formula and why equal-priority retries behave the way they do, at multi-model API gateway and the LLM API pricing guide.
The lesson: adopt a gateway for the accounting and the schema normalization. Do not adopt it thinking it makes you highly available for free.
Top comments (0)