I built an API gateway that routes between OpenAI, Anthropic and Gemini. I figured integrating both providers would be the hard part.
It wasn't. C...
For further actions, you may consider blocking this person and/or reporting abuse
This matches our experience almost exactly — the integration is an afternoon, the failure semantics are the actual project. The
rate_limitedvsoverloadedsplit you drew is the one that pays for itself, because they demand opposite responses:rate_limitedmeans back off on this same provider (jittered exponential), whileoverloaded/quota_exceededmeans don't wait, fail over to a different provider right now. Collapse those into one bucket and you either hammer a provider that's telling you to slow down, or you sit waiting on capacity that isn't coming back.Two things I'd fold into the model when you're ready: honor
Retry-Afterheaders where providers send them (they often carry better information than the status code itself), and add acontent_filterclass — a moderation refusal is not aserver_errorand retrying it just burns tokens. The gnarliest one for us was streaming: a 200 that dies mid-stream. The request "succeeded," but the terminal frame never landed, so it needs its own classification path. Have you hit that one yet with the gateway?Great points, especially the distinction between Retry-After and the status code itself.
Right now I'm honoring Retry-After when it's available, but content_filter is a really good suggestion. I hadn't separated moderation failures into their own class yet, and I can already see how retrying them would just waste credits.
Streaming isn't supported in Apiarium yet, so fortunately I haven't had to deal with interrupted streams... but I suspect that's going to become one of the trickiest cases once I add it.
The 429 ambiguity is the part that bites hardest, since "slow down" and "we are out of capacity" need opposite responses and the status code flattens both into one number. I keep a retryable flag on top of your error_class set, because overloaded and rate_limited both back off but quota_exceeded and invalid_request should fail fast and never re-enter the queue. How are you deciding retry timing once the category is known, fixed backoff per class or something adaptive off the provider headers?
At the moment I keep it intentionally simple.
The classification exists so callers can make consistent decisions regardless of provider. Apiarium itself isn't retrying requests automatically yet.
Where providers expose useful headers like Retry-After, I think those should always take precedence over any generic backoff strategy. If I eventually add automatic retries or failover, that will probably become part of the routing logic instead of being hardcoded per provider.
This resonates a lot. Status codes are a leaky abstraction once you're dealing with multiple providers — a 429 from one API might mean "back off for a second," while from another it means "you're permanently rate-limited on this key." A 500 can mean "retry me" or "this request will never work, stop trying." Treating the code as ground truth instead of a hint is where most brittle retry logic comes from.
Building an actual error model — classifying by retryability, whether it's transient vs. structural, whether the failure is on the input (bad prompt, context too long) vs. the provider (overload, timeout, moderation block) — is the right instinct. It's basically the same lesson distributed systems folks learned with HTTP a decade ago: you can't treat the transport-layer status as the whole story, you need a domain-specific taxonomy on top of it.
One thing I'd be curious about: how do you handle providers that don't document their error semantics well and just change behavior silently (rate limit thresholds, timeout windows, etc.)? That feels like the hardest part to keep a model like this accurate over time — did you build in any drift detection, or is it mostly manual maintenance as providers change?
That's something I've been thinking about as well.
Right now it's mostly manual because provider error semantics don't change very often, but I definitely don't want the classification logic to become a pile of provider-specific exceptions over time.
One idea I'm exploring is making the routing layer more data-driven, so changes in provider behavior can be updated without touching the rest of the codebase. Still early, but I think that's a cleaner direction than hardcoding every edge case.
One thing worth thinking about early: the classifier is now its own failure surface, and it fails silently. If a provider ships a new error type tomorrow, or tweaks a message string you were matching on,
classifyProviderErrordoesn't throw. It just quietly buckets the thing intoserver_errorand your caller happily retries a request that will never work. Worth having an explicitunknownclass that never gets a retry, plus a log line loud enough that you find out a provider changed shape from your own alerts and not from a support ticket. Everyone in this thread is arguing about which classes belong in the set, which is the fun part, but the boring part is what the thing does when it doesn't recognize what it's looking at.The place this model earns its keep — and where it quietly breaks — is that
error_classisn't stable per status code even within one provider. A 429 is your classic example: same code, sometimesrate_limited, sometimesoverloaded. So whateverclassifyProviderErrordoes, it can't be a status-code lookup table; it has to read the body'stype/codeand, for the ambiguous cases, sometimes theRetry-Afterheader or error message text. That's the actual work, and it's the part that rots — providers change errortypestrings without changing the status code, and your classifier silently starts dumping things intoserver_error.Which is the caveat I'd flag: put a metric on the "fell through to
server_error" branch. That default bucket is where misclassifications go to hide, and since you're logging the raw error but only returning the class, a provider renamingoverloaded_errortooverloadedwon't page anyone — it'll just quietly turn a retryable failure into one your retry logic treats as fatal. The normalization is only as good as your visibility into what it failed to normalize.Gunna be a really useful feature when you start dealing with multi-account load-balancing. For the V.E.L.O.C.I.T.Y. IDE (side-branch of V.E.L.O.C.I.T.Y. OS), I've got an accounts table with 17 free accounts that load-balance and allows for higher concurrency for agents. What's important about it though is the distinction between out of quota and server out of supply. Both cases you need to switch gears, but 1 means swap to a different account, while the other means you switch models. The AI Gateway is taking solid shape now, ngl, it might become an industry standard tool if you keep going at this pace.
Curious about your setup, 17 accounts for load balancing sounds like exactly the kind of thing I built Apiarium to avoid managing by hand.
What's the part of that system that's been the most annoying to maintain?
Honestly, I just set it up with a simple failover, if a sequence is partially completed, it just fails over to the next one and passes the context, it just continues seamlessly sofar with Kimi K2.7. I guess the tedious part was mostly just getting the 17 API keys, but since they got added to the list, not a single missed beat.
Fair, though I'd expect that to hold until a provider quietly renames an error type on you. The failover logic sounds solid either way.
That's the nice part of cloudflare, they cant change anything, because if they do, half the internet needs to be compliant post-change. The risk outweighs the gain.
This is the part model gateways need more of. Status codes are too thin for operational routing. A useful error model should know whether the failure is retryable, provider-specific, quota-related, safety-policy related, malformed input, or output-shape drift. Otherwise failover just moves the confusion to another model.
This is the right abstraction layer to add before the system gets painful.
Status codes are useful transport hints, but they are not a product-level failure model. Once multiple providers are involved, callers need to know what kind of decision to make, not which vendor-shaped JSON blob happened to arrive.
The same pattern shows up in agent tools too. A raw error like “timeout”, “permission denied”, or “invalid request” is rarely enough. The runtime needs a small set of durable categories:
Then routing, retries, UX copy, alerts, and analytics can all depend on the category instead of reverse-engineering provider behavior every time.
Also +1 on keeping the raw provider error for internal logs. Normalizing for callers should not mean throwing away evidence for debugging.
The error taxonomy you landed on maps onto a pattern I keep seeing in a different context — agent quality gates. Same fundamental problem: you need a small set of durable categories that let downstream systems make consistent decisions, regardless of which upstream layer produced the failure.
Evans touched on the drift question and I think it's the hard one. Providers will silently change behavior, and your classification will start mislabeling. The same thing happens when you use an LLM as a quality inspector — its calibration drifts as the executing model gets updated. One approach that seems promising: keep a thin "calibration layer" between provider-specific behavior and your classification categories, updated via observed patterns rather than manual code changes. Basically: let the data tell you when a new error shape showed up, then decide which bucket it belongs to.
The streaming-mid-death case Max mentioned is going to be interesting. A 200 that never delivers is a fundamentally different failure class from a 500 that never started.
An error model is the right abstraction because provider failures are rarely interchangeable. A timeout, safety refusal, malformed JSON, quota issue, and degraded latency tail should not all become “retry.” Once errors are typed, routing and fallback logic can be much less superstitious.
the 429 taxonomy problem is exactly right. it gets worse on streaming responses though. a 429 mid stream can arrive AFTER the initial 200 header, so any code that just checks
res.statusis already on the wrong path. we hit this on OpenAI where a stream starts fine, a few tokens come through, then closes with anerrorobject in the SSE payload that technically comes back as a 200.built almost the same error model — we call it a failure intent classifier. the thing that broke us was Anthropic's
overloaded_errornot mapping cleanly onto any retry safe category; it varies by time of day.are you normalizing stream level errors too, or just the initial connection errors?
An error model is a much better abstraction than provider-specific status handling. AI APIs fail in ways that affect product behavior: partial output, policy refusal, timeout, malformed JSON, stale tool context, or silent quality drop. Classifying those failures makes fallback logic much less random.
Ran into this with multi-model tools too. The API call is easy, but keeping failures consistent is the real work. A shared error layer saves a lot of messy checks later.