Every AI provider fails in its own way. I stopped checking status codes and built an error model instead.

I built an API gateway that routes between OpenAI, Anthropic and Gemini. I figured integrating both providers would be the hard part.

It wasn't. Calling their APIs is maybe an afternoon of work each. The hard part showed up later, the first time something went wrong.

The moment it broke

Early on, my error handling was basically: catch whatever the provider throws, forward the status code, move on.

} catch (error) {
  res.status(error.status || 500).json({ error: error.message })
}

This worked fine until I actually looked at what each provider sends back when something goes wrong.

OpenAI wraps its errors in an object with a type and sometimes a code. Anthropic wraps its errors differently, with its own type field that means something else entirely. A 429 from one provider might mean "you're sending too fast, back off." A 429 from another context might mean something closer to "we're out of capacity right now, this isn't really about your rate at all."

If you're just forwarding error.status and error.message straight through, none of that nuance survives. Your own error handling ends up being provider-specific whether you meant it to be or not, because the shape of the failure is different depending on who you called.

What I built instead

Instead of trusting each provider's raw error shape, every call now normalizes into the same internal error model before it reaches the response:

} catch (error) {
  const classified = classifyProviderError(error)

  res.status(classified.httpStatus).json({
    error: 'AI provider error. Please try again.',
    error_class: classified.error_class,
    provider: classified.provider
  })
}

error_class is one of a small fixed set: rate_limited, overloaded, quota_exceeded, invalid_request, authentication_error, server_error. That's true regardless of which provider actually failed. The raw provider error still gets logged for me to debug, but what the caller sees is the category of failure, not the provider's specific wire format.

The point isn't that this retries anything automatically. It doesn't. It's that "should I retry, and how" becomes a decision you can make once, based on error_class, instead of once per provider you happen to be calling that day.

The part that surprised me

I expected the differences between providers to be about capability, better reasoning, different context windows, that kind of thing. What actually ate my time was smaller and dumber: two APIs disagreeing about what a 429 even means.

Once you stop trusting the HTTP status code to tell you the whole story and start asking "what actually happened here, independent of who I called," a lot of the provider-specific special-casing just goes away.

If you're building anything that calls more than one LLM provider, worth checking early: are you branching your error handling on the provider, or on the actual failure type? Those aren't the same question, and the difference only shows up once something breaks in production.

This eventually became part of Apiarium, the AI gateway I'm building.

Not because I wanted another abstraction layer.

Because I got tired of writing provider-specific error handling.

Comments

[Max Quimby]

This matches our experience almost exactly — the integration is an afternoon, the failure semantics are the actual project. The rate_limited vs overloaded split you drew is the one that pays for itself, because they demand opposite responses: rate_limited means back off on this same provider (jittered exponential), while overloaded / quota_exceeded means don't wait, fail over to a different provider right now. Collapse those into one bucket and you either hammer a provider that's telling you to slow down, or you sit waiting on capacity that isn't coming back.

Two things I'd fold into the model when you're ready: honor Retry-After headers where providers send them (they often carry better information than the status code itself), and add a content_filter class — a moderation refusal is not a server_error and retrying it just burns tokens. The gnarliest one for us was streaming: a 200 that dies mid-stream. The request "succeeded," but the terminal frame never landed, so it needs its own classification path. Have you hit that one yet with the gateway?

[Lolo]

Great points, especially the distinction between Retry-After and the status code itself.

Right now I'm honoring Retry-After when it's available, but content_filter is a really good suggestion. I hadn't separated moderation failures into their own class yet, and I can already see how retrying them would just waste credits.

Streaming isn't supported in Apiarium yet, so fortunately I haven't had to deal with interrupted streams... but I suspect that's going to become one of the trickiest cases once I add it.

[Kartik N V J K]

The 429 ambiguity is the part that bites hardest, since "slow down" and "we are out of capacity" need opposite responses and the status code flattens both into one number. I keep a retryable flag on top of your error_class set, because overloaded and rate_limited both back off but quota_exceeded and invalid_request should fail fast and never re-enter the queue. How are you deciding retry timing once the category is known, fixed backoff per class or something adaptive off the provider headers?

[Lolo]

At the moment I keep it intentionally simple.

The classification exists so callers can make consistent decisions regardless of provider. Apiarium itself isn't retrying requests automatically yet.

Where providers expose useful headers like Retry-After, I think those should always take precedence over any generic backoff strategy. If I eventually add automatic retries or failover, that will probably become part of the routing logic instead of being hardcoded per provider.

[Evans Owusu]

This resonates a lot. Status codes are a leaky abstraction once you're dealing with multiple providers — a 429 from one API might mean "back off for a second," while from another it means "you're permanently rate-limited on this key." A 500 can mean "retry me" or "this request will never work, stop trying." Treating the code as ground truth instead of a hint is where most brittle retry logic comes from.
Building an actual error model — classifying by retryability, whether it's transient vs. structural, whether the failure is on the input (bad prompt, context too long) vs. the provider (overload, timeout, moderation block) — is the right instinct. It's basically the same lesson distributed systems folks learned with HTTP a decade ago: you can't treat the transport-layer status as the whole story, you need a domain-specific taxonomy on top of it.
One thing I'd be curious about: how do you handle providers that don't document their error semantics well and just change behavior silently (rate limit thresholds, timeout windows, etc.)? That feels like the hardest part to keep a model like this accurate over time — did you build in any drift detection, or is it mostly manual maintenance as providers change?

[Lolo]

That's something I've been thinking about as well.

Right now it's mostly manual because provider error semantics don't change very often, but I definitely don't want the classification logic to become a pile of provider-specific exceptions over time.

One idea I'm exploring is making the routing layer more data-driven, so changes in provider behavior can be updated without touching the rest of the codebase. Still early, but I think that's a cleaner direction than hardcoding every edge case.

[Nazar Boyko]

One thing worth thinking about early: the classifier is now its own failure surface, and it fails silently. If a provider ships a new error type tomorrow, or tweaks a message string you were matching on, classifyProviderError doesn't throw. It just quietly buckets the thing into server_error and your caller happily retries a request that will never work. Worth having an explicit unknown class that never gets a retry, plus a log line loud enough that you find out a provider changed shape from your own alerts and not from a support ticket. Everyone in this thread is arguing about which classes belong in the set, which is the fun part, but the boring part is what the thing does when it doesn't recognize what it's looking at.

[Wren Calloway]

The place this model earns its keep — and where it quietly breaks — is that error_class isn't stable per status code even within one provider. A 429 is your classic example: same code, sometimes rate_limited, sometimes overloaded. So whatever classifyProviderError does, it can't be a status-code lookup table; it has to read the body's type/code and, for the ambiguous cases, sometimes the Retry-After header or error message text. That's the actual work, and it's the part that rots — providers change error type strings without changing the status code, and your classifier silently starts dumping things into server_error.

Which is the caveat I'd flag: put a metric on the "fell through to server_error" branch. That default bucket is where misclassifications go to hide, and since you're logging the raw error but only returning the class, a provider renaming overloaded_error to overloaded won't page anyone — it'll just quietly turn a retryable failure into one your retry logic treats as fatal. The normalization is only as good as your visibility into what it failed to normalize.

[UnitBuilds]

Gunna be a really useful feature when you start dealing with multi-account load-balancing. For the V.E.L.O.C.I.T.Y. IDE (side-branch of V.E.L.O.C.I.T.Y. OS), I've got an accounts table with 17 free accounts that load-balance and allows for higher concurrency for agents. What's important about it though is the distinction between out of quota and server out of supply. Both cases you need to switch gears, but 1 means swap to a different account, while the other means you switch models. The AI Gateway is taking solid shape now, ngl, it might become an industry standard tool if you keep going at this pace.

[Lolo]

Curious about your setup, 17 accounts for load balancing sounds like exactly the kind of thing I built Apiarium to avoid managing by hand.
What's the part of that system that's been the most annoying to maintain?

[UnitBuilds]

Honestly, I just set it up with a simple failover, if a sequence is partially completed, it just fails over to the next one and passes the context, it just continues seamlessly sofar with Kimi K2.7. I guess the tedious part was mostly just getting the 17 API keys, but since they got added to the list, not a single missed beat.

[Lolo]

Fair, though I'd expect that to hold until a provider quietly renames an error type on you. The failover logic sounds solid either way.

[UnitBuilds]

That's the nice part of cloudflare, they cant change anything, because if they do, half the internet needs to be compliant post-change. The risk outweighs the gain.

[Alex Shev]

This is the part model gateways need more of. Status codes are too thin for operational routing. A useful error model should know whether the failure is retryable, provider-specific, quota-related, safety-policy related, malformed input, or output-shape drift. Otherwise failover just moves the confusion to another model.

[Mads Hansen]

This is the right abstraction layer to add before the system gets painful.

Status codes are useful transport hints, but they are not a product-level failure model. Once multiple providers are involved, callers need to know what kind of decision to make, not which vendor-shaped JSON blob happened to arrive.

The same pattern shows up in agent tools too. A raw error like “timeout”, “permission denied”, or “invalid request” is rarely enough. The runtime needs a small set of durable categories:

• retryable later
• retryable with smaller input
• blocked by auth or policy
• invalid tool arguments
• provider capacity issue
• permanent unsupported request

Then routing, retries, UX copy, alerts, and analytics can all depend on the category instead of reverse-engineering provider behavior every time.

Also +1 on keeping the raw provider error for internal logs. Normalizing for callers should not mean throwing away evidence for debugging.

[Xiao Man]

The error taxonomy you landed on maps onto a pattern I keep seeing in a different context — agent quality gates. Same fundamental problem: you need a small set of durable categories that let downstream systems make consistent decisions, regardless of which upstream layer produced the failure.

Evans touched on the drift question and I think it's the hard one. Providers will silently change behavior, and your classification will start mislabeling. The same thing happens when you use an LLM as a quality inspector — its calibration drifts as the executing model gets updated. One approach that seems promising: keep a thin "calibration layer" between provider-specific behavior and your classification categories, updated via observed patterns rather than manual code changes. Basically: let the data tell you when a new error shape showed up, then decide which bucket it belongs to.

The streaming-mid-death case Max mentioned is going to be interesting. A 200 that never delivers is a fundamentally different failure class from a 500 that never started.

[Alex Shev]

An error model is the right abstraction because provider failures are rarely interchangeable. A timeout, safety refusal, malformed JSON, quota issue, and degraded latency tail should not all become “retry.” Once errors are typed, routing and fallback logic can be much less superstitious.