Your AI stack works fine with one model. The moment you add a second provider, a fallback, a rate limit, or a cost ceiling, everything breaks. Rout...
For further actions, you may consider blocking this person and/or reporting abuse
Great breakdown, this matches what we've seen operationally. One thing I'd add to the "how to pick" section: the fallback behavior under partial failure matters more than the feature checkbox. A lot of teams pick a gateway because it has "fallback ✅" on a comparison table, then get burned in production because it only triggers on hard errors (5xx) and not on a provider that's just gotten slow. Circuit breakers with latency-based tripping are the real differentiator, and it's not something you can tell from a feature matrix, you have to load test it yourself.
Also worth flagging for anyone choosing based on this table: LiteLLM's proxy can become a genuine bottleneck at high concurrency if you don't run it with enough workers and a properly tuned Redis backend for the cache/budget layer. It's easy to underprovision it because the initial setup feels so lightweight.
Solid overview overall, especially the point that this is an infra decision, not a model decision. Curious whether you've tested any of these under simulated provider degradation (not just outage), that's usually where the real differences show up.
Really good addition, and honestly the more important one. "Fallback: ✅" on a comparison table tells you almost nothing about the condition that actually triggers it. Most implementations I've looked at are wired to hard failures, connection refused, 5xx, timeout past some generous threshold, which covers the easy case and completely misses the one that actually costs people money in production: a provider that's still returning 200s but has quietly degraded to 3x normal latency. Nothing trips, nothing fails over, and you just eat the slow responses until someone notices error budgets look fine but users are complaining.
Latency-based circuit breaking is the real signal, and you're right that it's invisible from a feature matrix. You have to actually inject degraded latency (not just kill the endpoint) and watch whether the gateway's failover logic reacts to it. A lot of them don't, by design or by oversight, because "is it up" is a much easier thing to build detection for than "is it behaving normally."
The LiteLLM point is a good one to flag explicitly. The proxy is deceptively easy to stand up, which I think causes people to treat that initial config as the steady state instead of a starting point. Under real concurrency the worker count and the Redis tuning for the cache and budget layer stop being optional details and start being the thing that determines whether the gateway itself becomes the tail latency problem you were trying to avoid in the first place.
On your question, yes, simulated degradation is where most of the differentiation actually showed up for us too. Clean outage tests make almost every gateway look fine since binary failure is the easy case to handle. Once you inject partial degradation, elevated latency without errors, intermittent slow responses, a provider that's healthy but rate limiting, you see which ones have real health-check logic versus which ones are just watching for connection failures. That gap doesn't show up until you go looking for it, which is probably the single biggest thing missing from most public comparisons of these tools.
It's been a good exchange, thanks for engaging with it in depth. Good luck with the load testing, and if you end up writing up the degraded-latency results somewhere, that'd be a genuinely useful public data point given how little of that testing gets shared.
This is a useful list, thanks for putting it together. Would add one nuance: "maturity" in the table is doing a lot of work for Kong and APISIX. The core gateways are rock solid, but the AI-specific plugins on both are genuinely new, so "mature (AI layer newer)" is a bit generous when you're evaluating things like semantic caching correctness or token counting accuracy under streaming responses. Worth a longer bake-off before betting production spend controls on them.
Also curious how you're defining "broadest provider support" for LiteLLM in practice, since a lot of the 100+ list is thin wrappers where things like function calling or structured outputs don't behave consistently across providers. Provider count and provider parity are two different claims.
Good writeup either way, bookmarking this for the next time someone on my team asks "just use LangChain right?" and I get to say no.
Fair pushback on both counts, thanks for pressing on them.
On Kong and APISIX, you're right that I collapsed two different maturity claims into one cell. The core proxy in both cases has years of production hardening behind it, but the AI plugins are a recent layer bolted onto that foundation, and recent means exactly what you'd expect: less battle testing on the parts that are hardest to get right. Semantic caching correctness is a good example, since a cache hit on the wrong semantic match is a silent failure mode, not a loud one, and you won't find it without deliberately testing for it. Token counting under streaming is similar, the failure is a slow drift in accuracy rather than a crash. Both deserve their own bake-off rather than inheriting the reputation of the underlying gateway. I'll fix the table to split those out instead of averaging them into one "mature" label.
On LiteLLM, that's a real distinction I should have been more careful with. Provider count is trivially true and easy to verify, you can count the entries in the docs. Provider parity is the claim that actually matters for production and it's much harder to verify, since it means digging into whether function calling, structured outputs, and streaming behavior are consistent rather than just present. A lot of that 100+ list is coverage in the sense of "a request will not immediately error out," not coverage in the sense of "the contract behaves the same way across providers." I was sloppy conflating the two and "broadest provider support" should really be read as "broadest provider count," with parity flagged as a separate, unverified claim that needs testing per feature per provider before you rely on it.
Appreciate you keeping the writeup honest. Consider both corrections noted for the next revision.
Nice comparison, this is a topic that needs more clear-eyed writeups. I'd push back slightly on grouping Kong AI Gateway and Envoy AI Gateway with the "purpose-built" options in the takeaways section. If a team isn't already running Kong or Envoy, the operational cost of adopting either just for LLM routing is significant, you're taking on an entire API gateway or service mesh layer to get token rate limiting. That's a very different commitment than standing up LiteLLM or Helicone.
Would also add that "self-hostable" hides a lot of variance in actual maintenance burden. LiteLLM's proxy is a fairly small footprint to run yourself. Kong and Envoy are not, even before you touch the AI-specific plugins.
Appreciate that you called out the cost attribution gap for APISIX specifically, that's the kind of detail that only shows up after you've actually tried to get finance an answer, not from reading docs.
That's a fair pushback, and honestly the grouping was doing too much work in that takeaways section.
The distinction you're drawing is the right one. Kong and Envoy are infrastructure layers that happen to have AI gateway plugins now, not tools built for the AI gateway use case first. If you're already running Kong for your API estate or Envoy as your mesh data plane, adding the AI plugin is a marginal cost, you're extending something you operate anyway. But if a team's starting from zero, "just add the AI gateway plugin" undersells what they're actually taking on: config model, upgrade cadence, the whole operational surface of a gateway or mesh, before they've routed a single LLM request. That's a different category of decision than "install LiteLLM proxy and point traffic at it."
Good call on self-hostable hiding variance too. I was using it as a binary when it's really a spectrum of operational weight. LiteLLM's proxy is close to a single deployable service with a config file. Kong and Envoy come with their own control plane concerns, their own failure modes, their own people-need-to-know-this-tool cost, independent of anything AI-specific. Lumping all three under "self-hostable" in a takeaways table flattens exactly the distinction a team evaluating this needs.
On APISIX, that one came from watching someone actually try to hand finance a per-team cost breakdown and hit a wall the docs don't warn you about. Glad it landed, that kind of gap only shows up in practice, never in a feature comparison table.
Appreciate you pushing on the grouping, I'll rework that section.
Solid comparison, and I like that you framed this as an infrastructure decision rather than a "which gateway is best" ranking. Too many of these roundups pretend there's one winner.
One thing I'd add to the picking criteria: it's worth checking how each gateway handles config drift once multiple teams start using it. A single YAML file or control plane works great with one team defining routes, but once five teams are adding models, budgets, and fallback rules to the same gateway, you need real ownership boundaries (per-team config scoping, review workflows, etc.) or you end up with the exact mess this article is trying to help people avoid, just one layer up.
Also agree with the comment thread below about fallback being a checkbox that hides a lot of nuance. "Handles failover" and "handles a provider that's slow but still returning 200s" are very different bars to clear, and most teams don't find out which one they actually have until it's a production incident.
Good writeup either way, bookmarking it for the next "just build it ourselves" conversation.
Really appreciate this perspective. Config drift and ownership boundaries are exactly the kind of problems that only become obvious once multiple teams start sharing the same gateway.
And your point on fallback is spot on, provider failure and provider degradation require very different strategies. A 200 response doesn't necessarily mean the request is healthy.
Thanks for adding this nuance to the discussion, and glad you found the writeup useful!