A lot of AI apps die in the same place.
Not during the prototype phase.
Not while testing prompts.
Not even during the “which model should we use?...
For further actions, you may consider blocking this person and/or reporting abuse
Really solid breakdown.
I’ve run into this pattern myself when building small AI prototypes; everything feels fine until streaming and tool calls start stacking up.
Yeah, that’s exactly the point where things stop being just a prototype 😄
Streaming and tool calls can look harmless at first, but once they stack up, the whole system starts behaving very differently in production.
Glad the breakdown resonated with your experience.
One axis I'd add for the agentic use case specifically: how the gateway handles governance over tool/model calls, not just traffic.
For agent workflows the interesting policy isn't "rate limit this route", it's "this agent identity can call these tools, with these token budgets, and log every call for audit". That's closer to API management than to a reverse proxy.
Worth folding into the comparison alongside the ones listed:
Tyk.io which comes at this from the full-lifecycle/open-source angle rather than pure proxying
(disclosure: I work on Tyk, and the problems you describe, with some of the worlds largest brands, so this is in my wheelhouse).
Tyk brings a different trade-off to ngrok/Traefik, which are lighter but leave more of the policy layer to you.
That’s honestly one of the gaps most “classic gateway” comparisons miss.
Once you move into agentic systems, it stops being just traffic control and becomes much more about behavioral governance, who/what is allowed to call which tools, under what budget, and how that gets traced end-to-end for auditability. That layer feels much closer to API management than traditional proxying, like you said.
And agreed on the nuance with tools like Tyk; the trade-offs really shift depending on whether you want a lighter routing layer or a full lifecycle governance system baked in from the start.
There is a lot of experimentation going on in the space right now. Tyk.io took the stance of developing a separate Open Source AI Management tool : Tyk AI Studio, concerned with LLM management. The Tyk gateway covers more traditional routing and management concerns, including observability, for MCP.
I think open source is essential at this point, as the space is developing so rapidly. Proprietary is almost bound to become redundant in short order.
Would be great to get your take on this : github.com/TykTechnologies/ai-studio
the 'AI requests are expensive' section is the part teams discover too late. gateway level rate limiting helps, but the real footgun with MCP agents is per user token isolation. if 20 users share the same gateway auth context, one runaway agent burns the whole quota.
we hit this building a Next.js MCP server — per user OAuth had to live one layer closer to the model, not just at the gateway edge. rate limiting the wrong identity unit is almost as bad as no rate limiting.
are any of these gateways starting to support per user OAuth flows natively for MCP, or is that still custom middleware?
Nice overview. For Chinese LLM gateways, Meshs One (meshs.one) is worth a look — HK-based, DeepSeek V3/R1 + Qwen family + MiniMax, one endpoint. Priced 60-80% below official. Authorized channels. Happy to answer Qs if anyone's evaluating Chinese LLM options.