When you need an Agent Gateway, not just another LLM proxy

If your problem is "one OpenAI-compatible endpoint for 50 providers," a proxy like LiteLLM is usually the right default. Huge catalog, huge community, good enough ops for most teams.

If your problem is "agents that call tools (MCP) with the same identity, audit, budgets, and policy as model calls," a thinner Agent Gateway starts to matter — one control plane so auth doesn't drift between chat completions and tool traffic.

Honest fit table (no fake benches)

Need	Usually start with	Why
Broadest provider catalog, ecosystem glue	LiteLLM	Depth beats novelty
Hosted control plane, zero ops	Portkey / Helicone-class	Ops off your plate
Self-host, low overhead Go proxy	Bifrost / similar	Latency + governance focus
Self-host LLM + MCP same policy/identity	Agent Gateway (e.g. TrustGate)	One plane for models + tools
K8s/service-mesh native	Envoy AI Gateway / Kong	Fits existing mesh

What "Agent Gateway" should mean

• OpenAI-compatible path for apps you already have
• MCP / tool gateway on the same identity + audit trail
• Fail-closed budgets / rate limits before the vendor 429s you
• Self-hostable so prompts stay in your perimeter

Consumers + registries (the MCP aggregation plane)

In practice the useful model is:

1. Registries hold upstream MCP servers and model providers (keys stay server-side).
2. Consumers get their own credentials into the gateway (per app / team / agent).
3. tools/list is filtered per consumer so agents never see tools they cannot call.
4. The same consumer identity covers chat completions and tools/call, so spend and audit line up.

That is the wedge: not "another 100 providers," but one aggregation plane for models + tools.

Try TrustGate (OSS, Apache-2.0)

Repo: https://github.com/NeuralTrust/TrustGate

Built in Go by NeuralTrust. Quickstart and "Why TrustGate" live in the README — including where it is not the right pick vs LiteLLM/Helicone/Portkey.

---

Disclosure: TrustGate / NeuralTrust DevRel.