Calling LLM providers directly works until it doesn't. The moment you're managing multiple API keys, tracking costs across providers, or scrambling during a provider outage, you need a gateway.
But not all AI gateways are created equal. Some are pure proxies. Some lock you into their ecosystem. Some charge you for features that should be free.
We evaluated eight AI gateways on what actually matters: provider coverage, pricing transparency, self-hosting, observability, and how fast you can get started.
1. LLM Gateway
Best overall. Open source. Self-hostable.
LLM Gateway is an open-source API gateway that routes requests to 200+ models across 40+ providers through a single OpenAI-compatible endpoint. Change your base URL, keep your existing code.
What sets it apart:
- Open source (AGPLv3) — inspect the code, self-host on your own infrastructure, no vendor lock-in
- Zero markup on BYOK — bring your own provider keys and pay nothing extra on top of provider costs
- Built-in caching — Redis-powered response caching cuts repeat request costs to zero
- Real-time analytics — cost tracking, latency monitoring, and usage breakdowns per model, project, and API key
- Automatic failover — requests reroute to backup providers when the primary goes down
- Guardrails (Enterprise) — content safety rules with configurable filters
-
AI SDK provider — first-class Vercel AI SDK integration via
@llmgateway/ai-sdk-provider - Playground — test and compare models side-by-side with chat, group chat, and image generation
Pricing: Free tier with credits to start. Pay-as-you-go with no hidden fees. No markup when using your own keys.
Best for: Teams that want full control over their AI infrastructure without building it from scratch.
import OpenAI from "openai";
const client = new OpenAI({
apiKey: "YOUR_GATEWAY_KEY",
baseURL: "https://api.llmgateway.io/v1",
});
// Works with any of 200+ models
const response = await client.chat.completions.create({
model: "claude-sonnet-4-5",
messages: [{ role: "user", content: "Hello!" }],
});
2. OpenRouter
Large model catalog, community-driven.
OpenRouter aggregates models from many providers and offers a unified API. It has a wide selection of models including open-source and fine-tuned variants.
Strengths:
- Large model catalog including community and fine-tuned models
- OpenAI-compatible API
- Usage tracking dashboard
Weaknesses:
- Not open source — you can't self-host or audit the code
- Charges a 5.5% fee on credit purchases (no per-token markup)
- BYOK is free up to 1M requests/month, then a 5% fee
- Observability has improved (request logs, exports, OpenTelemetry) but lives behind its own dashboard
- Response caching is newer and still in beta
Pricing: Pay-as-you-go with a 5.5% credit-purchase fee; provider token rates pass through with no markup.
Best for: Developers who want quick access to a wide variety of models and don't need self-hosting.
3. Portkey
Enterprise-focused with governance features.
Portkey positions itself as an AI gateway for enterprises, with emphasis on observability, guardrails, and compliance features.
Strengths:
- Detailed request logging and traces
- Guardrails and content moderation
- Multi-provider routing with fallbacks
- Enterprise compliance features
Weaknesses:
- The gateway and much of the platform are open source (MIT), but persistent observability storage and compliance stay in the managed cloud
- Enterprise pricing can be opaque
- Heavier setup compared to simpler gateways
- Now part of Palo Alto Networks (acquired May 2026) — a consideration if you prefer an independent vendor
Pricing: Free tier with limited requests. Paid plans for higher volume and enterprise features.
Best for: Large enterprises with strict compliance and governance requirements.
4. LiteLLM
Open-source proxy with broad provider support.
LiteLLM is an open-source Python proxy that translates OpenAI-compatible requests to 100+ providers. It's popular as a self-hosted solution.
Strengths:
- Open source (MIT license)
- Wide provider support
- Active community
- Python-native
Weaknesses:
- Python-only — heavier runtime for deployment
- Built-in usage dashboard and spend logs, though deeper observability still leans on integrations
- No managed hosting — you handle infrastructure yourself
- Caching is built in, but you configure and run the backend yourself
- Less polished developer experience
Pricing: Free and open source. You pay for your own infrastructure.
Best for: Python teams comfortable with self-hosting and managing their own infrastructure. Outgrowing it? See the best LiteLLM alternatives.
5. Helicone
Observability-first with proxy capabilities.
Helicone started as an LLM observability platform and added gateway features. It excels at logging, monitoring, and cost tracking. Note: Helicone was acquired by Mintlify in 2026 and is now in maintenance mode rather than active development.
Strengths:
- Excellent request logging and analytics
- Easy integration (one-line header change)
- Cost tracking and alerting
- Prompt management features
Weaknesses:
- In maintenance mode since the 2026 Mintlify acquisition — not actively developed
- Gateway routing has historically been secondary to observability
- Not designed as a full gateway replacement
- Can become expensive at high request volumes
Pricing: Free tier with limited requests. Usage-based pricing beyond that.
Best for: Teams whose primary need is visibility into LLM usage rather than routing and failover.
6. Vercel AI Gateway
Zero markup, deep AI SDK integration.
Vercel AI Gateway routes to hundreds of models across 45+ providers through one endpoint, with both OpenAI- and Anthropic-compatible APIs. It went GA in 2025 and is the default provider for the Vercel AI SDK.
Strengths:
- Zero markup on tokens — including with your own keys (BYOK)
- First-class Vercel AI SDK integration (
@ai-sdk/gateway) - Automatic failover, provider routing, and automatic caching
- Observability and spend monitoring built in
Weaknesses:
- Not open source or self-hostable — managed cloud only
- Credits and BYOK are tied to a Vercel team account
- Some governance features (custom reporting, team-wide ZDR/allowlists) cost extra
- Strongest when you're already in the Vercel/Next.js ecosystem
Pricing: Pay-as-you-go credits with no token markup. Paid add-ons for custom reporting and team-wide governance.
Best for: Teams building on the Vercel AI SDK who want zero markup and tight ecosystem integration.
7. Cloudflare AI Gateway
Edge-based with Cloudflare ecosystem integration.
Cloudflare AI Gateway leverages their edge network to proxy and cache LLM requests. Tight integration with the Cloudflare ecosystem.
Strengths:
- Edge caching for low-latency responses
- Rate limiting and cost controls
- Simple setup if already on Cloudflare
- No per-request fees
Weaknesses:
- Limited to Cloudflare's supported providers
- Basic analytics compared to dedicated solutions
- Tightly coupled to Cloudflare ecosystem
- No BYOK — limited provider key management
- Fewer advanced routing features
Pricing: Free tier included with Cloudflare account. Paid plans for higher limits.
Best for: Teams already invested in the Cloudflare ecosystem who want basic gateway features.
8. AWS Bedrock
Cloud-native for AWS shops.
AWS Bedrock provides access to foundation models through AWS infrastructure. It's less of a traditional gateway and more of a managed model access layer within AWS.
Strengths:
- Deep AWS integration (IAM, VPC, CloudWatch)
- Enterprise security and compliance
- No infrastructure to manage
- Access to exclusive models (Amazon Nova)
Weaknesses:
- AWS lock-in — tightly coupled to the AWS ecosystem
- Limited model selection compared to dedicated gateways
- Complex pricing with multiple dimensions
- No OpenAI-compatible API — requires AWS SDK
- Slower to add new models from third-party providers
Pricing: Pay-per-token with AWS pricing. No upfront costs.
Best for: Organizations fully committed to AWS that need models within their existing cloud infrastructure.
Comparison Table
| Feature | LLM Gateway | OpenRouter | Portkey | LiteLLM | Helicone | Vercel AI GW | Cloudflare AI GW | AWS Bedrock |
|---|---|---|---|---|---|---|---|---|
| Open Source | Yes | No | Partial | Yes | Yes | No | No | No |
| Self-Hostable | Yes | No | Partial | Yes | Yes | No | No | No |
| BYOK (No Markup) | Yes | After 1M | Yes | Yes | N/A | Yes | No | N/A |
| OpenAI-Compatible | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No |
| Built-in Caching | Yes | Beta | Yes | Built-in | No | Yes | Yes | No |
| Analytics Dashboard | Yes | Yes | Yes | Built-in | Yes | Yes | Basic | CloudWatch |
| Automatic Failover | Yes | Yes | Yes | Config | Limited | Yes | No | No |
| Guardrails | Enterprise | Enterprise | Yes | No | No | No | No | Yes |
| Models | 300+ | 400+ | 1,600+ | 100+ | N/A | Hundreds | ~20 | ~30 |
| Playground | Yes | Yes | No | No | No | Yes | No | Yes |
How to Choose
You want full control and no lock-in: LLM Gateway is the only option that's open source, self-hostable, and charges zero markup on your own keys. You get enterprise features without enterprise pricing.
You want the widest model selection: LLM Gateway and OpenRouter both offer extensive catalogs. LLM Gateway covers 200+ models with the added benefit of self-hosting and BYOK.
You want observability above all: Helicone is purpose-built for logging and analytics, though LLM Gateway and Portkey offer comparable dashboards with full gateway capabilities included.
You're locked into a cloud provider: AWS Bedrock makes sense if your entire stack is AWS and you need models within that boundary. Just know you're trading flexibility for integration.
You're on the Vercel AI SDK: Vercel AI Gateway is the natural fit with zero token markup — though you trade away self-hosting and bring-your-own-infrastructure options.
You're cost-conscious: Gateways that support BYOK (LLM Gateway, LiteLLM, Vercel) let you avoid middleman markup entirely. Over thousands of daily requests, the savings compound fast.
Getting Started with LLM Gateway
If you want to try the top pick, you can be running in under two minutes:
- Sign up free — no credit card required
- Create a project and copy your API key
- Point your existing OpenAI SDK to
https://api.llmgateway.io/v1
That's it. Your existing code works. Every request gets logged, cached, and tracked automatically.
Top comments (0)