If you've shipped anything with more than one model provider in the last year, you've hit the same wall: your code is full of if provider == "anthropic" branches, your retry logic is duplicated three times, and nobody on the team can tell you this month's actual OpenAI spend without opening a billing dashboard and squinting. The fix everyone reaches for now is a "gateway" — a proxy layer that sits between your app and the model APIs, gives you one interface, and (in theory) handles routing, fallback, caching, and cost tracking for you.
The problem is that four of the most-recommended options — OpenRouter, LiteLLM, Portkey, and Helicone — all describe themselves with nearly identical marketing copy ("unified API," "one line of code," "100+ models"), and most comparison posts stop at a feature checklist. That checklist hides the actual decision, which isn't about model count. It's about who runs the proxy, who eats the failure when a provider goes down at 2am, and — the part vendors bury deepest — who takes a cut of your inference spend for the privilege of routing it.
Why this decision is live right now
Three things pushed "which gateway" from a nice-to-have to a required line item in most AI-product architecture docs this year. First, multi-model has stopped being an experiment — teams route reasoning-heavy calls to one model family and cheap classification calls to another as a matter of routine cost control, which means hardcoding a single SDK is no longer viable past a prototype. Second, agent frameworks (LangChain, CrewAI, AutoGen) multiplied the number of LLM calls per user action, which multiplied the blast radius of a single provider outage or a single misconfigured retry loop. Third, finance teams started asking for per-team, per-feature cost attribution on AI spend the same way they ask for it on cloud spend — something none of the raw provider SDKs give you out of the box.
That's the actual job these four tools compete for: unified API, provider failover, spend control, and observability, bundled in different ratios. None of them do all four equally well, and the ratio is the part worth understanding before you wire one into production.
The failure mode that makes this urgent is almost always the same story: a provider has a bad afternoon, every request that used to take 800ms starts timing out at 30 seconds, and the on-call engineer discovers that "retry logic" in the codebase means three different services each independently retrying against the same degraded endpoint, making things worse. A gateway centralizes that decision once instead of re-implementing it per service — which is genuinely valuable — but centralizing it into a system you don't operate just moves the single point of failure, it doesn't remove it. That trade-off is the real spine of this comparison, and it's the one thing all four vendors are least eager to put on their homepage.
What each one actually is
OpenRouter is a hosted marketplace, full stop. You get one API key, an OpenAI-compatible endpoint, and access to over 200 models across dozens of providers with no infrastructure to run yourself. It's the fastest possible path from zero to "my app talks to five different model families." There is no self-hosted version — OpenRouter's business is being the metered middleman, and every request either uses OpenRouter's own provider credits or your own keys routed through their infrastructure.
LiteLLM is the open-source option with the largest install base by a wide margin — its GitHub repository sits at roughly 56,000 stars, several times the others. It ships two ways: a Python SDK you import directly, or a standalone proxy server you run yourself (Docker, Kubernetes with Helm charts, or Terraform modules for AWS ECS/Aurora and GCP Cloud Run). LiteLLM's own materials cite Stripe, Google, and Netflix as users of the proxy in production, which — vendor-supplied logos aside — tells you the intended deployment shape: a team that wants a gateway living inside their own VPC, not a third party's.
Portkey splits the difference. Its AI Gateway component is a genuinely open-source (MIT-licensed), lightweight router — the project advertises a roughly 122kb footprint and sub-millisecond added latency, claims from its own benchmarks rather than independently verified numbers — that you can self-host for free. On top of that, Portkey sells a hosted SaaS layer and an enterprise private-deployment option that adds semantic caching, prompt management, and governance tooling (guardrails, RBAC, SOC2/HIPAA/GDPR/CCPA compliance claims) that the free gateway doesn't include.
Helicone started as an observability tool and grew a gateway, which is the inverse of Portkey's path. It's open source under Apache 2.0, with a cloud free tier (10,000 requests/month) and a self-hosted option. The gateway piece gives you OpenAI-compatible routing with automatic fallback across 100+ models; the older, more mature half of the product is request-level tracing, cost/latency/quality dashboards, and session debugging for agent and chatbot traffic. If you already run Helicone for observability, the gateway is a natural add; if you're shopping for a gateway first, it's the newer, less battle-tested half of the product.
Worth noting how differently "self-hosted" cashes out even between the two open-source-first options here. Run LiteLLM yourself and you're operating one proxy process against a config file and a database for keys and budgets — the kind of thing a single engineer can own. Run Helicone yourself and, per its own architecture docs, you're operating five coordinated services, because the product was built around ClickHouse-backed trace analytics from day one and the gateway was added later without collapsing that footprint. Neither is objectively wrong — one optimizes for "cheap to run," the other for "rich to query" — but they are not the same size of commitment, and a feature-matrix comparison won't show you that difference.
How they're actually built — and why the plumbing matters
This is where the checklist comparisons fall short, because "self-hosted" means wildly different things across these four.
LiteLLM's proxy is a single service you point your app at — one container, a Postgres-backed config for virtual keys and budgets, and you're done. That simplicity is a large part of why it has the deployment footprint it does among infra teams that don't want a new distributed system to operate.
Helicone's self-hosted stack is heavier: by its own documentation it's five separate services — a Next.js frontend, a Cloudflare Workers proxy, an Express server, a Supabase database, a ClickHouse instance for analytics, and MinIO for object storage. That's a legitimate architecture for a product built observability-first (ClickHouse is the right tool for high-cardinality trace data), but it means "self-hosting Helicone" is closer to standing up a small data platform than dropping in a proxy container. Worth knowing before you commit a sprint to it.
Portkey's open-source gateway is deliberately minimal — it's meant to be embedded, not operated as a platform — while the parts that look like a platform (dashboards, governance, semantic cache) live behind the hosted or enterprise product. That's a coherent design if you view the gateway as infrastructure and the governance layer as a separate buying decision; it's a source of confusion if you assumed "open source" meant the whole product.
OpenRouter, again, isn't self-hostable at all. Its architecture is entirely OpenRouter's problem, which is exactly the appeal for a two-person team and exactly the objection from a security team that needs prompts and completions to never leave a controlled network boundary.
What changed versus a year or two ago
The clearest shift is in how OpenRouter prices "bring your own key" (BYOK) usage — routing your own provider API keys through OpenRouter's infrastructure instead of paying OpenRouter directly for inference. The free allotment used to be denominated in request count (roughly 1 million requests/month on the pay-as-you-go tier, 5 million on Enterprise). It's now denominated in dollars of list-price inference — $25,000/month free on pay-as-you-go, $200,000/month on Enterprise — with a 5% fee on BYOK usage above that. For a team routing large volumes of already-cheap requests, counting by request was more generous; counting by spend is more generous for high-token, low-request workloads. Either way, it's a meaningful pricing-model change that's easy to miss if you priced OpenRouter out a year ago and haven't rechecked.
The other visible shift is Portkey and Helicone both converging toward "gateway plus governance" as the pitch, rather than "gateway" alone — a sign that raw request routing has become table stakes and the real competition has moved to caching, guardrails, and spend attribution sold as a layer on top.
The economics, worked through
The fee numbers are easy to skim past, so it's worth actually running them. Say a team is spending $40,000/month on inference and routes it through OpenRouter using their own provider keys (BYOK) rather than OpenRouter's own credits. The first $25,000 of that is free under the current pay-as-you-go terms; the remaining $15,000 is billed a 5% fee, which is $750/month, or $9,000/year, purely for routing traffic you were already going to send to the same providers directly. Double the spend to $80,000/month and the fee scales to roughly $2,750/month — because the $25,000 free allotment doesn't grow with usage, only the paid Enterprise tier's $200,000 threshold does, and Enterprise pricing is a separate negotiation, not a self-serve toggle.
Run the same $40,000/month through a self-hosted LiteLLM or Portkey OSS proxy instead, and the marginal fee is zero — the cost is whatever compute the proxy itself needs, which for a stateless routing service handling that volume is a rounding error next to the inference spend it's routing. The gap only widens with scale, which is exactly why LiteLLM's cited adopters (Stripe, Google, Netflix, per its own materials) skew toward companies operating at a volume where a percentage-of-spend fee stops being negligible. The flip side is equally real: at $2,000/month of inference spend, a 5% fee is $100 — plausibly cheaper than the engineer-hours it'd take to stand up and maintain a self-hosted proxy for a team that doesn't have one already. There's a real crossover point buried in these numbers, and it's closer to "tens of thousands of dollars a month" than most teams initially assume.
Why the choice actually costs you something
Cost. OpenRouter's fee structure is the one number that's easy to overlook until you're at scale. If you let OpenRouter pay providers on your behalf using purchased credits, topping up by card costs a 5.5% fee with an $0.80 floor per transaction — a real, recurring tax on top of whatever margin is baked into the displayed per-token prices. If you BYOK instead, you avoid that but pay 5% on inference once you clear the free monthly allotment. LiteLLM, Portkey's OSS gateway, and Helicone's OSS gateway charge nothing per token — you pay your own compute to run the proxy and, if you want the managed/enterprise tiers, a subscription. For high-volume production traffic, the self-hosted options are structurally cheaper; for low-volume or spiky traffic, the infrastructure cost of running your own proxy can easily exceed what OpenRouter's fee would have been.
Latency. Every gateway adds a hop. LiteLLM publishes an 8ms P95-at-1,000-RPS figure and Portkey claims sub-millisecond overhead for its gateway — both self-reported benchmarks, not third-party audited numbers, so treat them as "the vendor believes this is fast" rather than a guarantee under your traffic shape. The practical takeaway is the same for all four: a well-run proxy adds single-digit milliseconds; a poorly-scaled self-hosted deployment (or a shared-tenant SaaS gateway under someone else's load) can add much more, and you won't know which you have until you load-test your own path.
DX. OpenRouter wins here unambiguously for anyone starting from zero — no deployment, no config file, a model catalog you can browse in a UI, working in minutes. LiteLLM's DX is good once it's running but the initial setup (config YAML, database, virtual key provisioning) is a real onboarding cost. Portkey's split between a trivial-to-embed OSS gateway and a separately-configured hosted dashboard can feel disjointed until you've decided which half you actually need. Helicone's DX is strongest if observability is your primary need and routing is secondary — the tracing UI is the more mature product.
Lock-in. This is the one marketing pages actively obscure. A unified API format is portable in theory, but caching keys, guardrail configurations, and prompt-management tooling tied to one vendor's dashboard are not — migrating away later means rebuilding that config, not just swapping a base URL. Self-hosted, open-source gateways reduce this risk structurally, because the routing logic and your provider keys never leave infrastructure you control; a hosted marketplace like OpenRouter is a deliberate trade of that control for zero operational burden. Neither choice is wrong, but "unified API" marketing tends to imply portability that only actually exists at the routing layer, not at the governance layer most teams add on top.
Security. If prompts or completions contain anything you can't send to a third party — PII, regulated health or financial data, unreleased product content — a hosted marketplace is a harder sell regardless of its compliance certifications, because the data still transits infrastructure you don't control. This is the single strongest argument for LiteLLM or a self-hosted Portkey/Helicone gateway inside a VPC, and it's the argument most likely to override every other column in a feature comparison.
Maintainability. LiteLLM's single-service model is the easiest to keep running long-term with a small team. Helicone's five-service self-hosted stack is more capable but is a genuine ongoing-ops commitment — ClickHouse and MinIO don't run themselves. Portkey's OSS gateway is low-maintenance by design because it's intentionally thin; the moment you need the governance features, you're either paying for hosted/enterprise or building that layer yourself. OpenRouter has zero maintenance burden by construction, which is precisely what you're paying the fee for.
Practical fits
OpenRouter is the right call for prototypes, hackathon projects, indie products testing which model family performs best on their task, and any team that wants model diversity without provisioning a single server. A solo developer A/B-testing three model families for a new feature can be live in the time it takes to read this paragraph, with no proxy to deploy and no config to write. It's also a reasonable production choice for low-to-moderate volume products where the 5%/5.5% fees are a rounding error next to the engineering time saved — the mistake is leaving it in place unexamined once volume grows past the point where that's still true.
LiteLLM fits platform and infra teams standing up an internal AI gateway for many internal consumers — the virtual-key and budget system is built exactly for "give every team and service its own spend cap" governance, so a platform team can hand product teams a key with a hard monthly ceiling instead of a shared credential and a Slack message asking people to be careful. The self-hosted deployment also satisfies data-residency requirements that rule out a SaaS marketplace outright, which for regulated industries isn't a preference, it's a compliance gate.
Portkey suits teams that want to start with the free OSS gateway and grow into paid governance tooling without a re-architecture — you adopt the same routing layer on day one that you'd still be running once you're paying for semantic caching and formal guardrail policies, so the migration path is a billing conversation rather than a rewrite. Teams already standardized on LangChain, CrewAI, or AutoGen for agent orchestration will also find the integration surface smaller, since Portkey's gateway is built with those frameworks as first-class consumers rather than an afterthought.
Helicone is the pick when observability is the primary problem you're solving — debugging why an agent looped, tracing a multi-step pipeline, understanding cost per user session — and gateway routing is a secondary convenience you'll take advantage of since it's already there. A team that's already fighting fires trying to figure out why an agentic workflow burned through its token budget overnight will get more immediate value from Helicone's trace UI than from any of the other three's routing feature set.
What the marketing pages don't say
OpenRouter doesn't advertise that its BYOK free tier changed shape recently, or that credit purchases carry a separate 5.5%-plus-floor fee distinct from the BYOK fee — both are easy to discover only by reading the fine print or a third-party breakdown, not the front page.
LiteLLM's headline benchmark numbers are self-published, and the enterprise features gating SSO and dedicated support behind a commercial license aren't obvious from the open-source README — you'll find that boundary when you go looking for SSO, not before.
Portkey's compliance badges (SOC2, HIPAA, GDPR, CCPA) apply to the hosted and enterprise offerings; running the open-source gateway yourself puts the compliance posture back in your hands entirely, which is correct and expected but is easy to conflate with "the product is compliant" when skimming the homepage.
Helicone's self-hosted footprint — five services rather than one — isn't emphasized on the marketing site, which tends to lead with "one line of code" the same as everyone else; the operational reality only becomes clear in the self-hosting docs.
None of these are dishonest, exactly — they're the standard omissions of any vendor page. But they're precisely the details that change a "which gateway" decision from a five-minute pick to an informed one.
"Guardrails" and "caching" mean different amounts of product
All four vendors use the words "guardrails" and "caching" somewhere on their site, and the words carry very different weight depending which one you're reading. Portkey's open-source gateway ships with more than 40 pre-built guardrail checks according to its own repository documentation — input/output validation rules you can attach to a request without writing custom logic — while its semantic caching (matching a new prompt against a previously-cached one by meaning rather than exact string) is reserved for the hosted and enterprise tiers. LiteLLM's proxy supports guardrails and caching as configurable middleware in the self-hosted deployment itself, which is more capability in the free tier but requires you to wire and operate it rather than toggle it in a dashboard. Helicone's strength here is on the read side, not the write side: its dashboards surface cost, latency, and quality signals per request in more depth than the other three, but request-time controls like guardrails and caching are comparatively newer additions layered onto an observability-first product.
The practical implication: if guardrails and caching are a near-term requirement rather than a someday one, don't assume "has guardrails" on a comparison table means the same amount of actual policy engine in every row. Read the docs for what's configurable versus what's a dashboard toggle behind a paywall before you architect around it.
Head-to-head
| Dimension | OpenRouter | LiteLLM | Portkey (OSS gateway) | Helicone |
|---|---|---|---|---|
| Deployment | Hosted only, no self-host | Self-hosted proxy (Docker/K8s) or SDK | Self-hosted (MIT) or hosted/enterprise | Self-hosted (5-service stack) or hosted |
| License | Proprietary SaaS | Open source + commercial enterprise tier | MIT (gateway); paid hosted/enterprise | Apache 2.0; paid hosted/enterprise |
| GitHub stars (approx.) | N/A (closed source) | ~56,000 | ~12,700 | ~6,100 |
| Model/provider catalog | 200+ models, one key | 100+ providers via unified format | 250+ LLMs | 100+ models |
| Per-token fee | 5% on BYOK inference above $25k/mo free tier; 5.5%+$0.80 floor on credit top-ups | None (self-hosted; you pay compute) | None (OSS gateway; paid tiers are subscriptions) | None (OSS gateway; paid tiers are subscriptions) |
| Spend/budget controls | Account-level credit balance | Virtual keys with per-team/user budgets | Budget limits (hosted/enterprise) | Cost dashboards; budget alerts (hosted) |
| Self-host operational load | N/A | Low — single proxy service | Very low — thin gateway process | High — 5 services incl. ClickHouse, MinIO |
| Observability depth | Basic usage dashboard | Logging + admin dashboard | Advanced in hosted tier | Strongest of the four — its original product |
| Guardrails/caching | Not core to product | Available in proxy | 40+ checks (OSS); semantic caching (paid) | Available; deeper in paid tier |
| Compliance certifications | Not published | Enterprise tier only | SOC2/HIPAA/GDPR/CCPA (hosted/enterprise) | Enterprise tier |
| Best-fit team size | Solo/small team, prototypes | Platform team, internal gateway at scale | Growing team, OSS-to-paid path | Team prioritizing observability first |
An independent read
Judged purely on adoption signal, LiteLLM has won the "default open-source gateway" slot — its star count and cited enterprise users aren't proof of superior engineering, but they are proof that when infra teams need a self-hosted routing layer, LiteLLM is what they reach for first, which matters for hiring, community support, and the odds your specific edge case has already been hit by someone else. That's a real advantage independent of feature parity.
OpenRouter's value proposition hasn't weakened, but it's narrower than the marketing suggests: it's excellent for discovery and low-friction multi-model access, and it's a genuinely bad fit the moment data residency, cost at real scale, or provider-outage independence matter — because you've added a dependency on OpenRouter's own uptime and pricing decisions on top of the underlying providers'.
Portkey and Helicone are both, in effect, betting that governance and observability — not routing — are where the durable value sits, since routing itself has become commoditized enough that four different projects implement roughly the same fallback logic. That bet looks reasonable; a raw router is a weekend project, but good spend attribution and trace debugging across a fleet of agents is not. The open question is whether either company's free OSS gateway is a genuine on-ramp to their paid tier or a loss leader that gets deprecated in favor of push toward hosted — worth watching before betting a production architecture on the free tier's continued parity with the paid one.
It's also worth being honest that "which gateway" is frequently not a permanent decision, and treating it as one is where teams overinvest. The unified request/response format these tools share is the genuinely portable part — swapping OpenRouter for a self-hosted LiteLLM proxy later is a base-URL and auth-header change for most application code, assuming you haven't built business logic against a specific vendor's routing rules or dashboard. The parts that don't move cleanly are exactly the parts covered above: virtual-key structures, guardrail policies, and cached-response behavior. A pragmatic sequencing that shows up repeatedly in how teams actually adopt these tools is starting on OpenRouter to de-risk the product question — does multi-model routing even help our metrics — before spending engineering time self-hosting anything, then migrating to LiteLLM or Portkey once volume and data-residability requirements justify the operational cost. That sequencing costs a rewrite of the request layer later, but a rewrite of the request layer is far cheaper than guessing wrong on infrastructure before you know if the product needs it.
Who should pick what
Pick OpenRouter if you're pre-product-market-fit, need model breadth today, and the fee structure is negligible against your current volume — revisit once monthly spend clears low five figures.
Pick LiteLLM if you're a platform or infra team that needs self-hosted routing with real spend governance across multiple internal teams, and you'd rather operate one proxy service than pay a per-token fee at scale.
Pick Portkey if you want to start free and open-source now with a credible path to paid governance tooling later, without re-platforming when you get there.
Pick Helicone if debugging and understanding what your agents are actually doing is the more urgent problem than routing itself, and you're willing to operate a heavier self-hosted stack — or pay for the hosted tier — to get it.
What's your actual experience been migrating between these once you outgrew the first one you picked — did the "unified API" promise hold, or did guardrail/caching config end up being the part you had to rebuild by hand?
Sources:
Top comments (0)