DEV Community: Andrew S. Bandy

Best LiteLLM Alternative in 2026

Andrew S. Bandy — Fri, 25 Sep 2026 16:08:50 +0000

It was a Tuesday in March when my Slack lit up.

A friend who runs platform engineering at a mid-sized fintech sent me one line: "Did you pin LiteLLM?" He hadn't. On March 24, 2026, two LiteLLM releases on PyPI (1.82.7 and 1.82.8) shipped with a credential-stealing payload. It harvested environment variables, SSH keys, cloud credentials, Kubernetes tokens, and database passwords, then sent them to an attacker-controlled domain. The packages had been uploaded directly to PyPI, bypassing the project's CI. LiteLLM's own post-mortem covers the details.

PyPI quarantined the versions fast. That wasn't the worrying part. The worrying part was where LiteLLM sits in most stacks: in front of every LLM provider key your company owns. An AI gateway is the one box that holds the keys to everything.

After that week, many teams started searching for a LiteLLM alternative. This post is the guide I wish my friend had then.

Disclosure: I'm the founder of GoModel, one of the tools below. I'll tell you where it wins and where another option is the better choice. If I'm not honest about that, this post is just an ad.

TL;DR

If you need...	Pick
A self-hosted, lightweight, drop-in LiteLLM replacement	GoModel
An LLMOps platform with evals, A/B tests and fine-tuning loops	TensorZero
No infrastructure at all, and your data can go to a third-party cloud	OpenRouter

What people actually use LiteLLM for

Before you swap anything, look at what LiteLLM does for you today. In almost every team I've talked to, it comes down to five jobs:

One API for many providers. Your app speaks OpenAI format, and the gateway translates for Anthropic, Gemini, Bedrock, and the rest.
Key management. Provider keys live in one place, not in every service.
Cost tracking and budgets. Who spent $4,000 on GPT calls last weekend?
Fallbacks and routing. When a provider has an outage, your product keeps working.
Observability. Logs, metrics, traces.

A good LiteLLM alternative has to do all five. Ideally it should also be harder to compromise, and it shouldn't turn into another thing you have to babysit.

1. GoModel: the best LiteLLM alternative for self-hosting

Repo: github.com/ENTERPILOT/GoModel · License: MIT · Language: Go

I started building GoModel in late 2024 because I kept running into the same problems with LiteLLM in enterprise environments: a large Python dependency tree, heavy containers, and a lot of moving parts sitting in front of the most sensitive credentials in the company. March 2026 made those concerns very concrete.

Why it's my top pick

It ships as a single static Go binary. That's the main point, not a footnote. There's no pip install pulling in hundreds of transitive packages at deploy time, and no .pth file that can execute on interpreter start. You get one binary, a small (~17 MB) Docker image, and a dependency graph you can audit in an afternoon. The whole gateway is a smaller attack surface than the payload that hit LiteLLM users.

It's a real drop-in. GoModel exposes OpenAI-compatible and Anthropic-compatible APIs, so your existing SDK code usually only needs a new base_url:

docker run --rm -p 8080:8080 \
  -e LOGGING_ENABLED=true \
  -e OPENAI_API_KEY="your-openai-key" \
  enterpilot/gomodel

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="your-gomodel-key")
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello from GoModel"}],
)

GoModel detects which providers are available from the credentials you supply, so there's no long YAML file before your first request. Add ANTHROPIC_API_KEY or GEMINI_API_KEY and those models are available too.

Broad provider coverage: OpenAI, Anthropic, Google Gemini, Vertex AI, Azure OpenAI, Amazon Bedrock, DeepSeek, Groq, Fireworks, xAI, Alibaba Cloud, MiniMax, Oracle, OpenRouter, and self-hosted models through Ollama and vLLM, plus any OpenAI-compatible endpoint.

It covers the five jobs above:

Cost and token tracking, with budget controls
Exact and semantic caching
A guardrails pipeline
Prometheus metrics and OpenTelemetry (kept in the free MIT core, not paywalled)
An admin dashboard and REST API
A provider-native passthrough endpoint (/p/{provider}/...) for when you need a feature the unified API doesn't expose yet

It works where cloud gateways can't. If you're in banking, healthcare, the public sector, or anywhere with an air-gapped network or strict EU data-residency rules, you need prompts and keys to stay on your own infrastructure. GoModel was designed for that from the start.

Where GoModel is not the right choice

If you want a full experimentation and fine-tuning platform with built-in evals, see TensorZero below.
If you don't want to run any infrastructure, see OpenRouter.
GoModel is younger than LiteLLM. The core is solid and releases ship frequently, but if you depend on a niche LiteLLM integration, check the provider list first.

2. TensorZero: when you want an LLMOps platform, not just a gateway

Repo: github.com/tensorzero/tensorzero · License: Apache 2.0 · Language: Rust

TensorZero is a strong project, and I recommend it to people regularly. It's built for a different goal than GoModel.

GoModel sits between your app and your providers. TensorZero aims to close the loop: it stores every inference with the feedback you attach, and then helps you use that data to improve prompts and models. You get:

A high-performance Rust gateway (they report <1 ms p99 overhead at 10k+ QPS)
Observability backed by your own database
Evaluations with heuristics and LLM judges
Optimization workflows: supervised fine-tuning, RLHF, automated prompt engineering
Adaptive A/B testing, routing, fallbacks and retries

Choose TensorZero if you have an ML team that wants to run structured experiments on prompts and models, and you're willing to adopt its way of doing things. You define "functions" and "variants" in config rather than just pointing your SDK at a new URL.

Think twice if you only need a gateway. Its data and optimization layer is its main value, and it's also more to run and learn. For most teams leaving LiteLLM, a simpler drop-in gets them back to shipping faster.

3. OpenRouter: when the cloud is fine

Site: openrouter.ai · Managed SaaS

Sometimes the honest answer is: don't run a gateway at all.

OpenRouter gives you one API key and one bill for a very large catalog of models across many providers. There's nothing to deploy, patch, or monitor. For a side project, a hackathon, an early-stage startup, or a team that wants to try twenty models this week, that's great.

The trade-offs:

Your prompts and responses go through a third party. For many companies that's fine. For regulated industries, or anything covered by strict data-residency rules, it often isn't.
Fees. OpenRouter charges a fee on credit purchases (around 5.5% at the time of writing). Bring-your-own-key usage is free up to a monthly allowance, with a small fee after that. At low volume this doesn't matter. At scale you'll notice it. Check their current pricing before committing.
Less control over routing logic, caching, and guardrails than a self-hosted gateway gives you.

You don't have to choose one or the other. GoModel supports OpenRouter as an upstream provider, so you can self-host the control plane (keys, budgets, logs, guardrails) and still use OpenRouter's catalog for long-tail models.

Side-by-side

	GoModel	TensorZero	OpenRouter	LiteLLM (baseline)
Deployment	Self-hosted	Self-hosted	Managed cloud	Self-hosted
Language	Go (single binary)	Rust	n/a	Python
License	MIT	Apache 2.0	Proprietary	MIT core + enterprise
Drop-in OpenAI-compatible API	✅	✅	✅	✅
Anthropic-compatible API	✅	Partial	✅	✅
Cost tracking and budgets	✅	✅	✅ (billing)	✅
Caching (exact + semantic)	✅	Limited	Provider-level	✅
Guardrails	✅	Via functions	Limited	✅
Evals / fine-tuning loop	❌	✅	❌	❌
Air-gapped / on-prem friendly	✅	✅	❌	✅
Supply-chain surface	Small	Small	Vendor's problem	Large (PyPI tree)
Time to first request	~1 min	~15–30 min	~1 min	~5 min

Setup times are my rough experience, not formal benchmarks. Try them yourself.

How to choose (the human version)

Forget feature matrices for a minute and ask three questions:

1. Can my prompts leave my network?
If not, OpenRouter is out. Look at GoModel or TensorZero.

2. Do I need a gateway or an experimentation platform?
If you want your existing code to keep working and your security team to relax, choose GoModel. If you want to run your own fine-tuning and A/B testing loop, choose TensorZero.

3. Who's on call for this at 3 a.m.?
This question matters more than any benchmark. A single binary with few dependencies is easier to keep running than a stack with many services. Pick the tool your team can operate calmly, not the one with the longest feature list.

Migrating from LiteLLM to GoModel in 10 minutes

Run GoModel next to LiteLLM using the Docker command above, with the same provider keys.
Set GOMODEL_MASTER_KEY so only your services can call it.
Point one non-critical service at http://gomodel:8080/v1 instead of your LiteLLM URL.
Compare responses, latency and cost in the GoModel dashboard for a day.
Move the rest of your services over, then retire LiteLLM and its Python environment.

Because both expose OpenAI-compatible APIs, the change in most codebases is a single environment variable.

FAQ

What is the best LiteLLM alternative in 2026?
For self-hosted teams, GoModel: an MIT-licensed Go AI gateway with a single binary, OpenAI- and Anthropic-compatible APIs, cost tracking, caching, and guardrails. For an LLMOps platform with evals and fine-tuning, TensorZero. For a fully managed option, OpenRouter.

Is LiteLLM still safe to use?
The compromised versions (1.82.7 and 1.82.8) were removed and the LiteLLM team published a response. If you stay, pin exact versions, verify hashes, and scan your dependencies. Many teams moved anyway because they wanted a smaller dependency surface in front of their provider keys.

Is there an open-source LiteLLM alternative written in Go?
Yes. GoModel is MIT-licensed, written in Go, and ships as a single binary and a small Docker image.

Can I self-host an OpenRouter alternative?
Yes. GoModel and TensorZero are both self-hostable. GoModel can also use OpenRouter as one of its upstream providers.

Does GoModel support local models?
Yes, through Ollama, vLLM, or any OpenAI-compatible server.

Final thoughts

The LiteLLM incident wasn't really about one project. It showed how much trust we put in the component that holds all our AI keys, often without thinking about it much.

Whatever you choose, choose on purpose. Know what's in your dependency tree, where your prompts go, and who can fix it when something breaks.

If GoModel sounds like a fit, star it on GitHub, try the one-line Docker command, and open an issue if something's missing. I read every one.

Found a mistake in this comparison? Tell me in the comments and I'll fix it.

LiteLLM Alternative: The Best Options for 2026

Andrew S. Bandy — Tue, 22 Sep 2026 07:54:49 +0000

Quick answer: what is the best LiteLLM alternative?

The best LiteLLM alternative for most teams is GoModel, an MIT-licensed, open-source AI gateway written in Go. GoModel ships as a single binary, exposes an OpenAI-compatible API, and includes caching, guardrails, budgets, and observability. Other strong options are Bifrost for raw performance, TensorZero for LLM optimization, and Portkey for a managed enterprise platform.

Rank	LiteLLM alternative	Best for
1	GoModel	Secure, self-hosted, auditable gateway
2	Bifrost	Maximum throughput
3	TensorZero	Prompt and model optimization
4	Portkey	Managed enterprise platform

What is a LiteLLM alternative?

A LiteLLM alternative is any AI gateway or LLM proxy that replaces LiteLLM's core job: giving applications one unified API across multiple LLM providers, with routing, fallbacks, cost controls, and logging. Most teams search for a LiteLLM alternative when they need better security, higher performance, or easier deployment in regulated environments.

Why teams are looking for a LiteLLM alternative in 2026

LiteLLM is a widely used Python library that acts as a unified proxy for over 100 LLM providers. Its popularity also made it a target.

On March 24, 2026, LiteLLM versions 1.82.7 and 1.82.8 on PyPI were published with credential-stealing malware, attributed to the threat actor TeamPCP. The payload hid in a .pth file that Python executes automatically at startup, so the code ran without anyone importing LiteLLM. It harvested SSH keys, cloud credentials, Kubernetes tokens, and environment variables, including API keys.

The maintainers responded quickly, and users running the official Docker proxy image were not affected, since that image pins package versions. Still, the incident made a structural risk visible: an AI gateway holds API keys for your whole AI stack, and a Python package with a deep dependency tree is a large attack surface for that job.

The other common reasons teams look for a LiteLLM alternative are:

Performance. A Python proxy usually needs more replicas and tuning under heavy load than a compiled Go or Rust gateway.
Deployment constraints. Air-gapped and on-premises environments favor self-contained binaries over runtime package installs.
Upgrade predictability. LiteLLM's fast release pace can make upgrades harder to manage.

How to choose a LiteLLM alternative

A good LiteLLM alternative should offer:

An OpenAI-compatible API, so migration is a base-URL change.
A minimal supply-chain footprint, ideally a compiled binary with pinned dependencies.
Governance features: virtual keys, budgets, rate limits, and fallbacks.
Caching, both exact and semantic.
Open-standard observability through OpenTelemetry and Prometheus.
Self-hosting, including on-prem and air-gapped support.
Transparent licensing, with enterprise features clearly separated from the open core.

1. GoModel: the best overall LiteLLM alternative

GoModel is an open-source LiteLLM alternative written in Go, designed for secure, self-hosted AI gateway deployments.

Language: Go
License: MIT (open core)
Deployment: single binary, Docker image of about 17 MB
API: OpenAI-compatible
Website: gomodel.enterpilot.io

Why GoModel is the top LiteLLM alternative

GoModel was built specifically to address the limitations of Python-based gateways. It compiles to one static binary with no interpreter, no pip install at deploy time, and no .pth-style auto-execution path. Security teams can checksum it, scan it, and move it into a locked-down network as a single artifact.

GoModel covers the features LiteLLM proxy users rely on:

Exact and semantic caching
Budget controls and usage tracking
A guardrails pipeline for prompts and responses
Prometheus metrics and OpenTelemetry, both free in the MIT core
An admin dashboard
Per-provider-family translators that respect each provider's API differences
A passthrough endpoint (/p/{provider}/...) for native provider features

GoModel enterprise and cloud

GoModel's enterprise module adds SSO, RBAC, audit logs, and multi-tenancy. Unlike many open-core gateways, that code lives in a visible /ee folder in the public repository, so buyers can review it. GoModel Cloud offers a hosted bring-your-own-key tier for teams that don't want to run infrastructure.

Who should choose GoModel

Choose GoModel as your LiteLLM alternative if you left LiteLLM over security concerns, work in a regulated industry, need on-prem or air-gapped deployment, or want an AI gateway you can fully audit.

Trade-off: GoModel is younger than LiteLLM, so confirm your specific providers are supported before migrating.

2. Bifrost: the fastest LiteLLM alternative

Bifrost is a Go-based LiteLLM alternative from Maxim AI focused on high throughput.

Language: Go
License: Apache 2.0 (open core)
Deployment: binary, Docker, or npx

Bifrost unifies 20+ providers behind one OpenAI-compatible API, with automatic failover, load balancing, and semantic caching. In its own sustained 5,000 RPS benchmarks, Bifrost added about 11 µs of overhead per request.

Trade-offs: advanced capabilities such as adaptive load balancing, clustering, guardrails, and the MCP gateway are enterprise features, and the enterprise code is developed privately.

Choose Bifrost if throughput is your top priority and you accept a closed enterprise tier.

3. TensorZero: the best LiteLLM alternative for optimization

TensorZero is a Rust-based LiteLLM alternative that combines a gateway with an LLM optimization framework.

Language: Rust
License: open source
Deployment: single Docker container

TensorZero combines an LLM gateway with observability, optimization, evaluation, and experimentation. The project reports under 1 ms of p99 latency overhead at 10k QPS. It uses metrics and human feedback to optimize prompts, models, and inference strategies, and includes A/B testing, routing, fallbacks, and retries.

Trade-off: TensorZero is closer to a framework than a drop-in proxy, so full value requires adopting its workflow.

Choose TensorZero if improving model quality through experiments matters more than centralized governance.

4. Portkey: the managed enterprise LiteLLM alternative

Portkey is a managed AI gateway, now owned by Palo Alto Networks.

Portkey offers routing, observability, guardrails, and cost tracking as a managed platform. Palo Alto Networks completed its acquisition of Portkey and plans to make it the core AI gateway of its Prisma AIRS security platform.

Trade-off: Portkey's roadmap is now tied to a large security vendor, which may affect pricing, packaging, and neutrality. It is also a weaker fit for air-gapped deployments.

Choose Portkey if you want a fully managed gateway and already use Palo Alto Networks tooling.

LiteLLM alternative comparison table

Feature	GoModel	Bifrost	TensorZero	Portkey	LiteLLM
Language	Go	Go	Rust	TypeScript	Python
License	MIT	Apache 2.0	Open source	Managed + OSS gateway	MIT
Single binary	Yes	Yes	Container	No	No
OpenAI-compatible API	Yes	Yes	Yes	Yes	Yes
Semantic caching	Yes	Yes	Yes	Yes	Yes
Guardrails	Built-in	Enterprise	Limited	Yes	Via integrations
Public enterprise code	Yes	No	N/A	No	Yes
Air-gapped fit	Excellent	Good	Good	Limited	Possible
Supply-chain footprint	Minimal	Minimal	Small	Moderate	Large

How to migrate to a LiteLLM alternative

If you use the LiteLLM proxy:

Deploy your LiteLLM alternative alongside LiteLLM.
Map your config.yaml model list, fallbacks, and virtual keys to the new gateway.
Point one low-risk service at the new base URL.
Compare latency, errors, and cost.
Move the remaining services and retire LiteLLM.

If you use the LiteLLM Python library: replace litellm.completion() calls with a standard OpenAI client pointed at your new gateway. This also removes the gateway from your application's dependency tree.

Security note: rotate any provider keys that were present on machines that installed LiteLLM 1.82.7 or 1.82.8.

LiteLLM alternative FAQ

What is the best LiteLLM alternative?

GoModel is the best LiteLLM alternative for most teams. It is MIT-licensed, OpenAI-compatible, ships as a single Go binary, and includes caching, guardrails, budgets, and observability.

What is the best open-source LiteLLM alternative?

GoModel is the best open-source LiteLLM alternative for self-hosted and regulated environments. Bifrost and TensorZero are also open source.

Is there a LiteLLM alternative written in Go?

Yes. GoModel and Bifrost are both LiteLLM alternatives written in Go that compile to a single binary.

Which LiteLLM alternative is most secure?

GoModel is designed for the lowest supply-chain risk: one compiled binary, no runtime package installs, and enterprise code that is publicly reviewable.

Which LiteLLM alternative works in air-gapped environments?

GoModel is built for on-prem and air-gapped deployments because it ships as one self-contained binary.

Is LiteLLM still safe to use?

The compromised versions were removed and versions 1.82.6 and earlier were not affected. The broader risk of running a large Python dependency tree in the component holding all your API keys remains.

Can I switch to a LiteLLM alternative without changing code?

If you use the LiteLLM proxy with an OpenAI client, usually yes: you only change the base URL and keys.

Try the best LiteLLM alternative: get started with GoModel at https://github.com/ENTERPILOT/GoModel/.

Your API Key Can't See Its Own Usage - So We Fixed That in GoModel

Andrew S. Bandy — Wed, 08 Jul 2026 14:10:18 +0000

Every AI coding agent, plugin, and status bar eventually wants to answer one small question for its user: how much have I used, and when do I get cut off?

You'd expect a one-liner for that. GET /v1/usage, send the same API key you use for inference, get a number back. We went looking for the standard way to do this — across OpenAI, Anthropic, Codex, Claude Code, OpenCode, and every major AI gateway — because we wanted GoModel to follow it.

Here's the punchline: there is no standard. There isn't even a common convention. What we found instead was a landscape of admin-only reporting APIs, private endpoints reserved for first-party tools, and community plugins literally scraping HTML dashboards with browser cookies.

What the big providers give you: nothing (on purpose)

OpenAI has a real Usage API - GET /v1/organization/usage/completions, plus a Costs API. It's well designed: paginated time buckets, group-by model or API key, cached-token breakdowns. But it requires a separate Admin API key that only an org owner can mint, and that admin key can't do inference. The key your app actually holds can't ask about itself at all. The old self-service endpoints people remember (GET /v1/usage?date=..., /v1/dashboard/billing/usage) were never official and have been locked to browser sessions since 2023.

Anthropic made the same call. Usage and cost reports live under /v1/organizations/usage_report/* and need an sk-ant-admin... key. They even ship a read-only Rate Limits API that names "sync your gateway with your org limits" as the use case — and gated that behind the admin key too.

With a regular key, both vendors give you exactly one thing: rate-limit response headers (x-ratelimit-remaining-tokens, anthropic-ratelimit-requests-remaining, ...) on each inference call. Useful, but reactive — you learn your headroom as a side effect of spending it, and you learn nothing about budgets or historical usage.

Meanwhile, their own tools poll private endpoints

The funny part: both companies clearly know consumers need this, because their own products have it — privately.

Codex polls chatgpt.com/backend-api/wham/usage (OAuth-only, undocumented) about once a minute to render its usage display. The response is percent-of-window: {"primary_window": {"used_percent": 12, "reset_at": ...}}.
Claude Code's /usage command calls api.anthropic.com/api/oauth/usage — also undocumented, also OAuth-only, and it even rate-limits you into oblivion if you don't send the right User-Agent. Community usage monitors reverse-engineered it anyway.

So the pattern at the top of the market is: self-service usage exists, but only for our own clients, on internal endpoints you're not supposed to use.

And the tools downstream are hurting

The demand side is not subtle. The example that motivated this work: OpenCode — the open-source coding agent — has a gateway (Zen) with real usage windows and balances, and no API for any of it. The result:

A community plugin (opencode-quota) scrapes the billing dashboard's HTML using browser cookie auth to display quota percentages.
There are open issues asking for GET /zen/v1/balance and usage-window endpoints, and an open proposal for GET /zen/go/v1/usage returning {"windows": [{"name": "5 hour", "usagePercent": 42, "resetInSec": 4711}]}.
Ollama Cloud users are filing the same issues (/api/me, /api/usage) for the same reason.

When your users are scraping your dashboard with cookies, they have told you what API they want.

The closest thing to a convention

Two gateways actually shipped self-service introspection, and one got close:

OpenRouter: GET /api/v1/key with your normal inference key returns usage, usage_daily/weekly/monthly, limit, limit_remaining, limit_reset. This is the de-facto reference design.
Requesty: GET /v1/manage/apikey/self returns monthly_spend and monthly_limit with the same bearer key.
LiteLLM: GET /key/info with no parameter defaults to the calling key and returns spend, max_budget, budget_reset_at, tpm_limit, rpm_limit.

Distill those and you get the only "standard" that exists — a design pattern, not a schema: GET, same base URL, same bearer key, self-referential, no parameters required. Plus rate-limit headers on inference responses as the complementary channel.

How GoModel does it

GoModel now answers the whole question in one call. GET /v1/usage, authenticated with the same key you use for chat completions:

curl http://localhost:8080/v1/usage \
  -H "Authorization: Bearer sk_gom_..."

{
  "user_path": "/team/alpha",
  "server_time": "2026-07-07T09:30:00Z",
  "usage": {
    "start_date": "2026-06-08",
    "end_date": "2026-07-07",
    "total_requests": 128,
    "total_tokens": 91234,
    "cached_input_tokens": 12790,
    "total_cost": 1.73
  },
  "budgets": [
    {
      "user_path": "/team/alpha",
      "period_label": "monthly",
      "amount": 100,
      "spent": 41.2,
      "remaining": 58.8,
      "usage_ratio": 0.412,
      "period_end": "2026-08-01T00:00:00Z",
      "resets_in_seconds": 2125800,
      "exceeded": false
    }
  ],
  "rate_limits": [
    {
      "user_path": "/team/alpha",
      "period_label": "minute",
      "max_requests": 60,
      "requests_used": 3,
      "requests_remaining": 57,
      "requests_usage_ratio": 0.05,
      "window_end": "2026-07-07T09:31:00Z",
      "resets_in_seconds": 60,
      "exhausted": false
    }
  ]
}

Three blocks, three questions answered:

usage — requests, tokens (with cache breakdowns), and estimated cost over a date window you control (days, or start_date/end_date, up to 365 days; last 30 by default).
budgets — every spend limit that gates you, including ones inherited from parent paths, with spent, remaining, the period bounds, and an exceeded flag that mirrors what enforcement will actually do.
rate_limits — live counters for every rate-limit rule covering you: used, remaining, and when the window resets.

A few design decisions worth explaining:

It's scoped by user path, not by key. In GoModel, budgets, rate limits, and usage accounting all hang off a hierarchical user_path (like /team/alpha/service), and API keys bind to a path. Several keys can share one path, and when we add a full users system, a user will simply be a path. Every self-service precedent out there is key-centric; ours is identity-centric — but from the caller's perspective it behaves identically: send your key, get your own status, and you can't read outside your subtree.

Polling is free. The endpoint doesn't consume rate-limit quota, so a status widget checking every few seconds never eats into the budget it's displaying. (Codex polls its internal endpoint every 60 seconds for the same reason.)

Absolute numbers and the percentages. The consumer-facing idiom at OpenAI and Anthropic is percent-of-window (used_percent, utilization) — deliberately opaque about actual quotas. We return real counts and real money with real window boundaries, and then precompute the display idiom anyway. Every budget carries a usage_ratio, a resets_in_seconds countdown, and an exceeded flag. Every rate limit carries an exhausted flag, a usage ratio per limited dimension (requests_usage_ratio, tokens_usage_ratio), and — for windowed rules — the same countdown; a concurrency cap has no window, so it reports in-flight slots instead. Countdowns are relative to the server_time in the same payload, so client clock skew can't break them, and both flags mirror what enforcement will actually do. A status bar renders this with zero math; a plugin that wants the raw numbers has them too.

Headers still work. GoModel keeps emitting OpenAI-style x-ratelimit-* headers on inference responses, so the endpoint and the in-band signals tell one consistent story.

The endpoint is in GoModel today, and it exists because someone asked for exactly the OpenCode-plugin use case described above. Full request and response reference — every field, the date window, the error cases — is in the GoModel Usage API docs.

The takeaway

Usage visibility has quietly become admin-only across the LLM ecosystem — not because it's hard, but because nobody made it part of the API contract. The result is a cottage industry of OAuth token borrowing, dashboard scraping, and GitHub issues asking for the same three fields: used, limit, resets at.

If you run a gateway, your callers shouldn't need admin access to know when they're about to hit a wall. Give them one GET.

Benchmarking AI Gateways: GoModel vs LiteLLM vs Portkey vs Bifrost

Andrew S. Bandy — Fri, 26 Jun 2026 17:51:26 +0000

In October 2025 I tried to build my startup on top of LiteLLM.

At first it looked like the obvious choice. It supported many providers, it had
an OpenAI-compatible API, and it was already used by a lot of people. I did not
want to write an AI gateway. I wanted to build the product behind it.

Then I started running it on the hot path.

My opinion changed there.

A gateway is not a dashboard or integration glue you call once in a while. It
sits on every request, every retry, every stream, every tool call, every
fallback, every timeout.

A heavy gateway charges rent forever.

Most AI gateway comparisons miss that part. They talk about provider count,
dashboards, tracing, and "support for 1000+ models". Those things matter, but
they are not free. Before the gateway calls OpenAI, Anthropic, Gemini, vLLM, or
anything else, it has already spent your CPU, memory, cold-start time, and
operational budget.

I am not comparing full product maturity here. I am comparing how these gateways
behave on the hot path.

So I started writing GoModel: a small
open-source AI gateway and AI control plane in Go, with an OpenAI-compatible API
and explicit provider adapters.

When I launched GoModel on Hacker News,
I promised a real, reproducible benchmark. This article is that follow-up.

The benchmark question is simple:

How lean is each AI gateway when it sits on the request path?

That question runs through the whole benchmark: GoModel vs LiteLLM vs Portkey vs
Bifrost, measured by latency, throughput, memory, CPU, cold start, and image
size rather than landing pages or feature matrices.

The runtime footprint matters

Latency gets the easiest arguments. It rarely tells the whole story.

Most real LLM calls are dominated by inference time. If a model takes 2000 ms
to answer, the difference between 5 ms and 15 ms of proxy overhead is not
the main story.

The main story is the deployment envelope:

How much RAM does the gateway need under load?
How much CPU does it burn per request?
How many requests can it serve per core?
How fast does it cold-start?
How large is the Docker image?
Can you run it as a sidecar, on a small VM, in serverless, or near local models?
Is the core gateway actually open-source?

Those numbers decide whether the gateway can run where you want it to run.

A 372 MB compressed image (1.2 GB unpacked) that idles around gigabytes of
RAM and takes 25 s to cold-start is a different operational thing than a
16 MB image that peaks at 37 MB of RAM and is serving traffic 0.56 s after
launch.

So I care about the runtime footprint.

What this benchmark does not prove

This benchmark does not prove that one gateway is best for every company.

I am not measuring:

bug counts or overall correctness
semantic cache quality
tracing UI quality
guardrail quality
admin dashboards
long-term provider maintenance
every possible provider-specific feature
total provider count

Those things matter. Some of them matter a lot.

LiteLLM in particular has more integrated providers and more gateway features
than GoModel today. If your first requirement is maximum provider coverage right
now, LiteLLM has a real advantage. This benchmark does not erase that. It
measures the runtime footprint of putting each gateway on the request path. In
practice, many smaller or newer providers already expose an OpenAI-compatible
API, so provider count is not always the same as practical routing coverage.

The benchmark measures one narrower thing: runtime and deployment overhead on
the request path.

That still matters, because the gateway is on the hot path. If you run high
request volume, local models, serverless workloads, edge workloads, or many small
model calls, the overhead stops being theoretical.

AI gateway benchmark setup

I tested four AI gateways people actually compare:

GoModel
LiteLLM
Portkey
Bifrost

Every gateway talked to the same instant mock backend, on purpose. I did not
want to benchmark OpenAI, Anthropic, AWS networking, or random internet jitter.
I wanted to isolate the gateway itself.

Each gateway ran one at a time, in Docker, on an AWS c7i.large with
2 vCPU and 4 GiB RAM, running the latest Amazon Linux 2023 AMI. The whole
thing is Terraform'd, runs with one command, and tears itself down afterwards.

I first ran this on a free-tier t2.micro. That was cheap and easy to
reproduce, but unfair to the heavier gateways. A 1 GiB machine cannot hold a
gateway that wants gigabytes of memory, so it starts swapping. At that point you
are benchmarking the host being too small.

So I moved to c7i.large: still small, but non-burstable and large enough that
nothing swaps. It also makes the LiteLLM setup more honest. LiteLLM recommends
one worker per vCPU, and this machine has 2 vCPUs, so LiteLLM gets 2
workers. That gives it the multi-core access it is supposed to have instead of
pinning it to a single worker on a tiny box.

The test covered six workloads:

chat completions, non-streaming
chat completions, streaming
Responses API, non-streaming
Responses API, streaming
Anthropic messages, non-streaming
Anthropic messages, streaming

Each workload used 8,000 requests at concurrency 10, across two trials
with randomized gateway order. Latency is the median across trials, and I
report p99 with its min-max range so one noisy window cannot tell the whole
story.

I would not call this a statistically exhaustive study. It is a reproducible
engineering benchmark, and the harness is public so people can rerun it, change
the machine, or add their own workloads.

A few details matter if you want to reproduce or criticize the numbers:

Throughput is measured, not inferred. The latency runs report completed-req/s at fixed concurrency, but real capacity comes from a separate concurrency sweep that drives each gateway to saturation.
Every dialect is warmed up before measurement. LiteLLM lazily imports some per-dialect translation code on first use. A chat-only warmup made its Responses and Messages paths look worse than they should. I warmed up all dialects to avoid that.
Retries are disabled for all gateways. I also disabled GoModel's circuit breaker for this benchmark. In production, rejecting traffic after upstream trouble is the right behavior. In a saturation benchmark, it would make the throughput number unfairly low.
LiteLLM runs with its recommended worker count. A LiteLLM worker is effectively single-threaded, and its production guidance is one worker per vCPU. On this box that means 2 workers.
Streaming uses terminal-marker or idle-gap detection. If a gateway streams content but never sends a terminal event, the harness measures to last byte instead of hanging forever.

GoModel vs LiteLLM vs Portkey vs Bifrost

Representative latency is chat completions, non-streaming. All resource figures
are measured under load on the same box.

Metric	GoModel	Bifrost	Portkey	LiteLLM
Runtime	Go	Go	Node.js	Python
Latency overhead `p50`	`1.8 ms`	`2.5 ms`	`9.7 ms`	`30.6 ms`
Latency `p99`	`6.9 ms`	`18.3 ms`	`30.5 ms`	`39.3 ms`
Throughput (sustained)	`4900 req/s`	`3100 req/s`	`950 req/s`	`324 req/s`
Peak RAM under load	`37 MB`	`143 MB`	`112 MB`	`2.3 GB`
Efficiency (req/s per CPU %)	`52`	`25`	`8.2`	`2.6`
Cold start to first request	`0.56 s`	`7.1 s`	`1.1 s`	`25.5 s`
Docker image (compressed pull)	`16 MB`	`77 MB`	`59 MB`	`372 MB`
Workload coverage	`6/6`	`6/6`	`4/6`	`6/6`
Vendor-neutral core	Yes	Partial †	Yes	Yes
Core source available	Yes ‡	Partial ‡	Partial ‡	Yes

What stood out

GoModel had the lowest median latency and the tightest tail: 1.8 ms p50 and
6.9 ms p99.

Bifrost was close on median latency at 2.5 ms, which is a good result. The
gap opened at the tail and in memory: 18.3 ms p99 and 143 MB peak RAM under
load.

Portkey was heavier than I expected for this narrow proxy benchmark. It served
950 req/s sustained and used 112 MB peak RAM under load. In this setup it did
not serve the Anthropic /v1/messages dialect, so it gets 4/6 workload
coverage. Treat that as a setup limitation, not a claim that Portkey cannot
support Anthropic in a fuller virtual-key configuration.

LiteLLM was the outlier. At its recommended worker count, it used about
2.3 GB of RAM, cold-started in 25.5 s, and sustained 324 req/s.

Not because Python is morally bad. The language matters only when it changes the
deployment envelope. Here it does: memory floor, image size, cold-start time,
dependency graph, and throughput per core.

The later supply-chain incident around LiteLLM
also made me more confident in GoModel's design direction. A small Go binary
with a standard-library-heavy dependency tree is structurally less exposed to
that class of problem than a large Python dependency graph.

What AI gateway benchmarks do not capture

Forwarding JSON is not the hard part.

The hard part is provider drift.

OpenAI, Anthropic, Gemini, AWS Bedrock, Azure OpenAI, Groq, xAI, Cerebras, vLLM,
and local servers all disagree in small ways. Then they change those ways. Tool
calling changes. Streaming changes. Reasoning parameters change. Image inputs
change. Error formats change. Rate-limit semantics change.

An AI gateway or AI control plane has to absorb that without becoming magic.

GoModel's bet is not "support every model name on the internet".

The bet is:

support the providers people actually deploy
keep provider adapters explicit
accept OpenAI-compatible requests generously
translate only what needs translation
pass through what should stay provider-specific
return conservative OpenAI-compatible responses

For the same reason, GoModel starts as a small OpenAI-compatible gateway, not as
a dashboard with a proxy attached.

Why this matters for local models and vLLM

If all your traffic goes to a cloud model that takes several seconds to answer,
gateway overhead can look academic.

Local models change the math.

If you are routing through an AI gateway to vLLM, Ollama, LM Studio, llama.cpp,
or small specialized models on your own network, the model call can be much
faster. Then gateway overhead, cold starts, memory, and sidecar size matter more.

One reason I want GoModel to stay small: a gateway should be cheap enough to put
near the workload.

Notes on neutrality and open source

Bifrost is built by Maxim AI, an LLM
evaluation and observability platform. It routes to many model providers, but
the gateway also sits close to Maxim's eval and observability ecosystem. If you
want to choose your own eval platform, or stay independent from any eval
platform, ask whether Bifrost is the right match for you. Good software can
still have incentives attached. "Vendor-neutral" needs an asterisk here.

"Open-source" also needs care.

Portkey keeps observability storage, dashboard, multi-team RBAC, and at-scale
semantic caching in a closed managed tier. Bifrost's core gateway is Apache-2.0,
but its Enterprise edition adds closed or managed features. LiteLLM's proxy core
is MIT, but enterprise features like SSO, audit logs, and fine-grained access
control sit behind a proprietary commercial license.

GoModel is open-source today. Some enterprise-grade AI control plane features may
stay private. The core gateway is intended to remain useful without those private
features.

Reproduce it yourself

The benchmark is built to be self-verifiable. It provisions the AWS instance,
runs every gateway against the same backend, prints the tables, and destroys the
infrastructure.

Reproduce it yourself:

./run.sh

One caveat: it runs on paid AWS infrastructure, not the free tier. A
c7i.large is about $0.09/hour and the run self-destructs within an hour or
two, so budget under $1 per run to be safe.

If you pass KEEP=1 or teardown fails, you keep paying until you destroy the
box, so double-check the teardown.

Conclusion

I did not start GoModel because I wanted another AI gateway in the world.

I started it because the gateway I wanted to use became part of the problem. It
sat on the hot path, but did not feel like hot-path software: too heavy, too
slow to start, too expensive to keep around, too large for the job.

This benchmark is the result of turning that frustration into numbers.

The numbers say GoModel is small in the places I care about: 16 MB image,
37 MB peak RAM, 0.56 s cold start, 1.8 ms p50, 6.9 ms p99, and
4900 req/s sustained throughput on a small AWS box.

LiteLLM still has more providers and more features today. Portkey and Bifrost
have their own strengths. But if the gateway is going to sit between your users
and every model call, I think it should first be cheap, predictable, and boring
to run.

GoModel is my attempt to build that kind of gateway.

LiteLLM was compromised, but GoModel is a good alternative

Andrew S. Bandy — Tue, 24 Mar 2026 18:39:51 +0000

LiteLLM just had a serious supply chain incident.

According to the public GitHub reports, malicious PyPI versions of LiteLLM were published, including 1.82.8, with code that could run automatically on Python startup and steal secrets like environment variables, SSH keys, and cloud credentials. The reported payload sent that data to an attacker-controlled domain. A follow-up issue says the PyPI package was compromised through the maintainer's PyPI account, and that the bad releases were not shipped through the official GitHub CI/CD flow.

This is bigger than one package. It is a reminder that the AI infra layer is now part of your security boundary.

Fortunately, there is a good alternative. GoModel: a faster, simpler alternative to LiteLLM, written in Go. Simpler, smaller and better performance for teams that want a reliable LLM gateway.

Repo link: https://github.com/ENTERPILOT/GOModel/

Benchmarking GoModel, a LiteLLM alternative: lessons learned from building a simple benchmark

Andrew S. Bandy — Mon, 23 Mar 2026 15:36:23 +0000

When I started to look for an AI Gateway for my project I've encountered GoModel. I did not plan to spend much time on benchmarking.

I assumed benchmarking would be annoying, fragile, and probably much harder than it looked. In my head, it felt like one of those tasks that sounds simple at first, but turns into a mini research project once you actually start.

What I learned is the opposite: creating a useful benchmark is much easier than most people think.

And one big reason is that AI makes the whole process much easier than it was a few years ago.

That was the biggest lesson for me.

What is GoModel?

GoModel is an open-source AI gateway / LLM proxy written in Go. It sits between your app and model providers like OpenAI, Anthropic, Gemini, Groq, xAI, and Ollama, and exposes a single OpenAI-compatible API.

I built it because I wanted a lightweight, production-friendly gateway that was easy to deploy, easy to reason about, and fully open-source.

Why I decided to benchmark it

At some point, I kept making the same claim in my head:

“GoModel feels lighter and faster.”

That may be true, but “feels” is not evidence.

I was mostly comparing it against LiteLLM, because LiteLLM is the best-known option in this space and the default reference point for many people looking at LLM gateways.

So I decided to stop guessing and just measure it.

That turned out to be one of the most useful things I have done for the project, not only because of the results, but because of what I learned while building the benchmark itself.

The biggest change: benchmarking is easier now because you can just talk to AI (Lesson 1)

A few years ago, even starting a benchmark felt heavy.

First you had to think through the methodology. Then you had to decide what to measure. Then you had to write the scripts. Then you had to figure out how to run them, collect the numbers, and make sense of the results.

Now a lot of that work is much easier.

You can literally start by describing what you want in plain English:

I have two services
they do the same job
I want to compare throughput, latency, and memory usage
I want a simple repeatable benchmark
I do not need a perfect academic setup
I just want something fair and useful

That is already enough to get moving.

AI is very good at helping with exactly this kind of task. Not because it magically solves benchmarking for you, but because it removes a lot of the friction around getting started.

It can help you:

define a reasonable benchmark scope
generate load scripts
suggest what metrics to collect
point out obvious mistakes in the setup
format results
help you explain the limitations clearly

That part feels very different from how things used to be.

Before, benchmarking often felt blocked by setup cost.

Now it is much more like: just talk to AI, get a first version working, then iterate.

That does not mean you should trust every output blindly. You still need to think. You still need to validate the setup. You still need to understand what is actually being measured.

But the barrier to entry is much lower now.

And I think that is a big deal.

Lesson 2: a benchmark does not need to be perfect to be useful

This was the biggest mindset shift.

I think many developers avoid benchmarking because they imagine they need a huge setup: many machines, a big test matrix, production traffic replay, deep statistical analysis, and charts for every possible scenario.

In reality, you can learn a lot from a small benchmark if you ask a clear question.

My question was simple:

If both tools are used as an LLM gateway in front of the same kind of workload, how do they behave in terms of throughput, latency, and memory usage?

That is already enough.

You do not need to model the entire internet. You just need a test that is fair enough to reveal something meaningful.

AI also helps here because it forces you to phrase the question clearly. If you cannot explain the benchmark clearly to an AI assistant, there is a good chance your scope is still too vague.

Lesson 3: benchmarking forces product clarity

This part surprised me.

I expected benchmarking to tell me about performance.

What it also did was clarify the product itself.

Once you measure something, you are forced to answer questions like:

What is this product actually optimized for?
Where should it be better?
What trade-offs did I make intentionally?
What should users care about most?

In my case, the benchmark made the positioning much clearer.

GoModel is not just “an AI gateway.”

It is a Go-based, open-source, single-binary gateway designed to be lightweight, simple to deploy, and efficient in the hot path of LLM requests.

Without benchmarking, those are just words.

With benchmarking, they become testable claims.

Lesson 4: benchmarking is also a debugging tool

Before doing this, I mostly thought about benchmarks as something you publish.

That was a mistake.

A benchmark is also one of the fastest ways to find weak spots in your own system.

As soon as you push something under repeatable load, you start noticing where memory grows faster than expected, where latency becomes uneven, and where parts of the system become bottlenecks.

Even if I had never published the results, building the benchmark would still have been worth it.

It gave me a much more honest picture of the system.

And again, AI helps here not by replacing the benchmark, but by helping you move faster once you find a problem. You can ask it to review the script, suggest what might be skewing the result, or help you isolate one part of the test.

My biggest takeaway

The biggest lesson I learned is very simple:

Benchmarking is much more accessible today with AI tools.

You do not need a lab.

You do not need a giant team.

You do not need a perfect methodology.

And now, you also do not need to start from a blank page.

You can just describe what you want to measure, use AI to help generate a first version, and improve it from there.

You still need to think.

You still need to validate the setup.

You still need to be honest about the limits.

But getting started is much easier than it used to be.

Final thought

If you are building infrastructure, developer tools, or performance-sensitive software, I think it is worth benchmarking earlier than you expect.

Not because you need a marketing graph.

Because benchmarking forces clarity.

It helps you understand your product better, find bottlenecks faster, and communicate value more concretely.

And today, with AI, it is easier than ever to start.

That was true for me with benchmarking GoModel, and it is probably true for a lot of other projects too.

If you want to check out the project, GoModel is open-source and available on GitHub:

GOModel on GitHub

There is also a benchmark result published there by enterpilot start-up:

GoModel vs LiteLLM benchmark (March 2026)