It was a Tuesday in March when my Slack lit up.
A friend who runs platform engineering at a mid-sized fintech sent me one line: "Did you pin LiteLLM?" He hadn't. On March 24, 2026, two LiteLLM releases on PyPI (1.82.7 and 1.82.8) shipped with a credential-stealing payload. It harvested environment variables, SSH keys, cloud credentials, Kubernetes tokens, and database passwords, then sent them to an attacker-controlled domain. The packages had been uploaded directly to PyPI, bypassing the project's CI. LiteLLM's own post-mortem covers the details.
PyPI quarantined the versions fast. That wasn't the worrying part. The worrying part was where LiteLLM sits in most stacks: in front of every LLM provider key your company owns. An AI gateway is the one box that holds the keys to everything.
After that week, many teams started searching for a LiteLLM alternative. This post is the guide I wish my friend had then.
Disclosure: I'm the founder of GoModel, one of the tools below. I'll tell you where it wins and where another option is the better choice. If I'm not honest about that, this post is just an ad.
TL;DR
| If you need... | Pick |
|---|---|
| A self-hosted, lightweight, drop-in LiteLLM replacement | GoModel |
| An LLMOps platform with evals, A/B tests and fine-tuning loops | TensorZero |
| No infrastructure at all, and your data can go to a third-party cloud | OpenRouter |
What people actually use LiteLLM for
Before you swap anything, look at what LiteLLM does for you today. In almost every team I've talked to, it comes down to five jobs:
- One API for many providers. Your app speaks OpenAI format, and the gateway translates for Anthropic, Gemini, Bedrock, and the rest.
- Key management. Provider keys live in one place, not in every service.
- Cost tracking and budgets. Who spent $4,000 on GPT calls last weekend?
- Fallbacks and routing. When a provider has an outage, your product keeps working.
- Observability. Logs, metrics, traces.
A good LiteLLM alternative has to do all five. Ideally it should also be harder to compromise, and it shouldn't turn into another thing you have to babysit.
1. GoModel: the best LiteLLM alternative for self-hosting
Repo: github.com/ENTERPILOT/GoModel · License: MIT · Language: Go
I started building GoModel in late 2024 because I kept running into the same problems with LiteLLM in enterprise environments: a large Python dependency tree, heavy containers, and a lot of moving parts sitting in front of the most sensitive credentials in the company. March 2026 made those concerns very concrete.
Why it's my top pick
It ships as a single static Go binary. That's the main point, not a footnote. There's no pip install pulling in hundreds of transitive packages at deploy time, and no .pth file that can execute on interpreter start. You get one binary, a small (~17 MB) Docker image, and a dependency graph you can audit in an afternoon. The whole gateway is a smaller attack surface than the payload that hit LiteLLM users.
It's a real drop-in. GoModel exposes OpenAI-compatible and Anthropic-compatible APIs, so your existing SDK code usually only needs a new base_url:
docker run --rm -p 8080:8080 \
-e LOGGING_ENABLED=true \
-e OPENAI_API_KEY="your-openai-key" \
enterpilot/gomodel
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="your-gomodel-key")
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hello from GoModel"}],
)
GoModel detects which providers are available from the credentials you supply, so there's no long YAML file before your first request. Add ANTHROPIC_API_KEY or GEMINI_API_KEY and those models are available too.
Broad provider coverage: OpenAI, Anthropic, Google Gemini, Vertex AI, Azure OpenAI, Amazon Bedrock, DeepSeek, Groq, Fireworks, xAI, Alibaba Cloud, MiniMax, Oracle, OpenRouter, and self-hosted models through Ollama and vLLM, plus any OpenAI-compatible endpoint.
It covers the five jobs above:
- Cost and token tracking, with budget controls
- Exact and semantic caching
- A guardrails pipeline
- Prometheus metrics and OpenTelemetry (kept in the free MIT core, not paywalled)
- An admin dashboard and REST API
- A provider-native passthrough endpoint (
/p/{provider}/...) for when you need a feature the unified API doesn't expose yet
It works where cloud gateways can't. If you're in banking, healthcare, the public sector, or anywhere with an air-gapped network or strict EU data-residency rules, you need prompts and keys to stay on your own infrastructure. GoModel was designed for that from the start.
Where GoModel is not the right choice
- If you want a full experimentation and fine-tuning platform with built-in evals, see TensorZero below.
- If you don't want to run any infrastructure, see OpenRouter.
- GoModel is younger than LiteLLM. The core is solid and releases ship frequently, but if you depend on a niche LiteLLM integration, check the provider list first.
2. TensorZero: when you want an LLMOps platform, not just a gateway
Repo: github.com/tensorzero/tensorzero · License: Apache 2.0 · Language: Rust
TensorZero is a strong project, and I recommend it to people regularly. It's built for a different goal than GoModel.
GoModel sits between your app and your providers. TensorZero aims to close the loop: it stores every inference with the feedback you attach, and then helps you use that data to improve prompts and models. You get:
- A high-performance Rust gateway (they report <1 ms p99 overhead at 10k+ QPS)
- Observability backed by your own database
- Evaluations with heuristics and LLM judges
- Optimization workflows: supervised fine-tuning, RLHF, automated prompt engineering
- Adaptive A/B testing, routing, fallbacks and retries
Choose TensorZero if you have an ML team that wants to run structured experiments on prompts and models, and you're willing to adopt its way of doing things. You define "functions" and "variants" in config rather than just pointing your SDK at a new URL.
Think twice if you only need a gateway. Its data and optimization layer is its main value, and it's also more to run and learn. For most teams leaving LiteLLM, a simpler drop-in gets them back to shipping faster.
3. OpenRouter: when the cloud is fine
Site: openrouter.ai · Managed SaaS
Sometimes the honest answer is: don't run a gateway at all.
OpenRouter gives you one API key and one bill for a very large catalog of models across many providers. There's nothing to deploy, patch, or monitor. For a side project, a hackathon, an early-stage startup, or a team that wants to try twenty models this week, that's great.
The trade-offs:
- Your prompts and responses go through a third party. For many companies that's fine. For regulated industries, or anything covered by strict data-residency rules, it often isn't.
- Fees. OpenRouter charges a fee on credit purchases (around 5.5% at the time of writing). Bring-your-own-key usage is free up to a monthly allowance, with a small fee after that. At low volume this doesn't matter. At scale you'll notice it. Check their current pricing before committing.
- Less control over routing logic, caching, and guardrails than a self-hosted gateway gives you.
You don't have to choose one or the other. GoModel supports OpenRouter as an upstream provider, so you can self-host the control plane (keys, budgets, logs, guardrails) and still use OpenRouter's catalog for long-tail models.
Side-by-side
| GoModel | TensorZero | OpenRouter | LiteLLM (baseline) | |
|---|---|---|---|---|
| Deployment | Self-hosted | Self-hosted | Managed cloud | Self-hosted |
| Language | Go (single binary) | Rust | n/a | Python |
| License | MIT | Apache 2.0 | Proprietary | MIT core + enterprise |
| Drop-in OpenAI-compatible API | ✅ | ✅ | ✅ | ✅ |
| Anthropic-compatible API | ✅ | Partial | ✅ | ✅ |
| Cost tracking and budgets | ✅ | ✅ | ✅ (billing) | ✅ |
| Caching (exact + semantic) | ✅ | Limited | Provider-level | ✅ |
| Guardrails | ✅ | Via functions | Limited | ✅ |
| Evals / fine-tuning loop | ❌ | ✅ | ❌ | ❌ |
| Air-gapped / on-prem friendly | ✅ | ✅ | ❌ | ✅ |
| Supply-chain surface | Small | Small | Vendor's problem | Large (PyPI tree) |
| Time to first request | ~1 min | ~15–30 min | ~1 min | ~5 min |
Setup times are my rough experience, not formal benchmarks. Try them yourself.
How to choose (the human version)
Forget feature matrices for a minute and ask three questions:
1. Can my prompts leave my network?
If not, OpenRouter is out. Look at GoModel or TensorZero.
2. Do I need a gateway or an experimentation platform?
If you want your existing code to keep working and your security team to relax, choose GoModel. If you want to run your own fine-tuning and A/B testing loop, choose TensorZero.
3. Who's on call for this at 3 a.m.?
This question matters more than any benchmark. A single binary with few dependencies is easier to keep running than a stack with many services. Pick the tool your team can operate calmly, not the one with the longest feature list.
Migrating from LiteLLM to GoModel in 10 minutes
- Run GoModel next to LiteLLM using the Docker command above, with the same provider keys.
-
Set
GOMODEL_MASTER_KEYso only your services can call it. -
Point one non-critical service at
http://gomodel:8080/v1instead of your LiteLLM URL. - Compare responses, latency and cost in the GoModel dashboard for a day.
- Move the rest of your services over, then retire LiteLLM and its Python environment.
Because both expose OpenAI-compatible APIs, the change in most codebases is a single environment variable.
FAQ
What is the best LiteLLM alternative in 2026?
For self-hosted teams, GoModel: an MIT-licensed Go AI gateway with a single binary, OpenAI- and Anthropic-compatible APIs, cost tracking, caching, and guardrails. For an LLMOps platform with evals and fine-tuning, TensorZero. For a fully managed option, OpenRouter.
Is LiteLLM still safe to use?
The compromised versions (1.82.7 and 1.82.8) were removed and the LiteLLM team published a response. If you stay, pin exact versions, verify hashes, and scan your dependencies. Many teams moved anyway because they wanted a smaller dependency surface in front of their provider keys.
Is there an open-source LiteLLM alternative written in Go?
Yes. GoModel is MIT-licensed, written in Go, and ships as a single binary and a small Docker image.
Can I self-host an OpenRouter alternative?
Yes. GoModel and TensorZero are both self-hostable. GoModel can also use OpenRouter as one of its upstream providers.
Does GoModel support local models?
Yes, through Ollama, vLLM, or any OpenAI-compatible server.
Final thoughts
The LiteLLM incident wasn't really about one project. It showed how much trust we put in the component that holds all our AI keys, often without thinking about it much.
Whatever you choose, choose on purpose. Know what's in your dependency tree, where your prompts go, and who can fix it when something breaks.
If GoModel sounds like a fit, star it on GitHub, try the one-line Docker command, and open an issue if something's missing. I read every one.
Found a mistake in this comparison? Tell me in the comments and I'll fix it.
Top comments (0)