I've been building LLM applications for a while, and I kept hitting the same problem: every gateway I used was a bet.
Bet on LiteLLM? Great reach, but CVEs hit their API-key-verification path and admin endpoints in 2026.
Bet on Bifrost? Fast, but doesn't speak every provider.
Bet on Portkey? Managed, but do I want my keys leaving my network?
The problem is that most gateways fuse two very different jobs:
- Control plane - identity, guardrails, budgets, key vault, cost + audit
- Data plane - translating the OpenAI request shape into each provider's shape
When they're fused, a vulnerability in the translation code can reach your entire key vault. That's what happened with LiteLLM's 2026 CVEs - pre-auth SQL injection, admin file-read, RCE chain CVSS 9.9.
The Third Path
Agnos Proxy takes a different approach:
- Control plane + vault stay in your own infrastructure
- Translation engines are disposable, stateless adapters behind a hard boundary
- One provider key injected per request in-flight - engines store nothing
- Swap or combine engines - LiteLLM, Bifrost, Portkey, or built-in
- Per-provider routing - run several engines at once
Why This Matters
Security: A breached engine reaches at most one in-flight key, never the vault.
Flexibility: Use Bifrost for your high-volume OpenAI traffic, LiteLLM for niche enterprise models only it speaks. Same endpoint, same keys.
Future-proof: New gateway launches? Contain it. Prices change? Swap providers. Never rewrite your app.
Technical Details
- FastAPI control plane
- React dashboard (served at /app)
- Encrypted Fernet vault
- CEL guardrails + PII/secret detection
- Hierarchical budgets + rate-limits
- OpenTelemetry + Prometheus
- Helm chart, Docker, PyPI CLI
- Anti-coupling test enforces engine boundaries
Getting Started
siva010928
/
agnos-proxy-oss
Gateway-agnostic control plane for LLMs β an OpenAI-compatible governance proxy (auth, guardrails, budgets, routing, cost, observability) over a swappable engine: Bifrost / LiteLLM / Portkey / Direct. Own the control plane, swap the translator.
Agnos Proxy
The gateway-agnostic control plane for LLM routing, cost-tracking and observability.
Agnos Proxy is an OpenAI-compatible governance proxy that sits between your apps and the model
providers. Point any app at it (change one base_url, send one workspace key) and it inherits
centralized credential isolation, guardrails, budgets, rate-limits, routing/fallback, cost
attribution and full observability - for any provider, behind any translation engine.
Own the control plane. Swap the translator.
The governance boundary and the encrypted key vault stay in your own infrastructure. The provider-translation layer is a swappable, stateless commodity in a fixed slot: plug in Bifrost, LiteLLM, Portkey, the built-in Direct engine, or any OpenAI-compatible gateway - and swap them per provider, live, with one config change. We contain the engines we do not compete with them.
That is the point: the gateway market never stops moving. Prices change, a providerβ¦
curl -fsSL https://raw.githubusercontent.com/siva010928/agnos-proxy-oss/main/install.sh | sh
pipx install agnos-proxy-llm-gateway && agnos init
Top comments (0)