The Gateway-Agnostic LLM Control Plane - Why I Built Agnos Proxy

Siva Prakash K Kumar — Sat, 22 Aug 2026 16:57:34 +0000

I've been building LLM applications for a while, and I kept hitting the same problem: every gateway I used was a bet.

Bet on LiteLLM? Great reach, but CVEs hit their API-key-verification path and admin endpoints in 2026.

Bet on Bifrost? Fast, but doesn't speak every provider.

Bet on Portkey? Managed, but do I want my keys leaving my network?

The problem is that most gateways fuse two very different jobs:

Control plane - identity, guardrails, budgets, key vault, cost + audit
Data plane - translating the OpenAI request shape into each provider's shape

When they're fused, a vulnerability in the translation code can reach your entire key vault. That's what happened with LiteLLM's 2026 CVEs - pre-auth SQL injection, admin file-read, RCE chain CVSS 9.9.

The Third Path

Agnos Proxy takes a different approach:

Control plane + vault stay in your own infrastructure
Translation engines are disposable, stateless adapters behind a hard boundary
One provider key injected per request in-flight - engines store nothing
Swap or combine engines - LiteLLM, Bifrost, Portkey, or built-in
Per-provider routing - run several engines at once

Why This Matters

Security: A breached engine reaches at most one in-flight key, never the vault.

Flexibility: Use Bifrost for your high-volume OpenAI traffic, LiteLLM for niche enterprise models only it speaks. Same endpoint, same keys.

Future-proof: New gateway launches? Contain it. Prices change? Swap providers. Never rewrite your app.

Technical Details

FastAPI control plane
React dashboard (served at /app)
Encrypted Fernet vault
CEL guardrails + PII/secret detection
Hierarchical budgets + rate-limits
OpenTelemetry + Prometheus
Helm chart, Docker, PyPI CLI
Anti-coupling test enforces engine boundaries

Getting Started

siva010928 / agnos-proxy-oss

Gateway-agnostic control plane for LLMs — an OpenAI-compatible governance proxy (auth, guardrails, budgets, routing, cost, observability) over a swappable engine: Bifrost / LiteLLM / Portkey / Direct. Own the control plane, swap the translator.

Agnos Proxy

The gateway-agnostic control plane for LLM routing, cost-tracking and observability.

Agnos Proxy is an OpenAI-compatible governance proxy that sits between your apps and the model providers. Point any app at it (change one base_url, send one workspace key) and it inherits centralized credential isolation, guardrails, budgets, rate-limits, routing/fallback, cost attribution and full observability - for any provider, behind any translation engine.

Own the control plane. Swap the translator.

The governance boundary and the encrypted key vault stay in your own infrastructure. The provider-translation layer is a swappable, stateless commodity in a fixed slot: plug in Bifrost, LiteLLM, Portkey, the built-in Direct engine, or any OpenAI-compatible gateway - and swap them per provider, live, with one config change. We contain the engines we do not compete with them.

That is the point: the gateway market never stops moving. Prices change, a provider…

View on GitHub

agnos-llm-gateway.site

curl -fsSL https://raw.githubusercontent.com/siva010928/agnos-proxy-oss/main/install.sh | sh

pipx install agnos-proxy-llm-gateway && agnos init

DEV Community: Siva Prakash K Kumar