One API for every model, on a 2 GB VPS: meet Polyrail

Polyrail is an open-source, self-hosted AI gateway. One OpenAI-compatible endpoint in front of dozens of providers, with routing, failover, budgets, teams and observability, in a single Go binary.

If you build with LLMs, you probably know this week's problem. One provider is rate-limiting you, another just raised prices, a third has a model your teammate swears by. Your code now has three SDKs, three sets of keys in three places, and no idea who spent what.

Hosted routers like OpenRouter solve part of this. But sometimes you want the keys, the logs and the billing data on your own machine. That is the gap Polyrail fills.

What it is

Polyrail sits between your apps and your model providers. Your apps talk to one endpoint. Polyrail talks to everyone else.

• One API, three dialects. OpenAI, Anthropic Messages and Google GenAI request formats work on the same models. Point OpenAI SDKs, Codex, OpenCode, Cline or Open WebUI at /v1, Claude Code at /anthropic, and Gemini CLI at /genai.
• Every kind of model. Chat (streaming or not), embeddings, images, text to speech, speech to text, rerank, OCR, video jobs, files and batches.
• Many providers. OpenAI, Azure OpenAI, Anthropic, AWS Bedrock, Google Vertex AI and Gemini, Mistral, Groq, Cohere, DeepSeek, xAI, Ollama, vLLM and more, plus any OpenAI-compatible URL. The provider layer is the open-source Bifrost core, so Polyrail does not reinvent provider adapters.
• A built-in catalog. 265 models with prices and context lengths, generated from models.dev. Paste a provider key and its models light up.

Why you would run it

Routing that survives a bad afternoon

Give a model several targets and pick a strategy: weighted, priority, round-robin, lowest latency, lowest cost and more. If a target fails, Polyrail fails over to the next one, then to fallback models. Circuit breakers stop sending traffic to a target that keeps failing.

It also speaks OpenRouter-style request routing (models: [...], provider.order, max_price, :nitro, :floor) and saved presets, so existing habits carry over.

Key pools that look after themselves

Add many keys to one provider. Each key is tested, charted on its own, cooled down on rate limits, and quarantined after repeated authentication failures, with an alert when it happens.

Know what you spend, and cap it

Prices are per 1M tokens, including prompt-cache read and write prices. Every response carries its cost (usage.cost and an X-GW-Cost-Micros header). On top of that you get prepaid org credits, per-key monthly budgets, and budgets per org, workspace, member or key. An exact response cache, a semantic cache and idempotency keys keep repeated calls cheap.

Safe enough for a team

• Organizations, workspaces and roles (owner, admin, member, viewer).
• SSO with OIDC and SAML, verified domains, SCIM provisioning.
• API keys stored only as hashes, rotation with a grace period, and an audit log of every admin action.
• Guardrails: PII and credential masking, prompt-injection detection, regex filters that redact or block.
• Policies for allowed models and providers, zero-data-retention requirements and data regions, refused with a clear 403 reason.

See everything

Usage by model, key, member and workspace, request logs, CSV export, Prometheus metrics, JSON logs, OpenTelemetry trace export, signed webhooks, and alerts to Slack, Telegram, Discord, Teams and others.

There is also a console with a streaming playground and a Connect page that hands you the exact endpoint, key and config for each tool.

Light by design

Polyrail ships as one Go binary with an embedded database (Pebble). No Python runtime, no database server to babysit.

The README reports these measurements, taken on the development machine with a local mock provider, so they show gateway overhead and not model latency:

• Idle with the full catalog loaded: about 61 MB of memory and about 0.3% of one CPU core.
• 5,000 chat requests at concurrency 50 on a single core: about 7,700 requests per second, p50 5.3 ms, p99 18 ms, no errors, peak 69 MB.

The load generator shared the same CPU, and your numbers will differ. The method is in the repo.

That is why a 1 vCPU, 2 GB VPS is a supported target. When you outgrow it, the same binary runs as stateless replicas on Postgres (config), Valkey (rate limits) and ClickHouse (analytics).

Try it in two minutes

docker run -p 8080:8080 -v polyrail:/data \
  -e MASTER_KEY=$(openssl rand -hex 32) \
  -e ADMIN_EMAIL=[e-mail] \
  -e ADMIN_PASSWORD='choose-a-long-one' \
  ghcr.io/thuongh2/polyrail:edge

Open http://localhost:8080/app/, sign in, add a provider key under Admin → Providers, and create an API key. Then:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer sk-gw-..." \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Hello"}]}'

Any OpenAI SDK works by changing its base URL. Back up MASTER_KEY: it encrypts the provider keys stored in the database.

How it was built

Polyrail was built by an AI agent team working under written rules: a product owner, tech lead, developer, QA, security reviewer and an anti-slop reviewer, with tests first and verification before any claim of done. The rules and memory are in the repo (CLAUDE.md and .claude/agents/), so you can judge the process yourself. Every feature went through a security review, and the findings are in docs/SECURITY.md.

What it is not

Polyrail is not a hosted marketplace. You bring your own provider keys and run your own server. It is a young project, so expect rough edges, and read the docs before putting it in front of production traffic.

Get it

• Code and docs: github.com/thuongh2/polyrail
• Image: ghcr.io/thuongh2/polyrail
• License: Apache-2.0

If you try it, open an issue with what broke or what was missing. Stars help too.