Gurveer

Posted on Jul 23

Introducing Aurora — The Fastest LLM Gateway for Your apps

#go #ai #opensource #webdev

Every LLM provider has a different API, different SDK, different auth. Managing keys and endpoints across projects is painful. We kept copy-pasting different base_url values and API keys into every app.

But here's the thing most developers don't realize: you shouldn't be doing this at all. This is not per-app logic — it's infrastructure. You need an intelligent gateway layer in your backend, and building it yourself is a waste of time.

So we built Aurora Gateway — an LLM gateway you integrate into your backend. Not just a proxy. Your apps use a single OpenAI-compatible SDK; Aurora handles routing, load balancing, failover, caching, guardrails, and cost tracking transparently.

The OSS edition (Apache 2.0) ships with production features out of the box. Enterprise adds SSO, RBAC, and tenant isolation when you scale.

What it does

Aurora sits in your backend — between your app code and LLM providers. Your app sends requests to Aurora using the standard OpenAI SDK. Aurora routes them to whichever provider you've configured, handles failure, caches responses, and tracks costs. Your app never deals with provider differences.

# Before: hardcoded provider
client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

# After: Aurora Gateway integrated into your backend
client = OpenAI(base_url="http://localhost:8080/v1", api_key="your-aurora-key")

That's it. No SDK changes. No format changes. Just point at Aurora and let it handle everything.

Why Aurora instead of building your own?

Every team building AI apps faces the same temptations:

A config file mapping provider names to base URLs
A retry utility function with exponential backoff
A simple cache layer to avoid duplicate API calls
A spreadsheet tracking costs across providers

These are all symptoms of the same missing piece: a proper gateway layer. Aurora is that layer — pre-built, optimized, and open-source. Don't build your own.

Edition comparison

Feature	OSS (Apache 2.0)	Enterprise
Auto-discovery, provider pools	✅	✅
Model aliases, overrides, fallback	✅	✅
Semantic, exact, prompt caching	✅	✅
Guardrails (PII, injection, regex, LLM-based)	✅	✅
Managed API keys, rate limiting	✅	✅
Token saver, usage analytics, audit logging	✅	✅
Admin dashboard, workflows	✅	✅
Prometheus metrics, Grafana	✅	✅
SSO / SAML / OIDC	—	✅
RBAC, tenant isolation	—	✅
Compliance workflows, budget enforcement	—	✅
Production support SLA	—	✅

Feature Overview

Routing & Providers

Auto-discovery — Set an API key as an environment variable. Restart Aurora. The provider and all its models appear automatically. No config files needed.

export OPENAI_API_KEY="sk-..."
export GROQ_API_KEY="gsk_..."
export ANTHROPIC_API_KEY="sk-ant-..."
aurora

Learn more: Providers Overview

Provider pools — Group multiple API keys or endpoints into a pool. Round-robin or weighted load balancing with health-aware failover.

# config.yaml
pools:
  production:
    providers:
      - openai/key-1
      - openai/key-2
      - groq/backup
    strategy: round-robin

Learn more: Provider Pools

Model aliases & overrides — Rename any model to a clean identifier across the entire gateway. Enable or disable specific models per user path.

Learn more: Model Management

Fallback & resilience — Automatic failover on 5xx/429 responses, or manual rules mapping failed provider+model to backups. Circuit breaker per provider (closed → open → half-open) with exponential backoff and jitter.

Learn more: Failover

Passthrough routes — /p/{provider}/* for full upstream API access beyond chat completions. Filter which provider types get passthrough routes.

Config-driven workflows — Per-request routing, caching, guardrail, audit, usage, budget, and fallback behavior controlled by persisted workflow documents.

Learn more: Workflows

API Surface

Aurora exposes multiple API formats so any client library works:

Endpoint	Purpose
`/v1/chat/completions`	OpenAI-compatible chat
`/v1/embeddings`	OpenAI-compatible embeddings
`/v1/rerank`	OpenAI-compatible reranking
`/v1/models`	Model listing
`/v1/files`	File management
`/v1/batches`	Batch processing
`/v1/responses`	Responses API with full CRUD
`/v1/messages`	Anthropic-compatible (native wire format)

Streaming SSE for all endpoints, preserved end-to-end.

Learn more: API Reference

Caching

Exact cache — SHA-256 hash match on request body, Redis-backed, async writes. Sub-millisecond hit detection for identical requests.

Semantic cache — Vector similarity with configurable threshold. Supports Qdrant, pgvector, Pinecone, and Weaviate as backends.

Prompt cache — Forwards cache_control markers to Anthropic, OpenAI, and Gemini for native prompt caching. Configurable modes (auto, manual, off), component toggles, and minimum token threshold.

Learn more: Cache · Prompt Caching

Security & Guardrails

PII redaction — email, phone, SSN, credit card detection and masking
Prompt injection blocking — detects and blocks injection attempts
System prompt protection — inject, override, or decorate system prompts
Regex blocking — custom pattern matching with block or sanitize actions
Length limits — character/token count enforcement on requests
LLM-based altering — rewrite message content via auxiliary LLM call
Guardrail direction & ordering — run before dispatch (input), after response (output), or both; same-order guardrails run in parallel
Batch guardrails — apply configured guardrails to inline batch items
Managed API keys — scoped, rate-limited, per-key model authorization with usage stats

Learn more: Guardrails

Observability

Audit logging — full request/response capture, buffered writes, configurable retention, live SSE stream for real-time monitoring
Usage analytics — per-model token counting, cost tracking, daily aggregation by model and user path
Prometheus metrics — aurora_requests_total, aurora_request_duration_seconds, aurora_requests_in_flight, plus detailed gateway phase timing
Admin dashboard — React SPA built into the Go binary: providers, pools, models, aliases, guardrails, cache, usage, audit, auth keys, workflows, playground
Structured logging — JSON/text format, configurable levels, source info
pprof endpoints — Go runtime profiling (heap, goroutine, mutex, block)

Learn more: Benchmarks

Cost Control

Token saver — policy-driven output compression with profiles (concise, caveman, ultra, wenyan). Scoped to specific models/providers. Can reduce token spend by 30-50%.
Pricing management — per-model pricing overrides, recalculation, import/export
Usage budgets — per-key usage tracking and limits, per-request budget enforcement via workflow feature flags

Learn more: Token Saver · Model Pricing

Developer Experience

Single binary — npm install -g iaurora or docker pull aurorahq/aurora. No dependencies, no runtime, no VM.
CLI — aurora init, aurora models sync/diff/show, aurora update
Swagger docs — interactive API reference at /swagger/index.html
Config profiles — pre-built configs for local, local-power, and team deployments
3-layer config — code defaults → config.yaml → env vars (env vars win)
Helm chart — deploy on Kubernetes
Docker Compose — full infrastructure stack: Redis, PostgreSQL, Qdrant, Prometheus, Grafana
Grafana dashboard — pre-configured panels for request rate, errors, latency, in-flight requests, per-model breakdown

Learn more: Quick Start

Benchmarks

17ms p50 latency, 41ms p99 routing latency
7K rps throughput on a single node
Built in native Go — single binary, optimized for minimum latency
55x faster than litellm python.

Install

npm install -g iaurora
mkdir my-gateway && cd my-gateway
aurora init
# Edit .env with your API keys
aurora

Or Docker:

docker run -d -p 8080:8080 \
  -e AURORA_MASTER_KEY="your-key" \
  -e OPENAI_API_KEY="sk-..." \
  aurorahq/aurora

Or Kubernetes with Helm:

helm install aurora ./helm \
  --namespace aurora --create-namespace \
  --set auth.masterKey="your-key" \
  --set providers.openai.apiKey="sk-..."

Aurora Gateway is open-source (Apache 2.0), self-hosted, and production-ready. Try it today:

npm install -g iaurora

Links: Docs · GitHub · Website · Discord

DEV Community