Anthony Max

Posted on Jul 2

Enterprise LLM Gateway: Route, govern, and secure your AI traffic

#webdev #ai #programming #opensource

Your company uses six different AI providers. OpenAI for ChatGPT, Anthropic for Claude and Groq for speed critical inference.

Each one has different API formats. Different authentication models. Different rate limits and costs. Different failure modes.

Your application code has to know about all of them. Your security team has to audit requests across all of them. Your finance team has to track costs across all of them. Your compliance team has to ensure governance across all of them.

Bifrost Gateway solves this by doing what HTTP gateways have done for decades: centralizing control. But for AI.

🔎 The Bifrost Gateway

At its core, Bifrost LLM gateway is remarkably simple: a unified API endpoint that sits between your applications and all your AI providers.

Instead of:

// Traditional: Point directly at each provider
const response = await openai.chat.completions.create({...});

You do:

// With Bifrost: Route through your gateway
const response = await fetch('https://bifrost.yourcompany.com/v1/chat/completions', {
  headers: { 'x-bf-vk': 'vk-prod-main' },
  body: JSON.stringify({model: 'gpt-4o', messages: [...]})
});

That's it. One endpoint. Everything else flows through the gateway.

But now you've connected something magical: your entire AI infrastructure becomes visible and controllable.

Unified API Across 23+ Providers and 1000+ models

This is where Bifrost's real power emerges. You configure multiple providers once in the gateway:

{
  "providers": [
    {
      "name": "openai",
      "api_key": "sk-...",
      "models": ["gpt-4o", "gpt-4o-mini", "o1"]
    },
    {
      "name": "anthropic",
      "api_key": "sk-ant-...",
      "models": ["claude-3-sonnet", "claude-3-opus"]
    }
  ]
}

Now your applications can request models by name, and Bifrost handles routing them to the right provider:

// Request GPT-4o → routes to OpenAI
// Request Claude-3-Sonnet → routes to Anthropic
// All through the same /v1/chat/completions endpoint

⚙️ Gateway Capabilities

1. Intelligent Routing & Failover

Bifrost doesn't just route requests, it implements resilient routing out of the box.

Weighted Load Balancing

You can distribute traffic across providers based on cost, performance, or capacity:

{
  "virtual_key": "vk-prod-main",
  "provider_configs": [
    {
      "provider": "openai",
      "allowed_models": ["gpt-4o", "gpt-4o-mini"],
      "weight": 0.5  // 50% of traffic
    },
    {
      "provider": "anthropic",
      "allowed_models": ["claude-3-sonnet"],
      "weight": 0.3  // 30% of traffic
    },
    {
      "provider": "groq",
      "allowed_models": ["mixtral"],
      "weight": 0.2  // 20% of traffic (for speed)
    }
  ]
}

Traffic distribution happens automatically. Your application code never knows it's load balanced.

Automatic Failover

When your primary provider fails, Bifrost automatically retries with the next provider:

Request for gpt-4o
    ↓
Try OpenAI (50% weight) Timeout
    ↓
Try Anthropic (30% weight)
    ↓
Application receives response
(Doesn't know failover happened)

This transforms your infrastructure from "system is down if OpenAI is down" to "system stays up even if primary provider is down."

Direct Provider Targeting

For use cases where you need a specific provider, you can bypass load balancing:

// Load balanced (uses weights)
{"model": "gpt-4o"}

// Direct to provider (bypasses load balancing)
{"model": "openai/gpt-4o"}

2. Governance Through Virtual Keys

Virtual Keys are the core governance mechanism in Bifrost. They define what each team, application, or user can access:

{
  "virtual_key": "vk-engineering-team",

  // Which models/providers?
  "allowed_models": ["gpt-4o", "gpt-4o-mini", "claude-3-sonnet"],
  "allowed_providers": ["openai", "anthropic"],

  // Budget controls
  "budget": {
    "monthly_spend_limit": "$5,000",
    "alert_thresholds": ["$4,000", "$4,500"],
    "rate_limit": "100 requests per minute"
  },

  // Which API keys?
  "allowed_api_keys": ["key-prod-001", "key-fallback-002"],

  // MCP tool filtering
  "allowed_mcp_tools": ["github"],
  "blocked_mcp_tools": ["file_system", "subprocess"],

  // Security & guardrails
  "guardrails": {
    "pii_detection": true,
    "secret_detection": true,
    "content_safety": "moderate"
  },

  // Expiration
  "expires_at": "2026-12-31"
}

Now when you create a new user, team, or customer, you don't configure API keys everywhere. You create a Virtual Key and assign it.

3. Cost Control & Budgeting

Enterprise AI spending is out of control. Bifrost gives you visibility and control:

Per-Request Cost Tracking

Every request flows through the gateway and gets tracked:

{
  "request_id": "req-abc-123",
  "virtual_key": "vk-eng-team",
  "model": "gpt-4o",
  "provider": "openai",
  "cost": 0.015,
  "tokens": {"input": 250, "output": 150},
  "latency_ms": 342,
  "status": "success"
}

Cost Optimization Strategies

With this visibility, you can implement sophisticated strategies:

Route to cheaper providers when they're available
Use smaller models for simple tasks (gpt-4o-mini instead of gpt-4o)
Cache responses for repeated queries
Batch process non-urgent requests during off-peak hours
Allocate budgets based on team priorities
Enable MCP Code Mode for tool orchestration, which reduces token usage by 50% compared to natural language tool invocation delivering significant token cost savings when managing complex multi-tool workflows

4. Security & Compliance

Bifrost implements enterprise-grade security at the gateway level:

Guardrails

Bifrost supports popular guardrails such as Grey Swan, Patronus AI, Azure, and many others.

Real-time detection and blocking of:

Secret Detection: API keys, passwords, tokens
Custom Rules: Domain-specific policies you define

{
  "guardrails": {
    "pii_detection": {
      "enabled": true,
      "action": "block",
      "types": ["credit_card", "email"]
    },
    "secret_detection": {
      "enabled": true,
      "action": "block"
    },
    "content_safety": {
      "provider": "anthropic",
      "threshold": "medium"
    }
  }
}

Immutable Audit Logs

Every request is logged:

Who made it (Virtual Key, user ID)
What was requested (model, prompt)
Where it went (which provider)
What happened (success/failure, cost)
When it happened (timestamp)

5. Observability & Monitoring

You can't govern what you can't see. Bifrost provides complete observability:

Built-in Dashboards

The Bifrost console shows:

Real-time request volume and latency
Cost trends and budget utilization
Provider health and failover rates
Error rates and patterns
Top models and usage patterns

Prometheus Metrics

Native Prometheus integration for your monitoring stack:

bifrost_requests_total{provider="openai",model="gpt-4o",status="success"} 15234
bifrost_request_duration_seconds{provider="anthropic",quantile="0.95"} 0.342
bifrost_cost_dollars{virtual_key="vk-eng-team"} 1523.45
bifrost_budget_remaining_dollars{virtual_key="vk-eng-team"} 3476.55
bifrost_failover_count{provider="openai"} 23

6. Semantic Caching

Not all AI requests are the same. Similar questions often get similar answers. Bifrost's semantic cache recognizes this:

// Request 1
"Explain machine learning in simple terms"
// → Hits OpenAI, costs $0.03

// Request 2 (semantically similar)
"What is machine learning? Explain simply."
// → Hits cache (semantic match), costs $0.00

How it works:

Request comes in
Bifrost computes semantic embedding
Checks cache for semantically similar previous responses
If found and confidence > threshold, returns cached response
If not found, routes to provider and caches result

This typically reduces costs 15-25% without any application changes.

💻 Extended with Bifrost Edge

Bifrost Gateway centralizes governance in your infrastructure. But what about AI traffic on employee machines? Claude Desktop, ChatGPT apps, Cursor, browser-based AI?

This is where Bifrost Edge enters the picture.

The Gateway + Edge Architecture

Edge extends Gateway governance to the endpoint:

Gateway: Centralizes control for infrastructure-routed traffic
Edge: Enforces that same control on endpoint-routed traffic

The governance policies you define in the Gateway automatically apply to all traffic through Edge. No separate configuration needed.

What Edge Adds to Gateway

Gateway alone controls:

Your backend AI services
API integrations
Batch processing jobs

Gateway + Edge controls:

✅ + Desktop AI applications
✅ + Browser-based AI (ChatGPT, etc.)
✅ + IDE coding agents
✅ + MCP servers and tools
✅ + Employee machine AI traffic

Edge is the enforcement layer that makes Gateway governance truly comprehensive.

⚙️ Example: An Enterprise AI Stack

Let's walk through how a real organization uses Bifrost Gateway + Edge together:

Setup

Company: 500 employees, $100,000/month AI budget

Requirements:

Use multiple providers for cost optimization
Prevent data leaks (secrets)
Track spending by team
Ensure compliance (audit logs, access control)
Support employee AI tools without losing control

Configuration

Step 1: Configure Gateway Providers

{
  "providers": [
    {"name": "openai", "api_key": "...", "models": ["gpt-4o", "gpt-4o-mini"]},
    {"name": "anthropic", "api_key": "...", "models": ["claude-3-sonnet", "claude-3-opus"]},
    {"name": "groq", "api_key": "...", "models": ["mixtral"]}
  ]
}

Step 2: Create Virtual Keys for Each Team

{
  "virtual_keys": [
    {
      "name": "vk-engineering",
      "allowed_models": ["gpt-4o", "gpt-4o-mini", "claude-3-sonnet", "claude-3-opus"],
      "allowed_providers": ["openai", "anthropic"],
      "budget": {"monthly": "$50,000"},
      "allowed_mcp_tools": ["github"],
      "blocked_mcp_tools": ["file_system", "subprocess"]
    },
    {
      "name": "vk-product",
      "allowed_models": ["gpt-4o-mini", "claude-3-sonnet"],
      "allowed_providers": ["openai", "anthropic"],
      "budget": {"monthly": "$20,000"},
      "allowed_mcp_tools": ["notion"]
    },
    {
      "name": "vk-research",
      "allowed_models": ["*"],
      "allowed_providers": ["*"],
      "budget": {"monthly": "$100,000"},
      "allowed_mcp_tools": ["*"]
    }
  ]
}

Step 3: Deploy Bifrost Edge to All Machines

One browser sign in per employee. Edge takes care of the rest.

What Happens Next

Results After 90 Days:

Metric	Before	After	Change
Monthly AI Spend	$100,000	$130,000	-35%
Cost per Request	N/A	$0.012 avg	40% lower
Compliance Audit Time	40 hours	2 hours	95% faster
Shadow AI Tools	Unknown	47 identified	Full visibility
Provider Failovers	Never	12 successful	100% uptime

👀 Why This Matters

Traditional enterprise architecture treats AI as a service. You point your application at an API and hope for the best.

Bifrost treats AI as infrastructure. Like your database, cache, or message queue. With centralized control, monitoring, and governance.

This shift is critical because AI is no longer experimental. It's production-critical. Your SLA depends on it. Your compliance depends on it. Your costs depend on it.

✅ Getting Started

Self-Hosted

Download Bifrost, deploy to your infrastructure, configure providers and Virtual Keys:

npx @maximhq/bifrost -p 5000

# Configure your providers
# Point your applications at localhost:5000

Managed Service

Use Bifrost's managed service for zero ops overhead.

With Bifrost Edge

For complete endpoint control:

Deploy Bifrost Gateway in your infrastructure
Register for Bifrost Edge alpha
Deploy Edge to your fleet
One sign-in per employee
Complete governance everywhere

🖋️ Conclusion

Bifrost Edge extends that control all the way to the endpoint. Together, they don't just give you visibility into your AI spending and security posture, they give you complete governance from the data center to the desktop. This is no longer a "nice to have" for enterprises scaling AI. It's becoming table stakes. The ones that don't will still be pointing applications directly at provider APIs, manually managing failovers, running blind on costs, and hoping compliance audits don't find what they can't see.