When Team Maxim started building Bifrost, they assumed load balancing for LLM requests would work like traditional API load balancing. Configure some fallback rules, set priority orders, maybe add weighted routing. Ship it.
That assumption lasted about three weeks into production.
The problem became obvious during the first major provider incident they experienced. OpenAI didn't just "go down." Instead, one region started timing out. Then another spiked with 5xx errors. Latency drifted upward across certain endpoints while others worked fine. By the time it became a full incident, half our traffic had already degraded for 20 minutes.
Our carefully configured fallback rules - rate limits, cost priorities, manual route ordering - sat there doing nothing useful. They were designed for clean failures: provider up or provider down. Production failures don't work that way.
maximhq
/
bifrost
Fastest LLM gateway (50x faster than LiteLLM) with adaptive load balancer, cluster mode, guardrails, 1000+ models support & <100 µs overhead at 5k RPS.
Bifrost
The fastest way to build AI applications that never go down
Bifrost is a high-performance AI gateway that unifies access to 15+ providers (OpenAI, Anthropic, AWS Bedrock, Google Vertex, and more) through a single OpenAI-compatible API. Deploy in seconds with zero configuration and get automatic failover, load balancing, semantic caching, and enterprise-grade features.
Quick Start
Go from zero to production-ready AI gateway in under a minute.
Step 1: Start Bifrost Gateway
# Install and run locally
npx -y @maximhq/bifrost
# Or use Docker
docker run -p 8080:8080 maximhq/bifrost
Step 2: Configure via Web UI
# Open the built-in web interface
open http://localhost:8080
Step 3: Make your first API call
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-4o-mini",
"messages": [{"role": "user", "content": "Hello, Bifrost!"}]
}'
That's it! Your AI gateway is running with a web interface for visual configuration, real-time monitoring…
How LLM Failures Actually Happen
Last year's major provider incidents followed a predictable pattern that static configurations can't handle:
Phase 1: Partial Brownout (Minutes 0-15)
- A specific region (us-east-1) starts showing elevated latency
- Success rate stays at 98% - not low enough to trigger most failover rules
- Only users routing through that region experience degradation
- Your monitoring shows "everything mostly fine"
Phase 2: Gradual Spread (Minutes 15-30)
- More regions show issues
- Different models degrade at different rates (GPT-4 fine, GPT-3.5 struggling)
- 5xx errors start appearing intermittently
- Some API keys hit rate limits faster than expected due to retries
Phase 3: Full Incident (Minutes 30+)
- Provider acknowledges incident publicly
- Traffic shifts to your fallback provider
- Fallback provider now experiences elevated load
- Your rate limits hit on the backup provider too
Static load balancing fails here because the failure mode evolves faster than you can update configurations. By the time you realize what's happening and adjust routes, the incident has progressed to a new phase.
The Adaptive Load Balancing Solution
The Team built Adaptive Load Balancing for Bifrost Enterprise to solve this specific problem: routing that learns from live traffic and adapts in real time to minimize damage during messy, gradual degradations.
The core design constraint was non-negotiable: it had to be fast enough to run on the hot path for every request. They got it down to under 10 microseconds of overhead per request. Today it's routing production LLM traffic for some of the largest companies in the world.
How It Actually Works
The system maintains a continuously updated score for each route (provider + model + region + API key combination). These scores are computed using exponentially weighted moving averages (EWMAs) of live traffic signals.
Rather than routing to the single "best" route, Bifrost selects from a top-candidate band and applies lightweight exploration. This prevents winner-takes-all patterns while still preferring high-performing routes.
The scoring algorithm combines five key signals:
1. Error and Timeout Penalties (With Fast Recovery)
When a route returns errors or times out, its score drops immediately. But unlike simple error-rate tracking, the penalty includes a recovery mechanism.
Brief incidents don't permanently damage a route's score. If errors stop, the route can earn back traffic quickly through the EWMA smoothing. This prevents the common failure mode where a 2-minute blip puts a route in the penalty box for an hour.
2. TACOS: Token-Adjusted Conformal Outlier Scoring
This is the component that handles latency-based routing intelligently.
Raw latency is a terrible signal for LLM routing decisions. A 500-token GPT-4 response taking 2.5 seconds isn't slower than a 50-token Claude response taking 800ms - it's actually faster per token.
TACOS (Token-Adjusted Conformal Outlier Scorer) solves this by:
- Normalizing latency by token count for every request
- Learning what "normal" latency looks like for each specific route
- Scoring based on deviation from that learned baseline, not absolute milliseconds
The conformal prediction approach means the system adapts gracefully as distributions shift. If a provider rolls out infrastructure changes that uniformly improve latency, TACOS adjusts the baseline rather than treating better performance as an anomaly.
Example: Route A normally does 15ms/token and suddenly spikes to 40ms/token - that's a strong signal of degradation. Route B normally does 35ms/token and is currently at 37ms/token - that's within normal variance.
TACOS catches these deviations minutes before they become obvious in aggregate metrics.
3. Utilization Shaping
The algorithm prevents overload by shaping traffic distribution based on current utilization.
If one API key is handling 80% of traffic while others sit idle, utilization shaping gradually redistributes load - even if that key is performing well. This prevents:
- Single key exhaustion during traffic spikes
- Rate limit cascades where one overloaded route triggers failover that overloads the next
- Winner-takes-all patterns that leave backup capacity unused
4. Momentum Boosts
When a degraded route recovers, it needs to earn traffic back to prove stability. But waiting too long means you're underutilizing recovered capacity while overloading other routes.
Momentum boosts solve this: routes that show consistent improvement get accelerated traffic increases rather than slow, linear recovery. A route that was at 10% allocation due to errors can jump to 40% within minutes if it demonstrates stable, fast performance.
This prevents the "penalty box" problem where routes sit underutilized for hours after brief incidents.
5. Starvation Guards and Exploration
Even healthy routes can end up underused if other routes maintain slightly better scores. But you want those routes in rotation so:
- You can detect if they degrade
- You maintain diverse traffic distribution for resilience
- You don't overfit to a single "winner" that might fail suddenly
The exploration mechanism keeps all healthy routes seeing some traffic. Starvation guards ensure no route drops to zero allocation unless it's actively failing.
Learning From Rate Limits
Beyond real-time scoring, the system learns from rate-limit events.
When a TPM (tokens per minute) or RPM (requests per minute) limit hits on a specific key or region, the algorithm:
- Records the limit as a constraint for that route
- Adjusts future traffic allocation to keep that route just under its limit
- Redistributes excess traffic to routes with available headroom
This is particularly important during incidents. When your primary provider degrades and traffic shifts to backups, those backups might hit rate limits they never reached before. The system learns these limits dynamically rather than requiring pre-configuration.
Automatic Fallback Assignment
When a route degrades beyond recovery thresholds, the system automatically assigns fallbacks:
- Same model from a different provider (GPT-4 from Azure instead of OpenAI)
- Different model if configured (Claude 3.5 Sonnet instead of GPT-4)
The key is that fallback selection considers the same scoring signals. If your primary GPT-4 route is degraded and your backup Azure GPT-4 route is also showing elevated latency, the system might route to Claude instead - even if your manual configuration specified Azure as the first fallback.
The goal: end users should never need to think about outages, brownouts, rate limits, or provider quirks. The infrastructure adapts automatically.
The Complete Optimization Loop
For every request, the load balancer continuously searches for the best tradeoff across:
- Reliability: Prefer routes with low error rates and stable latency
- Speed: Factor in per-token latency via TACOS
- Balanced utilization: Prevent overload on any single route
- Cost (optional): If configured, prefer cheaper providers when performance is equivalent
All of this happens in under 10 microseconds per request. The overhead is low enough that it adds less latency than a single network round trip within the same datacenter.
Why This Matters in Production
The difference between static and adaptive load balancing becomes obvious during incidents.
Static configuration scenario:
- Primary provider degrades gradually (15 minutes of elevated latency)
- Doesn't trip failover thresholds because success rate is still 95%
- All traffic continues routing to degraded provider
- Users experience slow responses
- Eventually trips threshold, all traffic shifts to backup
- Backup provider immediately overloaded
- You're manually updating configs while on a bridge call
Adaptive load balancing scenario:
- Primary provider degrades gradually
- TACOS detects latency deviation within 2 minutes
- Score drops, traffic automatically redistributes to other routes
- 60% of traffic shifted away before most users notice
- When provider fully degrades, remaining 40% shifts smoothly
- Backup routes already warmed up, no overload spike
- You're reading the incident post-mortem instead of fighting fires
The Technical Details
The algorithm uses exponentially weighted moving averages with configurable decay rates. Recent signals weigh more heavily than historical performance, but not so heavily that brief anomalies cause routing chaos.
The scoring function is purely mathematical - no ML inference, no external state lookups, no database queries. Just arithmetic on in-memory metrics. This is how they maintain sub-10-microsecond overhead.
Route scores update continuously as new request data arrives. The system doesn't batch updates or wait for time windows. Every completed request immediately influences future routing decisions.
What This Enables
Adaptive load balancing transforms how you operate LLM infrastructure:
During normal operations:
- Balanced utilization across all API keys prevents any single key from exhausting rate limits
- Cost optimization through intelligent provider selection
- Automatic capacity discovery as you add new routes
During degradations:
- Graceful degradation instead of cliff-edge failures
- Automatic traffic redistribution without manual intervention
- Faster recovery as incidents resolve
During incidents:
- Protection against cascade failures
- Rate limit awareness prevents backup overload
- Continuous optimization even as conditions change
Try It Yourself
Adaptive Load Balancing is available in Bifrost Enterprise. The open-source version includes standard load balancing; adaptive routing with TACOS and real-time learning requires an enterprise license.
Full technical documentation: https://docs.getbifrost.ai/enterprise/adaptive-load-balancing
For teams running production LLM infrastructure, the question isn't whether you need adaptive routing - it's whether you can afford the downtime and manual intervention that static configurations require.
GitHub: https://github.com/maximhq/bifrost
Documentation: https://docs.getbifrost.ai
Enterprise Contact: https://www.getmaxim.ai/contact
Top comments (0)