DEV Community

Kuldeep Paul
Kuldeep Paul

Posted on

Switching from LiteLLM to Bifrost: What to Know and How to Do It

#ai #architecture #llm #tutorial

LiteLLM earned its place in the early multi-provider LLM stack. It abstracted away API differences, simplified model switching, and made it easy to experiment across providers without rewriting integration code. But production workloads introduce a different set of demands. The gateway layer stops being a convenience and starts being load-bearing infrastructure. That shift changes what you need from it.

This guide covers the practical reasons teams outgrow LiteLLM, walks through a full migration to Bifrost, and addresses SDK compatibility, parallel deployment, and the features available after the switch.

Where LiteLLM Hits Its Ceiling

The issues below are not edge cases. They are repeatable pain points that surface as traffic scales up and agent architectures grow more complex.

  • Throughput limits tied to Python's runtime constraints. LiteLLM's Python foundation means it inherits GIL bottlenecks and async scheduling overhead. At 500 requests per second, P99 latency balloons to 90.72 seconds. Push past that and the service starts throwing out-of-memory errors
  • Memory management that requires manual intervention. LiteLLM's own deployment docs suggest setting max_requests_before_restart: 10000 to work around known memory leaks. In practice, teams report scheduling periodic restarts to keep the gateway stable
  • Logging infrastructure that degrades request performance. Once logging tables exceed 1M rows, API request latency climbs noticeably. At a rate of 100K requests per day, you cross that threshold in under two weeks, forcing teams to offload logs to cloud blob storage
  • Per-request overhead that compounds in agent chains. Every request adds roughly 500 microseconds of gateway-level latency. For an agent workflow chaining 10 sequential LLM calls, that stacks to 5ms of overhead before any provider processes a token
  • Operational burden of supporting infrastructure. Production LiteLLM deployments depend on the proxy server, PostgreSQL, and Redis. The community edition carries no uptime guarantee

Bifrost is designed around a different set of tradeoffs. Written in Go, it contributes only 11 microseconds of overhead per request while sustaining 5,000+ RPS. It ships as a single binary with no database requirements for core routing. Deployment takes 30 seconds with zero configuration. Enterprise capabilities like adaptive load balancing, semantic caching, guardrails, and a native MCP gateway are included from the start.

Head-to-Head Performance Numbers

All benchmarks were conducted on equivalent AWS t3.xlarge instances.

  • Gateway overhead at 500 RPS: 11 microseconds (Bifrost) vs. approximately 40 milliseconds (LiteLLM), making LiteLLM roughly 40x slower
  • P99 latency at 500 RPS: 1.68 seconds (Bifrost) vs. 90.72 seconds (LiteLLM), a 54x difference
  • Sustained throughput ceiling: Bifrost maintains stable performance above 5,000 RPS. LiteLLM becomes unreliable under equivalent load
  • Memory footprint: Bifrost consumes 68% less memory under the same traffic conditions
  • Container image size: 80 MB for Bifrost vs. 700+ MB for LiteLLM

Full methodology and raw data are available on the Bifrost migration benchmarks page.

Three-Step Migration Process

Expect to spend 15 to 30 minutes. Bifrost exposes an OpenAI-compatible API surface, so the only code change for most teams is swapping the base URL.

Step 1: Launch Bifrost

Pick whichever deployment path suits your stack. No config files needed to get started.

# NPX (quickest path)
npx -y @maximhq/bifrost

# Docker
docker run -p 8080:8080 maximhq/bifrost

# Docker with volume mount for persistent data
docker run -p 8080:8080 -v $(pwd)/data:/app/data maximhq/bifrost
Enter fullscreen mode Exit fullscreen mode

The gateway starts immediately. A built-in web dashboard for configuration and monitoring is available at localhost:8080.

Step 2: Add Provider Credentials

Register your LLM provider API keys through the dashboard or a config file.

  • Open http://localhost:8080 in your browser
  • Navigate to "Providers" in the sidebar
  • Enter API keys for OpenAI, Anthropic, AWS Bedrock, Google Vertex, or any of the 20+ supported providers
  • Set up model mappings and fallback sequences

Step 3: Swap the Base URL

One line changes. Your application code, SDK calls, and prompt logic stay exactly the same.

# LiteLLM configuration
client = openai.OpenAI(
    api_key="your-litellm-key",
    base_url="http://localhost:4000"
)

# Bifrost configuration
client = openai.OpenAI(
    api_key="your-bifrost-key",
    base_url="http://localhost:8080"
)
Enter fullscreen mode Exit fullscreen mode

Bifrost uses the provider/model naming convention (e.g., openai/gpt-4o) for explicit routing, so every request has clear provider attribution.

Drop-In SDK Support

Bifrost works with any framework that can target an OpenAI-compatible endpoint. The integration pattern is uniform: update the base URL, leave everything else unchanged.

  • OpenAI SDK: Set base_url to http://localhost:8080/openai
  • Anthropic SDK: Set base_url to http://localhost:8080/anthropic
  • Google GenAI SDK: Set api_endpoint to http://localhost:8080/genai
  • LangChain: Set openai_api_base to http://localhost:8080/langchain
  • LlamaIndex: Set api_base to http://localhost:8080/openai
  • Vercel AI SDK: Point the provider config to the Bifrost endpoint

Using the LiteLLM Python SDK with Bifrost

If you want to migrate infrastructure without touching application code, you can keep the LiteLLM SDK and route it through Bifrost as the backend.

import litellm

litellm.api_base = "http://localhost:8080/litellm"
response = litellm.completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)
Enter fullscreen mode Exit fullscreen mode

Bifrost's LiteLLM compatibility mode automatically translates text completion requests into chat format, so legacy calls continue working even with chat-only models.

Handling Specific Migration Scenarios

Virtual Key Migration

LiteLLM virtual keys for budget and access management have direct equivalents in Bifrost. You get the same team budgets, rate limits, and model restrictions, plus hierarchical budget management and MCP tool filtering on top.

curl -X POST http://localhost:8080/api/keys \
  -H "Content-Type: application/json" \
  -d '{
    "name": "team-engineering",
    "budget": 1000,
    "rate_limit": 100,
    "models": ["openai/gpt-4o", "anthropic/claude-sonnet-4-20250514"]
  }'
Enter fullscreen mode Exit fullscreen mode

Routing and Fallback Configuration

LiteLLM relies on model aliases and ordered fallback lists. Bifrost replaces this with provider/model routing, configurable fallback chains, weighted load balancing, and usage-based routing rules. All of this is manageable through the web dashboard or config files. Bifrost also includes adaptive load balancing that shifts traffic weights in real time based on provider health, latency, and throughput.

Zero-Downtime Parallel Operation

You do not need to cut over all at once. Run Bifrost alongside LiteLLM simultaneously and shift traffic incrementally. This approach lets you validate latency, reliability, and compatibility in production before fully decommissioning LiteLLM.

Capabilities Unlocked After Migration

Moving to Bifrost goes beyond raw speed improvements. It opens up infrastructure-level features that LiteLLM cannot provide.

  • Adaptive load balancing that redistributes traffic dynamically using real-time signals like provider success rates, response latency, and available capacity
  • Semantic caching that detects similar queries and returns cached results, cutting redundant provider calls and lowering spend
  • Native guardrails with out-of-the-box integrations for AWS Bedrock Guardrails, Azure Content Safety, Patronus AI, and GraySwan Cygnal
  • Built-in MCP gateway for centralized tool management, governance, and authentication across agentic workflows
  • Peer-to-peer clustering for high availability without leader election or coordination overhead
  • Prometheus metrics and OpenTelemetry natively embedded, no sidecars or third-party dependencies required
  • In-VPC deployment options on AWS, GCP, Azure, Cloudflare, and Vercel for regulated environments
  • Secrets management integration with HashiCorp Vault, AWS Secrets Manager, Google Secret Manager, and Azure Key Vault

Signals That It Is Time to Switch

Consider making the move if your team is experiencing any of these:

  • Production traffic has outgrown the performance ceiling of a Python-based gateway
  • Agent workflows with multiple chained LLM calls are accumulating noticeable gateway overhead
  • You need enterprise-grade budget management, access controls, and audit logging
  • Intermittent timeout spikes, memory leaks, or latency unpredictability are disrupting reliability
  • Compliance or safety requirements demand infrastructure-level guardrails
  • Cost optimization through caching and intelligent routing is a priority

Start the Migration

Bifrost is open source under the Apache 2.0 license. The migration process takes 15 minutes, involves changing a single line of code, and delivers measurable performance gains immediately.

npx -y @maximhq/bifrost
Enter fullscreen mode Exit fullscreen mode

For enterprise capabilities including adaptive load balancing, clustering, guardrails, the MCP gateway with federated authentication, vault integrations, and in-VPC deployments, book a demo to start a 14-day free trial of Bifrost Enterprise.

Detailed migration documentation and benchmark data are available at getmaxim.ai/bifrost/resources/migrating-from-litellm.

Top comments (0)