DEV Community: Pranay Batta

MCP at Scale: Access Control, Cost Governance, and 92% Lower Token Costs

Pranay Batta — Tue, 14 Apr 2026 08:44:38 +0000

The Hidden Tax on Every MCP Request

Here is something nobody talks about when they demo MCP integrations: token costs at scale.

I have been running MCP setups with increasing numbers of connected servers. The pattern is always the same. You connect a few servers, everything works brilliantly. You connect a dozen, costs start climbing. You connect sixteen servers with 500+ tools, and suddenly your token budget is gone before the model even starts thinking about your actual query.

Why? Every tool definition from every connected server gets injected into the model's context on every single request. 150+ tool definitions can consume the majority of your token budget. And there is zero access control. Any consumer can call any tool. No cost tracking at tool level.

This is unsustainable for production deployments.

I Tested Bifrost's Code Mode Approach

Bifrost takes a fundamentally different approach to this problem. Instead of dumping all tool definitions into the context window, it exposes a virtual filesystem of Python stub files. The model discovers tools on-demand through four meta-tools:

listToolFiles - discover available servers and tools
readToolFile - load specific function signatures
getToolDocs - fetch detailed documentation only when needed
executeToolCode - run scripts in a sandboxed Starlark interpreter

The key insight: the model only loads what it actually needs for the current query. If you ask it to read a file, it does not need to know about your Slack, GitHub, Jira, and database tools all at once.

Here is what a typical tool discovery flow looks like:

# Model calls listToolFiles to see available servers
available = listToolFiles()
# Returns: ["filesystem/", "github/", "slack/", "jira/", ...]

# Model identifies it needs filesystem tools for this query
tools = readToolFile("filesystem/read.py")
# Returns only the function signature for filesystem_read

# Model fetches docs only if needed
docs = getToolDocs("filesystem", "read")

# Executes with full sandboxing
result = executeToolCode("filesystem/read.py", {"path": "/src/main.go"})

This is lazy loading for LLM tool contexts. Simple idea. Massive impact.

Benchmark Results: 3 Controlled Rounds

I ran three controlled rounds, scaling from 6 servers to 16 servers. Every round maintained a 100% task pass rate. The model completed every task correctly while using dramatically fewer tokens.

Round	Tools	Servers	Token Reduction	Cost Savings
1	96	6	58.2%	55.7%
2	251	11	84.5%	83.4%
3	508	16	92.8%	92.2%

At roughly 500 tools, Code Mode reduces per-query token usage by about 14x. From 1.15M tokens down to 83K. That is not an incremental improvement. That is a different cost structure entirely.

The savings compound non-linearly. As you add more tools, the percentage saved increases because Code Mode's overhead stays roughly constant while traditional mode scales linearly with tool count.

For full benchmark methodology, check the benchmarking docs.

Access Control That Actually Works

Token savings are great, but production MCP deployments need governance. Bifrost handles this through two mechanisms.

Virtual Keys let you create scoped credentials per user, team, or customer. You can scope at the tool level:

virtual_key:
  name: "data-team-key"
  allowed_tools:
    - database_read
    - database_query
  blocked_tools:
    - database_delete
    - filesystem_write

Allow filesystem_read, block filesystem_write. Allow database_query, block database_delete. Fine-grained, declarative, no code changes needed.

MCP Tool Groups are named collections of tools from multiple servers. You create a group, attach it to keys, teams, or users. No database queries at resolve time. This is important when you are running at 5000 RPS and cannot afford lookup latency.

Per-Tool Observability

Every tool execution gets logged with:

Tool name and server source
Arguments passed and results returned
Execution latency
Virtual key that initiated the call
Parent LLM request context

You can track cost at the tool level alongside LLM token costs. This matters when your finance team asks why the AI bill doubled last month. You can point to exactly which tools, which teams, and which queries drove the spend.

Budget and limits let you set spending caps per virtual key, so no single team can blow through the monthly allocation.

Connection Flexibility

Bifrost supports four MCP connection types: STDIO, HTTP, SSE, and in-process via the Go SDK. OAuth 2.0 with PKCE and automatic token refresh is built in. Health monitoring with automatic reconnects keeps things running without manual intervention.

You can run it in manual approval mode where a human reviews tool calls, or in autonomous agent loop mode where the model chains tool calls independently.

For Claude Code and Cursor users, the /mcp endpoint integrates directly. Setup takes minutes.

Honest Trade-offs

No tool is perfect. Here is what I noticed:

Learning curve for Code Mode. The virtual filesystem abstraction is elegant, but it is a new mental model. Teams used to traditional MCP tool injection will need to understand why their tools are now "files" the model reads on demand.

Meta-tool overhead on simple queries. If you only have 10-20 tools, the overhead of the four meta-tools (listToolFiles, readToolFile, etc.) might not save you much. The real wins kick in above 50-100 tools. Below that threshold, traditional mode works fine.

Starlark sandbox limitations. The sandboxed Starlark interpreter is secure by design, but it means tool code runs in a restricted environment. Complex tool implementations may need adjustments.

Dependency on gateway availability. Adding a gateway layer means one more component to monitor. Bifrost's 11 microsecond latency and Go-based architecture make this a non-issue in practice, but it is still an additional piece of infrastructure.

Who Should Care

If you are running fewer than 50 MCP tools, you probably do not need Code Mode yet. Traditional tool injection works fine at that scale.

If you are running 100+ tools across multiple servers, or if you need per-team access control, or if your CFO is asking questions about AI infrastructure costs, this is worth evaluating.

The 92% cost reduction at 500+ tools is the headline number, but the governance features (virtual keys, tool groups, audit logging) are what make it production-ready.

Try It

Bifrost is open-source and written in Go.

GitHub repo - star it if this is useful
MCP documentation - full setup guide
Governance docs - virtual keys, tool groups, budgets
Getting started - up and running in minutes

I have been testing a lot of MCP tooling lately. Bifrost's approach to the context window problem is the most practical solution I have seen. The lazy loading pattern for tool definitions should honestly be how all MCP gateways work.

Check the docs and give it a spin. Happy to discuss benchmarks or setup in the comments.

How to Track LLM Costs and Rate Limits on AWS Bedrock with an AI Gateway

Pranay Batta — Mon, 13 Apr 2026 10:21:38 +0000

Running LLM workloads on AWS is easy. Knowing what they cost is not. You spin up Bedrock, call Claude or Mistral a few thousand times, and the bill shows up three days later as a single line item. No breakdown by team. No per-model cost tracking. No rate limits unless you build them yourself.

I spent the last two weeks evaluating how teams can get proper cost governance over LLM usage on AWS. Native tools, third-party gateways, open-source options. Here is what I found.

The Problem with AWS Native Cost Tracking

AWS gives you CloudWatch and Cost Explorer. Both are built for general AWS resource monitoring. They work fine for EC2, Lambda, S3. For LLM workloads on Bedrock, they fall short.

What you get from CloudWatch + Cost Explorer:

Aggregate Bedrock spend per region
Invocation counts at the service level
Basic alarms on total spend thresholds

What you do not get:

Per-model token-level cost breakdowns
Team or project-level budget enforcement
Rate limiting by user, team, or API key
Real-time cost tracking per request
Automatic routing away from providers that exceed limits

If you are running one model for one team, native tools are fine. The moment you have multiple teams, multiple models, or need to enforce granular budgets, you are building custom infrastructure.

The Gateway Approach

An LLM gateway sits between your application and Bedrock. Every request passes through it. That gives you a single place to track costs, enforce rate limits, and control routing.

I tested three approaches:

Feature	AWS Native (CloudWatch + Cost Explorer)	LiteLLM	Bifrost
LLM-specific cost tracking	Aggregate only	Per-request, per-model	Per-request, per-model
Budget hierarchy	Account-level billing alerts	Basic budget controls	4-tier: Customer > Team > Virtual Key > Provider
Rate limiting	No native LLM rate limits	Basic rate limiting	VK + Provider Config level, token and request limits
Reset durations	N/A	Limited options	1m, 5m, 1h, 1d, 1w, 1M, 1Y (calendar-aligned UTC)
Bedrock support	Native	Yes	Yes (provider type "bedrock")
Overhead	None	~8ms (Python)	11 microseconds (Go)
Deployment	N/A	Self-hosted or cloud	Self-hosted (runs in your VPC)
Language	N/A	Python	Go

The numbers tell the story. For teams that need real LLM cost governance on AWS, a dedicated gateway is the right call.

Setting Up Bifrost with AWS Bedrock

Bifrost runs in your VPC alongside Bedrock. No data leaves your infrastructure. That matters for teams with compliance requirements.

Start the gateway:

npx -y @maximhq/bifrost

Full setup guide here.

Configure Bedrock as a provider:

accounts:
  - id: "ml-team"
    providers:
      - id: "bedrock-claude"
        type: "bedrock"
        region: "us-east-1"
        model: "anthropic.claude-sonnet-4-20250514-v1:0"
        weight: 80
      - id: "bedrock-mistral"
        type: "bedrock"
        region: "us-west-2"
        model: "mistral.mistral-large-2407-v1:0"
        weight: 20

Weighted routing across models. 80% of requests go to Claude Sonnet on Bedrock, 20% to Mistral. Both running through your AWS account. The provider configuration docs cover all Bedrock model formats and region options.

Four-Tier Budget Hierarchy

This is where Bifrost separates itself from everything else I tested. The budget system has four levels: Customer, Team, Virtual Key, and Provider Config. All four must pass for a request to go through.

budgets:
  customer:
    - id: "acme-corp"
      limit: 5000
      period: "1M"

  team:
    - id: "ml-engineering"
      customer_id: "acme-corp"
      limit: 2000
      period: "1M"

  virtual_key:
    - id: "staging-key"
      team_id: "ml-engineering"
      limit: 500
      period: "1w"

  provider_config:
    - id: "bedrock-claude"
      limit: 1000
      period: "1M"

Customer gets $5,000/month. ML Engineering team gets $2,000 of that. The staging key is capped at $500/week. And the Bedrock Claude provider itself is capped at $1,000/month. If any tier hits its limit, the request is blocked.

Cost is calculated from provider pricing, token usage, request type, cache status, and batch operations. Not estimated. Calculated from actual usage data.

The governance docs have the full breakdown.

Rate Limiting That Actually Works for LLMs

AWS does not give you LLM-specific rate limits. Bedrock has service quotas, but those are blunt instruments. You cannot limit a specific team to 100 requests per minute or cap token consumption per API key.

Bifrost handles rate limiting at two levels: Virtual Key and Provider Config. You can set both request limits (calls per duration) and token limits (tokens per duration).

rate_limits:
  virtual_key:
    - id: "staging-key"
      requests:
        limit: 100
        duration: "1h"
      tokens:
        limit: 50000
        duration: "1h"

  provider_config:
    - id: "bedrock-claude"
      requests:
        limit: 500
        duration: "1h"

Reset durations: 1m, 5m, 1h, 1d, 1w, 1M, 1Y. The daily, weekly, monthly, and yearly resets are calendar-aligned in UTC. So "1d" resets at midnight UTC, not 24 hours from first request.

Here is the clever part: if a provider config exceeds its rate limit, that provider gets excluded from routing. But other providers in the account remain available. Traffic shifts automatically. No downtime, no manual intervention.

Observability at Sub-Millisecond Overhead

Every request through Bifrost is captured: tokens used, latency, cost, response status. The observability layer adds less than 0.1ms of overhead. Storage backend is SQLite or PostgreSQL.

What makes this useful for AWS teams:

14+ API filter options for querying logs. Filter by model, provider, team, cost range, status code, time window.
WebSocket live updates. Watch requests flow through in real time. Useful during load testing or incident debugging.
Single pane across providers. If you are running Bedrock plus OpenAI or Gemini as failover, all logs are in one place.

Compare that to checking CloudWatch for Bedrock, then the OpenAI dashboard for your fallback, then manually correlating timestamps. The centralised view saves real time.

Honest Trade-offs

No tool solves everything. Here is what to know:

Bifrost is self-hosted only. You run it, you maintain it. For teams already on AWS with VPC infrastructure, this is straightforward. For smaller teams without DevOps, it is extra work.

LiteLLM has broader provider coverage. 100+ providers out of the box. If you need niche providers, LiteLLM may have them. Bifrost focuses on major providers but adds the Go performance advantage and deeper governance features.

AWS native tools have zero overhead. If all you need is aggregate cost visibility and basic billing alerts, CloudWatch is already there. No extra infrastructure.

Go vs Python matters at scale. Bifrost's 11 microsecond overhead versus LiteLLM's ~8ms becomes significant when you are processing thousands of requests per minute. At low volume, both are fine. At scale, the difference compounds. The benchmarks back this up: 5,000 RPS on a single instance.

Bifrost is a newer project. The community is growing but smaller than LiteLLM's. Documentation is solid. Edge cases may require checking GitHub issues.

When to Use What

Stick with AWS native tools if: You have one team, one model, and just need billing alerts.

Consider LiteLLM if: You need maximum provider coverage and are comfortable with Python-based overhead.

Use Bifrost if: You need granular cost governance, multi-tier budgets, LLM-specific rate limiting, and minimal latency on AWS. Especially if you are already running in a VPC and want semantic caching and automatic failover alongside cost controls.

Quick Start

# 1. Start Bifrost in your VPC
npx -y @maximhq/bifrost

# 2. Configure Bedrock providers in bifrost.yaml

# 3. Set budget and rate limit tiers

# 4. Point your application at the gateway
export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
export ANTHROPIC_API_KEY=your-bifrost-virtual-key

Every Bedrock request now has cost tracking, rate limiting, and observability built in.

GitHub | Docs | Website

AWS makes it easy to run LLM workloads. It does not make it easy to govern them. If your team is scaling Bedrock usage and needs real cost controls, a dedicated LLM gateway fills the gap that CloudWatch and Cost Explorer leave open.

Check the repo if you want to dig into the source.

Best Claude Code Gateway for Multi-Model Routing

Pranay Batta — Fri, 10 Apr 2026 22:29:45 +0000

Claude Code is great until you need more than one model. You hit a rate limit on Anthropic, want Gemini for long context, or need GPT-4o for a specific task. The default setup gives you no way to route across providers.

I spent a week testing gateways that sit between Claude Code and LLM providers. The goal was simple: configure multiple models, set routing weights, get automatic failover, and keep Claude Code working normally.

Bifrost was the clear winner. Open-source, written in Go, 11 microsecond overhead per request. Here is how I set up multi-model routing and what I learned.

Why Multi-Model Routing Matters

Different models are good at different things. Claude Sonnet handles tool use well. GPT-4o is strong at certain code generation tasks. Gemini 2.5 Pro handles massive context windows. Using one model for everything means you are leaving performance on the table.

Multi-model routing lets you:

Split traffic across providers by weight
Fail over automatically when a provider goes down
Pin specific models for specific tasks
Control costs by routing cheaper models for simpler operations

The problem: Claude Code talks to api.anthropic.com by default. No native multi-model support. You need a gateway.

The Setup: Bifrost as a Claude Code Gateway

Bifrost exposes an Anthropic-compatible endpoint. Claude Code does not know a gateway exists. It sends standard requests, and Bifrost translates and routes them to whatever provider you configure.

Full Claude Code integration docs here.

Install and Connect

npx -y @maximhq/bifrost

That starts the gateway locally. Setup guide has the details.

Point Claude Code at Bifrost:

export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
export ANTHROPIC_API_KEY=your-bifrost-virtual-key

The ANTHROPIC_API_KEY here is a Bifrost virtual key, not your actual Anthropic key. Provider keys live in the Bifrost config. This is a drop-in replacement for the Anthropic API.

Done. Every Claude Code request now flows through Bifrost.

Weighted Routing Configuration

This is the core of multi-model routing. You assign weights to providers, and Bifrost distributes traffic accordingly. Weights auto-normalize to sum 1.0, so you can use any numbers.

Here is a config that splits traffic between GPT-4o and Claude Sonnet:

accounts:
  - id: "dev-team"
    providers:
      - id: "openai-primary"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        weight: 70
      - id: "anthropic-secondary"
        type: "anthropic"
        api_key: "${ANTHROPIC_API_KEY}"
        model: "claude-sonnet-4-20250514"
        weight: 30

70% of requests go to GPT-4o. 30% to Claude Sonnet. I used this to compare output quality across providers in real coding sessions without manually switching anything.

The routing docs cover all the configuration options.

Important detail: cross-provider routing does not happen automatically. You must explicitly configure each provider in your config. Bifrost does not guess or infer routing rules.

Automatic Failover

Weighted routing is useful. Automatic failover is essential. Providers go down. Rate limits hit. You do not want your Claude Code session to break mid-task.

Bifrost sorts providers by weight and retries on failure. If the primary provider fails, the next one picks up the request.

accounts:
  - id: "dev-team"
    providers:
      - id: "openai-primary"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        weight: 80
      - id: "gemini-fallback"
        type: "gemini"
        api_key: "${GEMINI_API_KEY}"
        model: "gemini-2.5-pro"
        weight: 15
      - id: "anthropic-fallback"
        type: "anthropic"
        api_key: "${ANTHROPIC_API_KEY}"
        model: "claude-sonnet-4-20250514"
        weight: 5

OpenAI goes down, Bifrost retries with Gemini. Gemini fails, falls back to Anthropic. My coding session never interrupts.

Model Pinning for Bedrock and Vertex AI

If your team uses AWS Bedrock or Google Vertex AI, you can pin specific models directly:

# Bedrock
export ANTHROPIC_DEFAULT_SONNET_MODEL="bedrock/global.anthropic.claude-sonnet-4-6"

# Vertex AI
export ANTHROPIC_DEFAULT_SONNET_MODEL="vertex/claude-sonnet-4-6"

You can also override the model mid-session using the --model flag or the /model command inside Claude Code. Useful when you want to switch between models for different parts of a task. Start with Sonnet for scaffolding, switch to GPT-4o for a tricky implementation, then back again. The gateway handles the translation layer for each provider.

This is one area where the Anthropic SDK compatibility matters. Bifrost maintains full compatibility with the Anthropic message format, so model pinning and switching work without any client-side changes.

Provider configuration docs list all supported providers and model formats.

Budget Controls Across Providers

Once all traffic flows through one gateway, cost management becomes straightforward. Bifrost has a four-tier budget hierarchy: Customer, Team, Virtual Key, Provider Config.

budgets:
  - level: "virtual_key"
    id: "claude-code-dev"
    limit: 200
    period: "monthly"

Set a limit. When it is reached, requests get blocked. No surprise bills from a runaway Claude Code session.

The full governance layer handles rate limiting, access control, and spend management across all configured providers.

Observability Across All Providers

Every request through Bifrost gets logged: latency, token count, cost, provider used, response status. The observability layer gives you a single view across all providers.

This is particularly useful with multi-model routing. You can see exactly which provider handled each request, compare response times across models, and track per-provider costs. When I was running 70/30 weighted routing between GPT-4o and Claude Sonnet, the observability data showed me exactly how each model performed on real coding tasks. Response times, token consumption, and cost per request, all in one place.

Without centralized logging, you are checking multiple provider dashboards and guessing which model handled what. That is not sustainable when you are running multiple providers through Claude Code daily.

Honest Trade-offs

No tool is perfect. Here is what I found:

OpenRouter streaming limitation. OpenRouter does not stream function call arguments properly. This causes file operation failures in Claude Code. If you use OpenRouter as a provider, expect issues with tool use.

Non-Anthropic model requirements. Any non-Anthropic model you route through must support tool use. Claude Code relies heavily on function calling. Models without proper tool support will fail on file operations, search, and other agent tasks.

Self-hosted only. The open-source version requires you to run and maintain the gateway. There is no managed cloud offering. That means monitoring, updating, and debugging are on you.

Newer project. Bifrost's community is growing but still smaller than older alternatives. Documentation is solid, but edge cases may require digging through issues on GitHub.

Extra hop. You are adding a process between Claude Code and your provider. The 11 microsecond overhead is negligible, but it is one more thing in the chain to keep running.

Performance

I ran benchmarks matching the benchmarking guide. The numbers held up: 11 microseconds of routing overhead, 5,000 requests per second on a single instance. The Go implementation makes a real difference. Python-based gateways I tested added significantly more latency.

For a gateway that sits in the critical path of every LLM call, low overhead matters.

Quick Start Summary

# 1. Start Bifrost
npx -y @maximhq/bifrost

# 2. Configure providers in bifrost.yaml (weighted routing + failover)

# 3. Point Claude Code at Bifrost
export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
export ANTHROPIC_API_KEY=your-bifrost-virtual-key

# 4. Use Claude Code normally

That is it. Your Claude Code session now routes across multiple models with automatic failover and budget controls.

GitHub | Docs | Website

If you are running Claude Code for real work, multi-model routing is not optional. Single-provider setups break at the worst times. A gateway that handles routing, failover, and cost controls in one place saves hours of debugging and thousands in unexpected spend.

Open an issue on the repo if you run into anything.

Best MCP Gateway for 50% Token Cost Savings

Pranay Batta — Tue, 07 Apr 2026 08:32:48 +0000

TL;DR: Classic MCP dumps 100+ tool definitions into every LLM call. Bifrost's Code Mode generates TypeScript declarations instead, cutting token usage by 50%+ and latency by 40-50%. If you are running 3 or more MCP servers, this is the single biggest cost lever you have.

The Problem with Classic MCP

I have been testing MCP setups for a few months now. The standard approach is simple. You connect your MCP servers, and every tool definition gets sent to the LLM as part of the context window. Every single call.

With 3 MCP servers, you might have 30-40 tools. With 10 servers, easily 100+. Each tool definition includes the name, description, input schema, and parameter types. That is a lot of tokens. And you are paying for every single one of them on every request.

The math is straightforward. If your average tool definition is 200 tokens, and you have 50 tools, that is 10,000 tokens of overhead per call. At scale, this adds up fast.

How Bifrost Code Mode Changes This

Bifrost takes a different approach with its Code Mode. Instead of exposing raw tool definitions to the LLM, it generates TypeScript declaration files (.d.ts) for all connected MCP tools.

The LLM then writes TypeScript code to orchestrate multiple tools in a restricted sandbox environment. Instead of the model making 5 separate tool calls (each requiring a round trip), it writes one code block that handles all 5 operations.

Here is what this means in practice:

Token reduction: 50%+ compared to classic MCP. The TypeScript declarations are more compact than full JSON schemas, and the model makes fewer round trips.
Latency reduction: 40-50% compared to classic MCP. Fewer round trips means faster overall execution.
Recommended when: You are using 3 or more MCP servers.

What Code Mode Actually Does

The execution model is restricted by design. Here is what is available in the sandbox:

Available: ES5.1+ JavaScript, async/await, TypeScript, console.log/error/warn, JSON.parse/stringify, and all MCP tool bindings as globals.

Not available: ES Modules, Node.js APIs, browser APIs, DOM, timers (setTimeout/setInterval), network access.

This is not a general-purpose runtime. It is a controlled environment where the LLM can orchestrate tools safely. No arbitrary code execution, no network calls outside of the tool bindings.

You can configure tool bindings at the server level or tool level, depending on how granular you need the control to be. The docs cover the binding configuration in detail.

The Latency Numbers

Bifrost itself adds 11 microseconds of latency overhead per request. It is written in Go and handles 5,000 RPS sustained throughput. That is roughly 50x faster than Python-based alternatives.

For MCP-specific operations:

Sub-3ms MCP latency overall
InProcess connections: ~0.1ms
STDIO connections: ~1-10ms
HTTP connections: ~10-500ms (network dependent)

The MCP tool discovery is cached after the first request, so subsequent calls hit ~100-500 microseconds for discovery and ~50-200 nanoseconds for tool filtering.

Agent Mode: The Other Side

Bifrost also has an Agent Mode that turns the gateway into an autonomous agent runtime. You configure which tools are auto-approved via tools_to_auto_execute, set a max_depth to prevent infinite loops, and let the agent handle iterative execution.

This is a different use case from Code Mode. Agent Mode is for workflows where you want the LLM to act autonomously within boundaries. Code Mode is for when you want to reduce token costs on tool-heavy operations.

Deployment

Setup is zero-config. You can start with npx or Docker. The gateway supports 19+ providers out of the box (OpenAI, Anthropic, Azure, Bedrock, Gemini, Mistral, Cohere, Groq, and others), all through an OpenAI-compatible API format.

# npx
npx -y @maximhq/bifrost

# Docker
docker run -p 8080:8080 maximhq/bifrost

Who Should Use Code Mode

If you are running fewer than 3 MCP servers, classic mode is probably fine. The overhead is manageable.

If you are running 3+, especially with 50+ tools across those servers, Code Mode is worth testing. The 50%+ token savings are significant at scale, and the 40-50% latency improvement compounds across multi-step agent workflows.

I tested this on a setup with 5 MCP servers and 80+ tools. The token savings were immediately visible in the cost dashboard. The reduced round trips also made the overall agent response noticeably faster.

GitHub: git.new/bifrost
Docs: getmax.im/bifrostdocs
Website: getmax.im/bifrost-home

How to Connect Any Model with Gemini CLI Using Bifrost AI Gateway

Pranay Batta — Mon, 06 Apr 2026 10:12:17 +0000

TL;DR: Gemini CLI works with Google's models out of the box. But if you want to route requests through multiple providers, add failover, or track costs, you can point Gemini CLI at Bifrost. One config change. Every model available through a single endpoint.

The Problem with Single-Provider CLI Tools

Gemini CLI connects to Google's Generative AI API. That is fine if you only use Gemini models. But most production setups involve multiple providers. OpenAI for some tasks. Anthropic for others. Maybe a local Ollama instance for development.

Switching between CLIs and API keys for each provider gets old fast.

I tested Bifrost, an open-source LLM gateway written in Go, as a unified routing layer for Gemini CLI. The setup took about 5 minutes.

How Bifrost Works with Gemini CLI

Bifrost exposes a fully Google GenAI compatible endpoint:

http://localhost:8080/genai/v1beta/models/{model}/generateContent

This means Gemini CLI can talk to Bifrost without any code changes. Just point the base URL to your Bifrost instance.

Bifrost then routes the request to whatever provider and model you specify. OpenAI, Anthropic, Vertex AI, Bedrock, Groq, Ollama. All through the same endpoint.

Setup

Step 1: Install Bifrost

npx @anthropic-ai/bifrost@latest

Zero config. Starts on port 8080 by default.

Step 2: Configure Providers

Add your provider keys to the config:

{
  "providers": {
    "openai": {
      "keys": [{"name": "openai-1", "value": "env.OPENAI_API_KEY", "models": ["gpt-4o", "gpt-4o-mini"], "weight": 1.0}]
    },
    "anthropic": {
      "keys": [{"name": "anthropic-1", "value": "env.ANTHROPIC_API_KEY", "models": ["claude-sonnet-4-20250514"], "weight": 1.0}]
    },
    "gemini": {
      "keys": [{"name": "gemini-1", "value": "env.GEMINI_API_KEY", "models": ["gemini-2.5-flash"], "weight": 1.0}]
    }
  }
}

Step 3: Point Gemini CLI to Bifrost

Set the base URL to your Bifrost instance:

export GEMINI_API_BASE_URL=http://localhost:8080

Now every request from Gemini CLI goes through Bifrost. You can target any provider using the provider-prefixed model format:

gemini/gemini-2.5-flash      → Google Gemini
openai/gpt-4o                → OpenAI
anthropic/claude-sonnet-4-20250514  → Anthropic
vertex/gemini-pro             → Vertex AI

What You Get

Multi-Provider Routing

One CLI, every model. No more switching between tools or managing separate API keys per provider.

Automatic Failover

Set up fallback chains in your requests. If Gemini is rate-limited, the request goes to OpenAI. If OpenAI is down, it goes to Anthropic. Each fallback is a fresh request. All plugins still run.

Budget Controls

Bifrost has a four-tier budget hierarchy: Customer, Team, Virtual Key, and Provider Config. Set a monthly spending cap on your Virtual Key. When it is hit, the gateway stops routing to paid providers. Your local Ollama instance can serve as the fallback.

Cost Tracking

Every request is logged with token counts and cost calculations. The Model Catalog tracks pricing across all providers automatically.

Performance

I ran 1,000 requests through Bifrost targeting Gemini models. The gateway adds 11µs of overhead per request. At 5,000 RPS sustained throughput, the bottleneck is always the provider, never the gateway. That is 50x faster than Python-based alternatives like LiteLLM.

The Overhead Question

The concern with adding a proxy layer is always latency. In practice, LLM API calls take 500ms to 5 seconds depending on the model and prompt. An 11µs gateway overhead is invisible.

The semantic caching layer (currently Weaviate-backed) can actually reduce latency for repeated queries by serving cached responses instead of hitting the provider again.

When This Makes Sense

This setup is useful if you:

Use Gemini CLI but also need OpenAI or Anthropic models
Want failover so your workflow does not break during provider outages
Need to track costs across providers in one place
Want to set budget limits so you do not get surprise bills

If you only use Gemini models and do not care about failover or cost tracking, direct connection is fine. The gateway adds value when you are working across providers.

Top 5 Enterprise AI Gateways for Dynamic Routing in 2026

Pranay Batta — Fri, 03 Apr 2026 05:42:24 +0000

If you are running multiple LLM providers in production, routing logic becomes a critical infrastructure decision. Send everything to one provider and you get single points of failure. Hardcode routing rules and you lose flexibility when latency spikes or rate limits hit.

I spent the last few weeks evaluating five AI gateways specifically for their dynamic routing capabilities. The criteria: latency overhead, failover behaviour, weighted distribution, and how much config it takes to get routing working in production.

The short version: Bifrost came out on top for raw performance and routing flexibility. 11 microsecond latency overhead, written in Go, with weighted routing and automatic failover built in. You can run it right now with npx -y @maximhq/bifrost. Full docs here.

Why dynamic routing matters

Static routing is fine for prototypes. Pick a model, call the API, ship it.

Production is different. You need:

Failover: When OpenAI returns 429s or 500s, traffic should automatically shift to Anthropic or another provider. No manual intervention.
Weighted distribution: Split traffic 70/30 across providers for cost optimization or A/B testing model quality.
Latency-based routing: Send requests to whichever provider responds fastest at that moment.
Budget-aware routing: Stop sending traffic to a provider when your spend cap is hit.

The gateway layer is the right place to handle this. Application code should not care which provider serves a request.

The five gateways I tested

1. Bifrost

Language: Go | Overhead: 11 microseconds | Throughput: 5,000 RPS sustained

Bifrost is the fastest gateway I have tested. The 11 microsecond overhead is not a typo. That is roughly 50x faster than Python-based alternatives like LiteLLM, which adds around 8ms per request.

Routing configuration is declarative and clean. Here is what weighted routing across two providers looks like:

# bifrost-config.yaml
providers:
  - name: openai-primary
    provider: openai
    model: gpt-4o
    weight: 70
    api_key: ${OPENAI_API_KEY}
  - name: anthropic-fallback
    provider: anthropic
    model: claude-sonnet-4-20250514
    weight: 30
    api_key: ${ANTHROPIC_API_KEY}

routing:
  strategy: weighted
  fallback:
    enabled: true
    max_retries: 2

That splits 70% of traffic to OpenAI and 30% to Anthropic. If OpenAI fails, requests automatically fall back to Anthropic.

What I like: the governance layer ties routing to budgets. You can set a four-tier budget hierarchy (Customer, Team, Virtual Key, Provider Config) and routing decisions respect those limits. When a provider budget is exhausted, traffic shifts automatically.

Setup is genuinely fast. One command to start:

npx -y @maximhq/bifrost

Or Docker:

docker run -p 8080:8080 maximhq/bifrost

The setup guide covers both approaches. Provider configuration takes a few minutes.

Other features worth noting: semantic caching with dual-layer support (exact hash + semantic similarity), observability built in, MCP support with sub-3ms latency and 50%+ token reduction in Code Mode, and a drop-in replacement endpoint for the Anthropic SDK so you can migrate without changing application code. Anthropic SDK integration docs here.

Check the benchmarks if you want to verify the numbers yourself.

2. LiteLLM

Language: Python | Overhead: ~8ms | Providers: 100+

LiteLLM has the widest provider coverage I have seen. Over 100 providers through a unified interface. If you need to call a niche model API, LiteLLM probably supports it.

Routing is available through the proxy server. You can configure fallbacks and load balancing across models. The configuration is YAML-based and straightforward.

model_list:
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_key: sk-xxx
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: sk-yyy

router_settings:
  routing_strategy: least-busy
  num_retries: 3

The trade-off is performance. At ~8ms overhead per request, you are adding meaningful latency at high throughput. For applications doing thousands of requests per second, that adds up. The Python runtime is the bottleneck.

Credit where it is due: LiteLLM's provider coverage is unmatched and the community is active. For teams that prioritize breadth over speed, it is a solid choice.

3. Kong AI Gateway

Language: Lua/C (OpenResty) | Type: Enterprise, plugin-based

Kong is a well-established API gateway that added AI capabilities through plugins. If your organization already runs Kong for general API management, adding AI routing is incremental.

The AI plugin supports multiple providers and basic routing. Rate limiting, authentication, and logging come from Kong's mature plugin ecosystem.

The limitation: AI-specific routing features require the enterprise tier. The open-source version gives you basic proxying, but weighted routing, advanced failover, and AI-specific analytics are paid features. Configuration is also more complex because you are working within Kong's plugin architecture rather than a purpose-built AI gateway.

Credit: Kong's plugin ecosystem is mature and battle-tested for general API management.

4. Cloudflare AI Gateway

Type: Managed service | Setup: Minutes

Cloudflare AI Gateway is the easiest to set up on this list. If you are already on Cloudflare, you can enable it from the dashboard and start routing requests through their edge network.

It provides caching, rate limiting, and basic analytics out of the box. The managed nature means zero infrastructure to maintain.

The limitation: routing flexibility is constrained compared to self-hosted options. Custom routing strategies, weighted distribution, and provider-level budget controls are limited. You also depend on Cloudflare's edge network for all LLM traffic, which may not work for teams with data residency requirements.

Credit: For teams that want AI gateway functionality without managing infrastructure, Cloudflare delivers the simplest path to production.

5. Azure API Management

Type: Enterprise, Azure-native | Setup: Hours to days

Azure APIM is the default choice for organizations already invested in Azure. It supports routing to Azure OpenAI endpoints with built-in integration, and you can configure policies for retry, circuit breaking, and load balancing.

The routing configuration uses Azure's policy XML, which is verbose but powerful. You get deep integration with Azure Monitor, Key Vault, and other Azure services.

The limitation: it is Azure-native. If you are multi-cloud or use non-Azure LLM providers, the integration story gets complicated. Routing to Anthropic or other providers requires custom policy work. Setup is also significantly more complex than purpose-built AI gateways.

Credit: For Azure-first organizations, the deep integration with the Azure ecosystem and enterprise compliance features are genuinely valuable.

Comparison table

Feature	Bifrost	LiteLLM	Kong AI	Cloudflare AI	Azure APIM
Latency overhead	11 microseconds	~8ms	Low (Lua/C)	Varies (edge)	Varies
Language	Go	Python	Lua/C	Managed	Managed
Weighted routing	Yes	Yes	Enterprise only	Limited	Via policy
Automatic failover	Yes	Yes	Yes	Basic	Via policy
Budget-aware routing	Yes (4-tier)	Basic	No	No	No
Semantic caching	Yes (dual-layer)	Basic	No	Yes	No
Provider count	Growing	100+	Major providers	Major providers	Azure-focused
Open source	Yes	Yes	Partial	No	No
Self-hosted	Yes	Yes	Yes	No	No
Setup time	Minutes	Minutes	Hours	Minutes	Hours to days

Honest trade-offs

No tool is perfect. Here is what I found lacking in each.

Bifrost: Provider count is still growing. If you need a niche provider that is not yet supported, you will need to check the docs or request it. The project is newer than LiteLLM, so community resources are still building up.

LiteLLM: Performance at scale is the main concern. The ~8ms overhead is fine for low-throughput applications, but at 5,000+ RPS, you are looking at significant cumulative latency. Memory usage also climbs with the Python runtime under load.

Kong AI Gateway: The AI features feel bolted on rather than native. If you are not already a Kong customer, adopting the full Kong stack just for AI routing is overkill. Enterprise pricing for AI-specific features is a barrier.

Cloudflare AI Gateway: Limited control. You cannot implement custom routing strategies or complex failover logic. Data flows through Cloudflare's network, which is a non-starter for some compliance requirements.

Azure APIM: Vendor lock-in is real. Multi-provider routing outside Azure requires significant custom work. Configuration through XML policies is tedious compared to YAML-based alternatives.

Which one should you pick

Pick Bifrost if performance and routing flexibility are your top priorities. The 11 microsecond overhead and built-in governance features (budget-aware routing, weighted distribution, automatic failover) make it the strongest option for high-throughput production workloads. Star it on GitHub or check the docs to get started.

Pick LiteLLM if you need the widest provider coverage and performance is not your bottleneck.

Pick Kong if your organization already runs Kong and wants to add AI routing incrementally.

Pick Cloudflare if you want zero infrastructure overhead and can live with limited routing customization.

Pick Azure APIM if you are fully committed to the Azure ecosystem.

For most teams building production AI infrastructure, routing is a gateway-level concern that should not leak into application code. The right gateway depends on your throughput requirements, provider mix, and how much control you need over routing logic.

I would start with Bifrost. One command to run, sub-microsecond overhead, and routing that actually works at scale. Docs are here.

How to Connect Non-Anthropic Models to Claude Code with Bifrost AI Gateway

Pranay Batta — Wed, 01 Apr 2026 13:41:26 +0000

I tested five different LLM gateways to route non-Anthropic models through Claude Code. Bifrost was the fastest by a wide margin. 11 microseconds of overhead per request. 50x faster than the Python-based alternatives I benchmarked.

Here is exactly how I set it up, what worked, and where each feature matters.

TL;DR: Bifrost is an open-source Go gateway that exposes an Anthropic-compatible endpoint, letting you route Claude Code requests to GPT-4o, Gemini, Bedrock, or any supported provider by changing one environment variable. You get multi-provider failover, budget controls, and semantic caching at 11 microseconds of overhead per request.

This post assumes you are familiar with Claude Code and have used at least one LLM API.

Bifrost on GitHub -- open-source, written in Go, handles 5,000 RPS on a single instance.

The Problem

Claude Code locks you into api.anthropic.com. No native way to swap providers. You cannot route to GPT-4o, Gemini, or Bedrock models without building your own proxy or switching tools entirely.

I needed:

GPT-4o for certain coding tasks
Gemini 2.5 Pro for long context
Automatic failover when a provider goes down
One place to track costs across all models

Building a custom proxy was not worth the maintenance burden. So I went looking for something production-ready.

What Bifrost Does

Bifrost exposes an Anthropic-compatible endpoint at /anthropic. Claude Code sends standard Anthropic-format requests. Bifrost translates and routes them to whatever provider you configure -- OpenAI, Bedrock, Vertex AI, Gemini, others.

It is a drop-in replacement. Change one URL. No SDK modifications. No wrapper code.

Claude Code -> Bifrost (/anthropic) -> Any LLM Provider

The Anthropic SDK integration page has the full compatibility details.

Setup: 3 Minutes

Step 1: Install

npx -y @maximhq/bifrost

That starts the gateway locally. Full setup instructions here.

Step 2: Configure a Provider

Create bifrost.yaml. This routes everything to GPT-4o:

accounts:
  - id: "my-account"
    providers:
      - id: "openai-primary"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        weight: 100

See the provider configuration docs for all supported providers and options.

Step 3: Point Claude Code at Bifrost

export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic

Done. Claude Code now sends requests through Bifrost, which translates them to OpenAI format and forwards to GPT-4o. Zero code changes.

Multi-Provider Routing

This is where it gets interesting. I configured weighted routing across two providers:

accounts:
  - id: "my-account"
    providers:
      - id: "openai-primary"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        weight: 80
      - id: "anthropic-fallback"
        type: "anthropic"
        api_key: "${ANTHROPIC_API_KEY}"
        model: "claude-sonnet-4-20250514"
        weight: 20

80% of traffic goes to GPT-4o. 20% to Claude. Useful when you want to compare output quality across models in real usage.

Automatic Failover

This was the feature that sold me. Failover configuration took five minutes. If GPT-4o goes down, Bifrost tries the next provider automatically:

accounts:
  - id: "my-account"
    failover:
      enabled: true
    providers:
      - id: "openai-primary"
        type: "openai"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        priority: 1
      - id: "gemini-secondary"
        type: "gemini"
        api_key: "${GEMINI_API_KEY}"
        model: "gemini-2.5-pro"
        priority: 2
      - id: "anthropic-tertiary"
        type: "anthropic"
        api_key: "${ANTHROPIC_API_KEY}"
        model: "claude-sonnet-4-20250514"
        priority: 3

OpenAI fails, Bifrost tries Gemini. Gemini fails, falls back to Anthropic. My Claude Code session never breaks. No retry logic on my side.

Bedrock and Vertex AI

I also tested with AWS Bedrock and Vertex AI. Same pattern:

providers:
  - id: "bedrock-claude"
    type: "bedrock"
    region: "us-east-1"
    model: "anthropic.claude-sonnet-4-20250514-v2:0"
    priority: 1
  - id: "vertex-gemini"
    type: "vertex"
    project_id: "my-gcp-project"
    region: "us-central1"
    model: "gemini-2.5-pro"
    priority: 2

Same Anthropic-compatible endpoint. Claude Code does not know which provider is behind Bifrost. That is the point.

Features Worth Mentioning

Routing alone is useful. But once all requests flow through one gateway, you get access to several other capabilities I found genuinely practical.

Budget Enforcement

Bifrost has a four-tier budget hierarchy: Customer, Team, Virtual Key, Provider Config. I set team-level limits:

budgets:
  - level: "team"
    id: "engineering"
    limit: 500
    period: "monthly"

Budget runs out, requests get blocked. No surprise bills. The governance docs cover the full hierarchy.

Semantic Caching

This cut my costs noticeably. Bifrost supports dual-layer caching: exact hash matching plus semantic similarity. If I have already asked a similar question, it returns the cached response instead of hitting the provider.

Supported vector stores: Weaviate, Redis, Qdrant.

Observability

Every request gets logged with latency, tokens, cost, and provider information. The observability layer gives you full visibility into what is happening across all your providers.

MCP Support

Bifrost also works as an MCP server. I tested Code Mode -- it reduced tokens by over 50% and latency by 40-50%. Agent Mode is available for more complex workflows. Useful if you are connecting to Claude Desktop or other MCP-compatible clients.

Benchmarks

I ran my own tests and the numbers matched what is documented. 11 microseconds overhead. 5,000 RPS on a single instance. The Go implementation makes a real difference compared to Python gateways I tested.

The benchmarking guide explains how to reproduce these numbers yourself.

Trade-offs and Limitations

Worth being upfront about the downsides:

Relatively new project. Bifrost does not have the years of battle-testing that older proxies have. The community is growing but smaller than established alternatives.
Self-hosted only. The open-source version has no managed cloud offering. You run and maintain the infrastructure yourself.
Extra operational overhead. You are running a separate process between Claude Code and your LLM provider. That is one more thing to monitor, update, and debug compared to direct API calls.
Provider coverage is expanding but not exhaustive. Some niche providers or model variants may not be supported yet. Check the docs before committing.

Quick Recap

Install: npx -y @maximhq/bifrost
Configure providers in bifrost.yaml
Set ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
Use Claude Code normally. Bifrost routes to whatever model you configured.

I tried building custom proxies before. I tried other gateways. This is the fastest option I found, and the setup takes minutes not hours.

GitHub repo | Docs | Website

If you run into issues or want a specific provider supported, open an issue on the repo.

How Bifrost Reduces GPT Costs and Response Times with Semantic Caching

Pranay Batta — Wed, 01 Apr 2026 05:51:15 +0000

TL;DR

Every GPT API call costs money and takes time. If your app sends the same (or very similar) prompts repeatedly, you are paying full price each time for answers you already have. Bifrost, an open-source LLM gateway, ships with a semantic caching plugin that uses dual-layer caching: exact hash matching plus vector similarity search. Cache hits cost zero. Semantic matches cost only the embedding lookup. This post walks you through how it works and how to set it up.

The cost problem with GPT API calls

If you are building anything production-grade with GPT-4, GPT-4o, or any OpenAI model, you already know that API costs add up fast. Token-based pricing means every request burns through your budget, whether it is a fresh question or something your system answered three minutes ago.

Here is the thing: in most real applications, a significant portion of requests are either identical or semantically similar to previous ones. Think about it. Customer support bots get asked the same questions in slightly different words. Code assistants receive near-identical prompts from different users. RAG pipelines retrieve similar context and ask similar follow-ups.

Without caching, you pay full model cost for every single one of those requests. You also wait for the full round-trip to the provider each time, adding latency that your users notice.

The obvious fix is caching. But traditional exact-match caching has a big limitation: it only works when the prompt is character-for-character identical. Change one word, add a comma, rephrase slightly, and you get a cache miss. That is where semantic caching changes the game.

What semantic caching is and how it differs from exact-match caching

Exact-match caching hashes the entire request and looks up that hash. If the hash matches a stored response, you get a cache hit. If even one character is different, it is a miss. This works well for automated pipelines where prompts are templated and predictable. It falls apart for user-facing applications where people phrase things differently.

Semantic caching converts the request into a vector embedding and searches for similar embeddings in a vector store. If a stored request is semantically similar enough (above a configurable threshold), the cached response is returned. This means "How do I reset my password?" and "What are the steps to change my password?" can both hit the same cache entry.

Bifrost combines both approaches in a dual-layer architecture, giving you the speed of exact matching with the intelligence of semantic similarity as a fallback.

How Bifrost implements dual-layer caching

Bifrost's semantic cache plugin uses a two-step lookup process for every request that has a cache key:

Layer 1: Exact hash match. The plugin hashes the request and checks for a direct match. This is the fastest path. If it hits, you get the cached response with zero additional cost. No embedding generation, no vector search, no provider call.

Layer 2: Semantic similarity search. If the exact match misses, Bifrost generates an embedding for the request and searches the vector store for semantically similar entries. If a match is found above the similarity threshold (default 0.8), the cached response is returned. The only cost here is the embedding generation.

If both layers miss, the request goes to the LLM provider as normal. The response is then stored in the vector store with its embedding for future lookups.

You can also control which layer to use per request. If you know your use case only needs exact matching (templated prompts), you can skip the semantic layer entirely. If you want semantic-only, that is an option too. The default is both, with direct matching first and semantic as fallback.

Here is how the cost breaks down:

Scenario	LLM API Cost	Embedding Cost	Total Cost
Exact cache hit	Zero	Zero	Zero
Semantic cache hit	Zero	Embedding only	Minimal
Cache miss	Full model cost	Embedding generation	Full + embedding

Bifrost also handles cost calculation natively through CalculateCostWithCacheDebug, which automatically accounts for cache hits, semantic matches, and misses in your cost tracking. All pricing data is cached in memory for O(1) lookup, so the cost calculation itself adds no overhead.

Check out the full Bifrost documentation for the complete API reference.

Setting it up

Follow the setup guide to get Bifrost running, then configure two things: a vector store and the semantic cache plugin.

Step 1: Configure the vector store

Bifrost uses Weaviate as its vector store. You can run Weaviate locally with Docker or use Weaviate Cloud.

Local setup with Docker:

docker run -d \
  -p 8080:8080 \
  -p 50051:50051 \
  -e PERSISTENCE_DATA_PATH='/var/lib/weaviate' \
  semitechnologies/weaviate:latest

config.json (local Weaviate):

{
  "vector_store": {
    "enabled": true,
    "type": "weaviate",
    "config": {
      "host": "localhost:8080",
      "scheme": "http"
    }
  }
}

config.json (Weaviate Cloud):

{
  "vector_store": {
    "enabled": true,
    "type": "weaviate",
    "config": {
      "host": "your-cluster.weaviate.network",
      "scheme": "https",
      "api_key": "your-weaviate-api-key"
    }
  }
}

Step 2: Configure the semantic cache plugin

Add the plugin to your Bifrost config:

{
  "plugins": [
    {
      "enabled": true,
      "name": "semantic_cache",
      "config": {
        "provider": "openai",
        "embedding_model": "text-embedding-3-small",
        "ttl": "5m",
        "threshold": 0.8,
        "conversation_history_threshold": 3,
        "exclude_system_prompt": false,
        "cache_by_model": true,
        "cache_by_provider": true,
        "cleanup_on_shutdown": true
      }
    }
  ]
}

A few things to note about these settings:

threshold: The similarity score (0 to 1) required for a semantic match. 0.8 is a good starting point. Higher means stricter matching, fewer false positives, but more cache misses.
conversation_history_threshold: Defaults to 3. If a conversation has more messages than this, caching is skipped. Long conversations have high probability of false positive semantic matches due to topic overlap, and they rarely produce exact hash matches anyway.
ttl: How long cached responses stay valid. Accepts duration strings like "30s", "5m", "1h", or numeric seconds.
cache_by_model and cache_by_provider: When true, cache entries are isolated per model and provider combination. A GPT-4 response will not be returned for a GPT-3.5-turbo request.

Step 3: Trigger caching per request

Caching is opt-in per request. You need to set a cache key, either via the Go SDK or HTTP headers:

HTTP API:

# This request WILL be cached
curl -H "x-bf-cache-key: session-123" \
     -H "Content-Type: application/json" \
     -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "What is semantic caching?"}]}' \
     http://localhost:8080/v1/chat/completions

Go SDK:

ctx = context.WithValue(ctx, semanticcache.CacheKey, "session-123")
response, err := client.ChatCompletionRequest(ctx, request)

Without the cache key, requests bypass caching entirely. This gives you fine-grained control over what gets cached and what does not.

Per-request overrides (HTTP):

curl -H "x-bf-cache-key: session-123" \
     -H "x-bf-cache-ttl: 30s" \
     -H "x-bf-cache-threshold: 0.9" \
     http://localhost:8080/v1/chat/completions

Cache type control:

# Direct hash matching only (fastest, no embedding cost)
curl -H "x-bf-cache-key: session-123" \
     -H "x-bf-cache-type: direct" ...

# Semantic similarity search only
curl -H "x-bf-cache-key: session-123" \
     -H "x-bf-cache-type: semantic" ...

# Default: both (direct first, semantic fallback)
curl -H "x-bf-cache-key: session-123" ...

You can also use no-store mode to read from cache without storing the response:

curl -H "x-bf-cache-key: session-123" \
     -H "x-bf-cache-no-store: true" ...

When semantic caching helps vs when it does not

Semantic caching is not a universal solution. Here is where it works well and where it does not.

Good fit:

Customer support bots where users ask the same questions in different words
FAQ-style applications with predictable query patterns
RAG pipelines where similar contexts produce similar queries
Internal tools where multiple team members ask overlapping questions
Any high-volume application with repetitive prompt patterns

Not a good fit:

Conversations that are heavily context-dependent and unique every time
Long multi-turn conversations (the conversation_history_threshold exists for this reason, as longer conversations create false positive matches)
Applications where responses must reflect real-time data that changes frequently
Creative generation tasks where you want varied outputs for similar inputs

The key insight is that semantic caching works best when your application naturally produces clusters of similar requests. If every request is genuinely unique, caching of any kind will not help much.

Other performance details worth knowing

Beyond semantic caching, Bifrost caches aggressively at multiple levels:

Tool discovery is cached after the first request, bringing subsequent lookups down to roughly 100-500 microseconds.
Health check results are cached at approximately 50 nanoseconds.
All pricing data is cached in memory for O(1) lookups during cost calculations.

Cache entries use namespace isolation. Each Bifrost instance gets its own vector store namespace to prevent conflicts. When the Bifrost client shuts down (with cleanup_on_shutdown set to true), all cache entries and the namespace itself are cleaned up. You can also programmatically clear cache by key or clear cache by request ID via the API.

Cache metadata is automatically added to responses via response.ExtraFields.CacheDebug, so you can inspect whether a response came from direct cache, semantic match, or a fresh provider call. You can also use the log statistics API for deeper observability into your cache performance.

Wrapping up

If your GPT-powered application handles any volume of requests, there is a good chance a meaningful portion of those requests are semantically similar. Paying full API cost for every one of them does not make sense.

Bifrost's semantic cache plugin gives you dual-layer caching with exact matching and vector similarity search, opt-in per request, configurable thresholds, and built-in cost tracking. It is open source, written in Go, and designed for production workloads.

Check out the GitHub repo to get started, read the docs for the full configuration reference, or visit the Bifrost website to learn more about the gateway.

LLM Cost Tracking and Spend Management for Engineering Teams

Pranay Batta — Wed, 01 Apr 2026 05:43:30 +0000

Your team ships a feature using GPT-4, it works great in staging, and then production traffic hits. Suddenly you are burning through API credits faster than anyone expected. Multiply that across three providers, five teams, and a few hundred thousand requests per day. Good luck figuring out where the money went.

We built Bifrost, an open-source LLM gateway in Go, and cost tracking was one of the first problems we had to solve properly. This post covers what we learned, how we designed spend management into the gateway layer, and what the alternatives look like. You can get started with the setup guide in under a minute.

TL;DR: Bifrost gives you per-request cost logging, four-tier budget hierarchies (Customer, Team, Virtual Key, Provider Config), auto-synced model pricing, and cache-aware cost calculations. All at 11 microsecond latency overhead. You can run it right now with npx -y @maximhq/bifrost. Full docs here.

The actual problem with LLM costs

Cloud compute costs are predictable. You pick an instance type, you know the hourly rate, you can forecast monthly spend within a few percent.

LLM costs are nothing like that.

A single API call costs somewhere between $0.0001 and $0.50 depending on the model, the input length, the output length, whether you are sending images or audio, and whether the context crosses the 128k token threshold (where pricing tiers change). That is per request.

Now add multi-provider routing. Your app might use OpenAI for chat, Anthropic for analysis, and a smaller model for classification. Each provider has different pricing structures, different token counting methods, and different billing cycles.

The result: engineering teams have no idea what they are spending until the invoice arrives.

What cost tracking actually requires

Most teams start with "we will check the provider dashboard." That breaks down fast for three reasons.

Per-request granularity. You need to know the cost of every single API call, tied to which customer, which team, and which feature triggered it. Provider dashboards give you aggregate numbers, not per-request attribution.

Real-time budget enforcement. Knowing you overspent last month does not help. You need the system to reject requests when a budget limit is hit, before the money is gone.

Multi-modal cost calculation. If your app sends images, audio, or very long contexts, the cost calculation is not a simple token multiplication. You need tiered pricing support, per-image costs, per-second audio costs, and character-based pricing for certain models.

How we built cost tracking in Bifrost

We wanted cost management to be a gateway-level concern, not something each application team has to implement. Here is how the pieces fit together.

Model Catalog with auto-synced pricing

The Model Catalog is the foundation. It maintains pricing data for every supported model across all providers. You can also force a pricing sync at any time via the API.

On startup, Bifrost downloads the latest pricing sheet and loads it into memory. When a ConfigStore (SQLite or PostgreSQL) is available, it also persists the data and re-syncs every 24 hours automatically. All lookups are O(1) from memory.

The pricing data covers multiple modalities:

Text: token-based and character-based pricing for chat completions, text completions, and embeddings
Audio: token-based and duration-based pricing for speech synthesis and transcription
Images: per-image costs with tiered pricing for high-token contexts
Tiered pricing: automatic rate changes above 128k tokens, reflecting actual provider pricing

This means cost calculation is accurate for every request type, not an approximation based on token count alone.

Four-tier budget hierarchy

This is where spend management happens. Bifrost supports budgets at four levels:

Customer - set a spending cap for an entire customer account
Team - limit spend per team within a customer
Virtual Key - control costs per API key (useful for per-feature or per-environment budgets)
Provider Config - cap total spend on a specific provider

Each budget has a max_limit, a reset_duration (daily, weekly, monthly), and tracks current_usage in real time.

Here is what creating a customer with a budget looks like via the API:

curl --request POST \
  --url http://localhost:8080/api/governance/customers \
  --header 'Content-Type: application/json' \
  --data '{
    "name": "acme-corp",
    "budget": {
      "max_limit": 500,
      "reset_duration": "monthly"
    }
  }'

The response includes the budget object with current_usage tracked automatically:

{
  "customer": {
    "id": "cust-abc123",
    "name": "acme-corp",
    "budget": {
      "id": "bdgt-xyz",
      "max_limit": 500,
      "reset_duration": "monthly",
      "current_usage": 0
    }
  }
}

When current_usage hits max_limit, requests are rejected. No surprises on the invoice.

LogStore: per-request cost audit trail

Every request that passes through Bifrost gets logged with full cost data. The LogStore captures:

Provider and model used
Input tokens, output tokens, total tokens
Calculated cost (broken down into input cost, output cost, request cost, total cost)
Latency
Status (success or error)
Timestamps
Full input/output content (serialized as JSON)

You can query this data with filters. Want to see all requests to OpenAI that cost more than $0.10 in the last hour? That is a single API call.

curl --request POST \
  --url http://localhost:8080/api/logs/search \
  --header 'Content-Type: application/json' \
  --data '{
    "filters": {
      "providers": ["openai"],
      "min_cost": 0.10,
      "start_time": "2026-03-31T00:00:00Z"
    },
    "pagination": {
      "limit": 50,
      "sort_by": "cost",
      "order": "desc"
    }
  }'

The response includes aggregated stats alongside individual logs: total requests, success rate, average latency, total tokens, and total cost for the query. This is the data you need for cost attribution and chargeback.

Getting started

You can have this running in under a minute:

npx -y @maximhq/bifrost

Or with Docker if you prefer containerized deployment. Then point your LLM calls at the Bifrost endpoint instead of directly at the provider — it works as a drop-in replacement for the OpenAI SDK, Anthropic SDK, and Bedrock SDK. Cost tracking, budget enforcement, and logging happen automatically.

Check the setup docs for configuration details.

Cache-aware cost tracking

This is a detail that matters more than you would expect.

Bifrost includes a dual-layer semantic cache (exact hash matching + semantic similarity via Weaviate). When a request hits the cache, the cost calculation changes:

Direct cache hit (exact match): zero cost. The response comes from cache, no provider API call is made.
Semantic cache hit (similar query found): the cost is the embedding generation cost only. No model inference cost.
Cache miss with storage: the cost is the base model usage plus the embedding generation cost for storing the result.

If you are not tracking cache-aware costs, your cost reports will overcount. Every cache hit that gets reported at full model price inflates your numbers and hides the ROI of caching.

How other tools handle cost tracking

Credit where it is due. There are several tools in this space, and they each take a different approach.

Helicone is a proxy-based observability platform. It logs requests and provides cost analytics through a dashboard. The cost tracking is solid, with per-request granularity. Where it differs from Bifrost: Helicone is primarily an observability tool. Budget enforcement and cache-aware cost calculations are not its focus. It is a good choice if you want analytics without gateway-level controls.

OpenRouter acts as a unified API layer across multiple LLM providers. It handles routing and gives you a single bill, which simplifies accounting. However, OpenRouter is a hosted proxy — your requests pass through their infrastructure. There is no self-hosted option, no budget enforcement at the gateway level, and no per-customer or per-team spend hierarchy. If you need cost attribution beyond "which model was called," you will need to build that yourself on top of their logs.

AWS API Gateway + Bedrock is what many AWS-native teams reach for. You get IAM-based access control and CloudWatch metrics. The limitation is that cost tracking is coarse-grained — you get aggregate billing through AWS Cost Explorer, not per-request cost breakdowns tied to your internal teams or customers. Building a four-tier budget hierarchy on top of AWS services means stitching together Lambda, DynamoDB, and custom billing logic. It works but it is a lot of glue code.

Kong AI Gateway and Cloudflare AI Gateway both provide rate limiting and basic analytics for AI API traffic. Kong gives you plugin-based extensibility, and Cloudflare gives you edge caching and DDoS protection. Neither provides built-in per-request cost calculation with multi-modal pricing awareness, and neither offers the kind of budget hierarchy where you can set spending caps at the customer, team, and key level with automatic enforcement.

LiteLLM is the most well-known Python-based proxy. It supports cost tracking and has a wide model coverage. The trade-off is performance. LiteLLM adds roughly 8ms of latency overhead per request. Bifrost adds 11 microseconds, which is about 50x faster. At 5,000 RPS, that difference compounds. If your use case is low-throughput internal tooling, LiteLLM works fine. If you are running production workloads at scale, the latency overhead matters.

The math is straightforward: at 5,000 requests per second, 8ms overhead means 40 seconds of cumulative latency overhead per second of wall time. At 11 microseconds, it is 0.055 seconds.

What we learned building this

A few things surprised us during development.

Pricing data goes stale fast. Providers update pricing regularly. We started with a static pricing file and quickly realized it needed to be auto-synced. The 24-hour sync interval with O(1) memory lookups was the balance we settled on. You can also trigger a manual pricing sync via POST /api/pricing/force-sync if a provider drops prices and you want immediate accuracy.

Budget enforcement needs to be in the hot path. We tried implementing budgets as an async check initially. The problem: by the time the async check ran, the request was already sent to the provider and the cost was incurred. Budget checks have to happen before the request goes upstream. That is why Bifrost handles it at the gateway layer with in-memory state.

Multi-modal cost calculation is harder than it looks. Text-only cost is straightforward: multiply tokens by price per token. But when a request includes images, the cost depends on the image resolution and the token context length. Audio adds per-second pricing. Some models charge per character instead of per token. The Model Catalog handles all of this, but getting it right required modelling each provider's pricing structure individually.

Cost attribution needs hierarchy. Flat per-key budgets are not enough for real organizations. An engineering team needs to know: "How much is Customer X spending? How much of that is Team Y? Which virtual key is burning through budget?" That is why we built the four-tier hierarchy (Customer, Team, Virtual Key, Provider Config). You can create virtual keys via the API and attach budgets to each level.

Wrapping up

LLM cost management is not optional for production systems. If you are routing requests across multiple providers without per-request cost tracking, budget enforcement, and cache-aware calculations, you are flying blind. For enterprise teams, Bifrost also supports audit logs, log exports, and intelligent load balancing.

Bifrost is open-source, written in Go, and runs with a single command. It handles cost tracking at the gateway layer so your application code does not have to.

If you are dealing with LLM spend management, give it a try and let us know what is missing. We are actively building based on what teams actually need.

LiteLLM vs Bifrost: Why the Supply Chain Attack Changes Everything for LLM Gateways

Pranay Batta — Sat, 28 Mar 2026 05:10:03 +0000

If you're running LiteLLM in production, the March 2026 supply chain attack probably got your attention. Mine too. I spent the past few days digging into what happened, why it happened, and what it means for anyone choosing an LLM gateway in 2026.

This is not a hit piece. LiteLLM is a solid project with massive adoption. But this incident exposed something structural that every engineering team needs to think about. And it happens to make the case for Bifrost, a Go-based alternative, in ways that go beyond the usual performance benchmarks.

Let's break it all down.

TL;DR

Two backdoored versions of LiteLLM (1.82.7, 1.82.8) were published to PyPI on March 24, 2026, via stolen credentials.
The malware stole SSH keys, AWS/GCP/Azure credentials, and Kubernetes secrets. It used Python's .pth persistence mechanism to survive across interpreter restarts.
DSPy, MLflow, CrewAI, OpenHands, and Arize Phoenix all pulled the compromised version.
Bifrost is a Go-based LLM gateway that compiles to a single binary. The attack vector that hit LiteLLM simply does not exist in its architecture.
Beyond security, Bifrost adds 11 microseconds of overhead per request vs LiteLLM's roughly 8ms, supports 20+ providers, offers semantic caching via Weaviate, and has a four-tier budget hierarchy.

What Happened: The Full Attack Chain

Here's the sequence of events, based on Snyk's detailed investigation.

Step 1: The Trivy GitHub Action was compromised. A group called TeamPCP tampered with the widely-used Trivy security scanner GitHub Action.

Step 2: LiteLLM's CI/CD pipeline pulled the compromised Trivy Action. Because LiteLLM's workflow used an unpinned version of the Trivy GitHub Action (not pinned to a specific SHA), the compromised version ran inside LiteLLM's CI environment.

Step 3: The malicious Trivy Action exfiltrated LiteLLM's PYPI_PUBLISH token. With that token, the attackers could publish any package version to PyPI under LiteLLM's name.

Step 4: Two backdoored versions (1.82.7, 1.82.8) were published to PyPI. These looked like normal LiteLLM updates. Anyone running pip install --upgrade litellm got them.

Step 5: The malware deployed a .pth persistence file. This is the part that needs explaining.

What are .pth files?

If you're not deep into Python internals, .pth files might be new to you. They live in Python's site-packages directory and get executed automatically every time the Python interpreter starts up. Not when you import a specific package. Every single time Python runs. Anything.

The attackers placed a .pth file that loaded their malware on every Python interpreter startup. It did not matter whether your code imported litellm or not. If the package was installed in the environment, the malware was active.

What the malware stole:

SSH private keys
AWS, GCP, and Azure credentials
Kubernetes secrets
Crypto wallet keys

Step 6: The attackers used 73 compromised GitHub accounts to spam the disclosure issue with noise and eventually closed it using stolen maintainer credentials, trying to suppress the report.

The backdoored versions were live on PyPI for approximately 3 hours. LiteLLM has 3.4 million+ daily downloads. You can do the math on the blast radius.

Why Architecture Matters More Than You Think

Let's talk about why this specific attack cannot happen to Bifrost.

It is not just "Bifrost is written in Go, so it's safe." That would be a lazy argument. The actual reasons are architectural, and they matter.

No site-packages directory

Python packages install into site-packages. That directory is a shared space where any installed package can drop files, including .pth files that execute on interpreter startup. This is the mechanism the LiteLLM attackers exploited.

Go compiles to a single static binary. There is no site-packages equivalent. There is no shared directory where a compromised dependency could drop a persistence mechanism. The binary is the binary.

No .pth hook mechanism

Python's .pth file execution is a feature, not a bug. It exists for legitimate reasons (configuring import paths, running initialization code). But it also means any package you install can run arbitrary code on every Python startup without your knowledge or consent.

Go has no equivalent mechanism. When you compile a Go binary, what goes in is what comes out. There is no startup hook that third-party code can inject into after compilation.

No transitive pip dependency chain

LiteLLM has a substantial dependency tree. Each of those dependencies has its own dependencies. Each one is a potential attack surface. When you pip install litellm, you're trusting not just the LiteLLM maintainers but every maintainer of every transitive dependency.

Bifrost ships as a compiled binary via npx -y @maximhq/bifrost or Docker (docker pull maximhq/bifrost). Dependencies are resolved and compiled at build time by the Bifrost team. You're running a single binary, not managing a dependency tree.

The CI/CD surface area is smaller

The LiteLLM attack started with a compromised GitHub Action in CI/CD. Go binaries distributed via npm or Docker reduce the CI/CD surface area because the compilation and dependency resolution happen upstream, not in your pipeline.

This is not about Go being "more secure" than Python as a language. It's about the deployment model. A compiled binary distributed as a single artifact has a fundamentally smaller attack surface than a package installed via a package manager with a transitive dependency tree and runtime hook mechanisms.

Side-by-Side Feature Comparison

Here's an honest look at both gateways.

Feature	LiteLLM	Bifrost
Language	Python	Go
Deployment	`pip install`, Docker	`npx`, Docker, Go binary
Provider support	100+ providers	20+ providers (OpenAI, Anthropic, Bedrock, Azure, Gemini, Vertex AI, Groq, Mistral, Cohere, xAI, and more) + custom providers
Overhead per request	~8ms	11 microseconds
Throughput	Varies (Python GIL limits)	5,000 RPS sustained
Caching	Redis-based key-value	Weaviate-powered dual-layer semantic caching
Budget management	Basic spend tracking	Four-tier hierarchy (Customer > Team > Virtual Key > Provider Config)
MCP support	Limited	Full MCP gateway with four connection types, sub-3ms latency
Web UI	Dashboard available	Built-in Web UI for visual setup, monitoring, and governance
OpenAI compatibility	Yes	Yes (drop-in replacement, single URL change)
Supply chain surface	PyPI + transitive deps + .pth hooks	Single compiled binary
Configuration	Config files, environment variables	Zero-config start, Web UI, API, or config.json

Let me be upfront: LiteLLM's provider count is significantly higher. If you need access to 100+ providers through a single gateway, that is a real advantage. Bifrost supports 20+ providers natively with the ability to add custom providers, which covers most production use cases, but it is not the same breadth.

Performance Deep Dive: What the Numbers Actually Mean

You'll see "11 microseconds vs 8 milliseconds" in Bifrost's benchmarks. That's roughly a 50x difference. But what does it mean in practice?

Let's do the math at different scales.

At 10,000 requests per day:

LiteLLM overhead: 10,000 x 8ms = 80 seconds of cumulative gateway latency
Bifrost overhead: 10,000 x 11 microseconds = 0.11 seconds

At 100,000 requests per day:

LiteLLM overhead: 100,000 x 8ms = 800 seconds (~13.3 minutes)
Bifrost overhead: 100,000 x 11 microseconds = 1.1 seconds

At 1,000,000 requests per day:

LiteLLM overhead: 1,000,000 x 8ms = 8,000 seconds (~2.2 hours)
Bifrost overhead: 1,000,000 x 11 microseconds = 11 seconds

At low volume, the difference doesn't matter much. Your LLM provider's response time (hundreds of milliseconds to seconds) dwarfs the gateway overhead either way.

But at scale, the difference becomes real. 13 minutes of cumulative latency at 100K requests/day isn't catastrophic, but it adds up across your user base. And 2.2 hours at a million requests/day starts affecting tail latencies and user experience, especially for streaming responses where gateway overhead is felt on every chunk.

The 5,000 RPS sustained throughput from Bifrost also matters. Python's GIL (Global Interpreter Lock) creates a concurrency ceiling that Go simply doesn't have. If you're running high-concurrency workloads, this is a material difference.

Here's What This Means for Your Stack

If you're evaluating LLM gateways right now, the LiteLLM incident should change your evaluation criteria. Not because LiteLLM is bad software, but because it highlighted a category of risk that most teams weren't thinking about.

Questions to ask about any LLM gateway:

What's the dependency footprint? How many transitive dependencies does it pull in? Each one is a potential attack surface.
What's the deployment model? Is it a package you install into your environment, or a standalone binary/container?
Does it have runtime hook mechanisms? Can dependencies execute code at startup without explicit imports?
How is it distributed? Via a package manager with mutable versions, or via immutable artifacts?
What's in the CI/CD chain? Are GitHub Actions pinned by SHA? Are publish tokens scoped and rotated?

These aren't questions most teams were asking about their LLM gateway a month ago. They should be now.

When LiteLLM Still Makes Sense

I want to be honest about this. There are real scenarios where LiteLLM is the better choice.

You need access to 100+ providers. LiteLLM's provider breadth is unmatched. If you're working with niche or specialized providers that Bifrost doesn't support yet, LiteLLM gets you there faster.
Your entire stack is Python and you want deep integration. LiteLLM plays well with the Python ML ecosystem. If you're already in that world and need tight integration with LangChain, LlamaIndex, or similar frameworks, LiteLLM fits naturally.
You need it as a library, not a gateway. LiteLLM can be imported and used as a Python library within your application code. Bifrost is a standalone gateway service.

If any of these are your primary requirement, LiteLLM may still be right for you. Just audit your versions, pin your dependencies, and check for .pth files in your site-packages directory.

When Bifrost Is the Better Choice

Bifrost wins when your priorities look like this:

Security surface area matters to you. If you're in a regulated industry, handle sensitive data, or simply don't want to worry about Python supply chain attacks in your infrastructure layer, a compiled Go binary is a different risk profile entirely.
Performance at scale. If you're pushing high request volumes and need minimal gateway overhead, 11 microseconds vs 8 milliseconds is not a rounding error.
You want governance out of the box. Bifrost's four-tier budget hierarchy (Customer > Team > Virtual Key > Provider Config) with independent budget checking at each level gives you cost control that's built into the gateway, not bolted on.
Semantic caching. Bifrost's Weaviate-powered dual-layer caching understands the meaning of requests, not just exact matches. Similar queries hit the cache even if they're worded differently.
MCP gateway support. If you're building agentic applications, Bifrost has native MCP support with four connection types and sub-3ms tool execution latency.
Zero-config setup. Run npx -y @maximhq/bifrost and you have a working gateway with a Web UI at localhost:8080. No config files, no environment variables, no setup ceremony.

The Bigger Question

Should your LLM gateway be a Python package at all?

This isn't Python-bashing. Python is great for ML research, data science, prototyping, and application-level code. But your LLM gateway sits in the critical path of every AI request your application makes. It's infrastructure.

Infrastructure components have different requirements than application code. They need to be fast, stable, have minimal dependencies, and present the smallest possible attack surface. This is why web servers, databases, load balancers, and message queues are almost never written in Python. They're written in C, C++, Go, or Rust.

The LiteLLM incident didn't happen because of a bug in LiteLLM's code. It happened because of a structural property of the Python packaging ecosystem. That's a different kind of risk, and it's one that applies to any Python package in your infrastructure layer.

Action Items

If you're currently using LiteLLM:

Check your installed version immediately. Versions 1.82.7 and 1.82.8 are compromised.
Search for .pth files in your Python site-packages directories.
Rotate all credentials that were accessible from environments where LiteLLM was installed (SSH keys, cloud provider credentials, Kubernetes secrets).
Pin your GitHub Actions by SHA, not by tag.
Evaluate whether a compiled gateway is a better fit for your security posture.

If you're evaluating LLM gateways:

Try Bifrost: npx -y @maximhq/bifrost (takes 30 seconds)
Check out the GitHub repo to see the codebase
Read the docs for the full feature set
Visit the website for architecture details

The LLM gateway space is going to look different after this incident. Supply chain security just became an evaluation criterion, and compiled gateways have a structural advantage that no amount of Python dependency scanning can match.

The LiteLLM Supply Chain Attack Broke Trust in Python-Based AI Infrastructure

Pranay Batta — Fri, 27 Mar 2026 04:48:05 +0000

If you run LiteLLM in production, you probably had a rough week.

On March 24, 2026, two backdoored versions of litellm (1.82.7 and 1.82.8) were published to PyPI using stolen credentials. The malware stole SSH keys, AWS/GCP/Azure credentials, Kubernetes secrets, cryptocurrency wallets, and deployed persistent backdoors on infected machines. It was live for about 3 hours. LiteLLM gets 3.4 million daily downloads.

This is the full breakdown of what happened, why it matters, and what you should actually do about it.

What Happened: The Full Attack Chain

The attack didn't start with LiteLLM. It started with Trivy, a popular container security scanner.

Here's the sequence:

A threat actor group called TeamPCP exploited a pull_request_target workflow vulnerability in Trivy's GitHub Action (GHSA-9p44-j4g5-cfx5)
They used this to exfiltrate the aqua-bot credentials and rewrite Trivy v0.69.4 release tags to point to malicious payloads
On March 23, they also compromised the Checkmarx KICS GitHub Action using similar techniques
LiteLLM's CI/CD pipeline pulled the Trivy action without pinning to a specific version or commit SHA
The malicious Trivy action exfiltrated LiteLLM's PYPI_PUBLISH token from the GitHub Actions runner
Using the stolen token, TeamPCP published two backdoored versions to PyPI with legitimate credentials

The full timeline is documented in Snyk's breakdown and Wiz's attribution analysis.

The Malware: Three Stages of Damage

This was not a simple credential stealer. It was a three-stage payload designed for maximum extraction and persistence.

Stage 1: Information Collection

The malware harvested:

System data (hostname, environment variables, processes, network routing, auth logs)
SSH private keys, .env files, Git credentials
API keys from shell history
Slack and Discord webhook tokens
Jenkins, Travis CI, and Terraform configs
/etc/shadow password hashes
Full AWS credential chain including IMDSv2, Secrets Manager, and SSM Parameter Store
GCP and Azure service account files
Docker config.json and Kubernetes kubeconfig, service account tokens, secrets, and certificates
Cryptocurrency wallet files (Bitcoin, Ethereum, Solana, Cardano, Monero, Ledger hardware wallet configs)

Stage 2: Encryption and Exfiltration

Everything was bundled, encrypted with AES-256-CBC, the session key wrapped with a hardcoded 4096-bit RSA public key, and exfiltrated to models.litellm.cloud, a domain registered just one day before the attack.

Stage 3: Persistence and Lateral Movement

The malware installed a persistent backdoor at ~/.config/sysmon/sysmon.py with a systemd service polling checkmarx.zone every 5 minutes for new payloads. In Kubernetes environments, it read all secrets across every namespace and deployed privileged pods to every kube-system node, mounting the host filesystem.

Source: Snyk's full technical analysis

Two Delivery Mechanisms (This Is the Scary Part)

Version 1.82.7 embedded a Base64-encoded payload directly in litellm/proxy/proxy_server.py. It executed whenever anything imported litellm.proxy.

Version 1.82.8 added something worse: a file called litellm_init.pth placed in site-packages/. Python's .pth mechanism fires on every interpreter startup. That means the malware ran when you ran pip install, when your IDE started a language server, when you ran python -c "print('hello')". It had nothing to do with importing litellm.

This is MITRE ATT&CK T1546.018 (Python Startup Hooks). The .pth file was correctly declared in the wheel's RECORD file, so pip install --require-hashes would have passed.

Who Was Affected

The compromised versions were pulled into multiple major projects:

Project	PR Merged	Status
DSPy	#9498	Affected
MLflow	#21971	Affected
OpenHands	#13569	Affected
CrewAI	#5040	Affected
Arize Phoenix	#12342	Affected
Aider	N/A	Safe (pinned litellm==1.82.3)

Aider survived because it pinned its dependency version. That one decision made the difference.

How Standard Defenses Failed

This attack bypassed almost every standard protection:

Hash verification passed because the packages were published with legitimate stolen credentials
No typosquatting to detect. The package name was exactly litellm
No suspicious domains at install time. The exfiltration domain was models.litellm.cloud, which looks legitimate
pip install --require-hashes would have passed because the .pth file was correctly declared in the wheel's RECORD

The only install-time defense that would have caught this is inspecting whether packages install .pth files containing subprocess, base64, or exec patterns. No widely deployed pip plugin does this automatically today.

The Bot Suppression Campaign

When researcher Callum McMahon at FutureSearch reported the compromise in GitHub issue #24512, TeamPCP used 73 previously compromised GitHub accounts to post 88 spam comments in 102 seconds, then closed the issue as "not planned" using the compromised maintainer account krrishdholakia.

76% of these accounts overlapped with the botnet used during the Trivy disclosure. This is documented in Wiz's analysis.

The Structural Problem Nobody Is Talking About

LiteLLM is a Python package. It sits between your application and your LLM providers. It holds API keys for OpenAI, Anthropic, AWS Bedrock, Google Vertex, and whatever else you route through it. It is, by design, a high-value target.

And it runs in a language where:

Any package can install .pth files that execute on interpreter startup
Transitive dependencies pull in dozens of packages you never explicitly chose
A single compromised CI/CD token can publish arbitrary code under a trusted package name
The GIL means you need to run multiple processes, each of which triggers .pth execution independently

This is not a criticism of Python as a language. Python is excellent for what it is good at. But the question is whether Python is the right choice for infrastructure that holds your most sensitive credentials and sits in the critical path of every LLM request.

Compiled languages like Go and Rust produce single binaries with no runtime dependency chain. There is no site-packages directory. There is no .pth execution mechanism. There is no pip install side effect. The attack surface is fundamentally smaller.

What You Should Do Right Now

1. Check Your LiteLLM Version

pip show litellm | grep Version

If you see 1.82.7 or 1.82.8, you were affected. LiteLLM's security update confirms these versions were compromised.

2. Scan for .pth Files

find $(python -c "import site; print(site.getsitepackages()[0])") -name "*.pth" -exec grep -l "subprocess\|base64\|exec" {} \;

3. Rotate Everything

If you ran the compromised version, assume all credentials on that machine are compromised:

AWS/GCP/Azure keys and service accounts
SSH keys
API keys (OpenAI, Anthropic, etc.)
Database passwords
Kubernetes service account tokens
CI/CD tokens

4. Pin Your Dependencies

Aider survived because it pinned litellm==1.82.3. Pin your versions. Better yet, pin by hash:

litellm==1.82.6 --hash=sha256:<known-good-hash>

5. Pin Your CI/CD Actions by SHA

Don't do this:

uses: aquasecurity/trivy-action@latest

Do this:

uses: aquasecurity/trivy-action@<full-commit-sha>

6. Evaluate Your Gateway Architecture

This is the harder conversation. If your LLM gateway is a Python package that you pip install, it shares the same supply chain as every other Python package on your system. Every transitive dependency is a potential attack vector.

Alternatives worth evaluating:

Bifrost: Open-source LLM gateway written in Go. Single compiled binary, 11 microsecond overhead at 5,000 RPS. No Python supply chain surface area. Supports OpenAI, Anthropic, Bedrock, Vertex, and 20+ providers.
TensorZero: Rust-based LLM gateway with sub-millisecond overhead. Similar compiled-binary benefits.
Cloudflare AI Gateway: Managed edge service. No self-hosted dependency chain at all.
Direct provider SDKs: If you only use one or two providers, you may not need a gateway at all. The official OpenAI and Anthropic SDKs are smaller attack surfaces.

The right choice depends on your scale, provider count, and security requirements. But "keep using the Python package that just got backdoored" should not be the default.

The Bigger Picture

TeamPCP is not done. They also deployed CanisterWorm, using the Internet Computer Protocol as a C2 channel. They used an AI agent called openclaw for automated attack targeting. Their target selection focuses on tools with elevated pipeline access: container scanners, infrastructure scanning tools, AI routing libraries.

LLM gateways are a perfect target. They hold credentials for multiple providers. They run in CI/CD environments. They have broad read access by design.

The question is not whether this will happen again. It is whether your infrastructure is designed to limit the blast radius when it does.

References:

Tags: litellm, supply-chain-attack, security, python, llm-gateway, ai-infrastructure, devops, cybersecurity

How to Set Up Weighted Load Balancing Across LLM Providers

Pranay Batta — Thu, 12 Mar 2026 12:25:13 +0000

TL;DR: Running all your LLM traffic through a single provider is a single point of failure. Weighted load balancing lets you split traffic across providers (say 70/30 GPT-4o/Claude), optimize for cost or latency per use case, and failover automatically when one provider goes down. Bifrost handles this at the gateway layer with 11 microsecond overhead. Here is how to set it up.

Why Single-Provider Is a Bad Idea

You have probably been here. Your entire app routes through OpenAI. One day, OpenAI hits capacity. Your 429 retry logic kicks in, but the retries also get 429'd. Your app is effectively down, and your only option is to wait.

This is not a hypothetical. Every major LLM provider has had multi-hour outages in the last 12 months. If your architecture assumes 100% availability from a single provider, your architecture is wrong.

The fix is not "add retry logic." The fix is routing traffic across multiple providers at the gateway layer, with weights you control.

What Weighted Load Balancing Actually Means

Instead of sending every request to one provider, you define a split:

70% of requests go to OpenAI (GPT-4o)
30% go to Anthropic (Claude Sonnet)

Or maybe:

50% to Gemini (cheaper for simple tasks)
50% to Anthropic (better for complex reasoning)

The gateway makes the routing decision per-request based on these weights. Your application code does not change. It sends requests to one endpoint (the gateway), and the gateway distributes them.

The key benefit: you change the weights in a config file, not in your application code. No redeploy. No code review. Just update the config and the traffic split changes immediately.

Setting This Up With Bifrost

Bifrost is an open-source LLM gateway written in Go. It sits between your app and LLM providers as a reverse proxy, adding 11 microseconds of overhead per request.

Here is a weighted routing config that splits traffic between OpenAI and Anthropic:

{
  "accounts": [
    {
      "id": "openai-primary",
      "provider": "openai",
      "api_key": "${OPENAI_API_KEY}",
      "weight": 70
    },
    {
      "id": "anthropic-secondary",
      "provider": "anthropic",
      "api_key": "${ANTHROPIC_API_KEY}",
      "weight": 30
    }
  ],
  "fallback_config": {
    "enabled": true,
    "on_status_codes": [429, 500, 502, 503]
  }
}

What this does:

70% of incoming requests route to OpenAI, 30% to Anthropic
If OpenAI returns a 429 or 5xx, the request automatically fails over to Anthropic
If Anthropic is also down, the request returns an error to the client
All of this happens at the gateway. Your app sends requests to http://localhost:8080/v1/chat/completions and does not know which provider handled it

Start the gateway with zero config:

npx -y @maximhq/bifrost

Then configure providers through the web dashboard at localhost:8080. No YAML, no environment variable chains. JSON config, web UI, done.

Three Load Balancing Strategies That Actually Work

1. Cost Optimization Split

Route simple tasks to the cheapest provider, complex tasks to the most capable.

{
  "accounts": [
    {
      "id": "gemini-flash-cheap",
      "provider": "gemini",
      "model": "gemini-2.5-flash",
      "weight": 60
    },
    {
      "id": "openai-capable",
      "provider": "openai",
      "model": "gpt-4o",
      "weight": 40
    }
  ]
}

Gemini Flash handles the bulk of requests at lower cost. GPT-4o handles the rest. You can check the per-model pricing on the model library to find the right split for your use case. If you want exact cost numbers before committing, the LLM cost calculator lets you compare across providers for your specific token volumes.

2. Latency-Optimized Split

If your app is latency-sensitive (chatbots, real-time agents), route more traffic to the fastest provider.

{
  "accounts": [
    {
      "id": "groq-fast",
      "provider": "groq",
      "model": "llama-3.3-70b-versatile",
      "weight": 50
    },
    {
      "id": "anthropic-quality",
      "provider": "anthropic",
      "model": "claude-sonnet-4-6",
      "weight": 50
    }
  ]
}

Groq is extremely fast for Llama models. Anthropic gives you better reasoning quality. 50/50 split means half your users get near-instant responses, half get higher quality. You tune the ratio based on what your users actually need.

For raw numbers on how much overhead each gateway adds, check the benchmarks page. Bifrost adds 11 microseconds. That is not a typo. Microseconds, not milliseconds.

3. Reliability-First Split

For production apps where uptime matters more than cost.

{
  "accounts": [
    {
      "id": "openai-primary",
      "provider": "openai",
      "weight": 40
    },
    {
      "id": "anthropic-secondary",
      "provider": "anthropic",
      "weight": 30
    },
    {
      "id": "gemini-tertiary",
      "provider": "gemini",
      "weight": 30
    }
  ],
  "fallback_config": {
    "enabled": true,
    "on_status_codes": [429, 500, 502, 503]
  }
}

Three providers. If any one goes down, traffic redistributes to the other two. The probability of all three being down simultaneously is near zero. This is the setup we recommend for anything that cannot afford downtime. The automatic failover guide walks through the full failover config in detail.

Adding Budget Controls to the Mix

Weighted routing becomes more powerful when combined with budget controls. You do not want one runaway team to blow through your entire OpenAI quota and leave other teams with nothing.

Bifrost has a four-tier budget hierarchy: Organization > Team > Virtual Key > Provider. Each level can have daily, weekly, or monthly caps.

{
  "virtual_keys": [
    {
      "id": "team-backend",
      "budget": {
        "monthly_limit_usd": 5000,
        "daily_limit_usd": 250
      },
      "rate_limit": {
        "request_max_limit": 1000,
        "request_reset_duration": "1h"
      }
    }
  ]
}

When a team hits their budget, requests can either fail (hard stop) or automatically route to a cheaper model (soft failover). This is configurable per virtual key.

If you are running Claude Code across a dev team, this is especially useful. Each developer gets a virtual key with their own budget. No one developer can burn through the team's allocation.

The Provider-Isolation Architecture

Here is something most gateways get wrong: they use a single request queue for all providers. When OpenAI starts rate limiting you and requests back up, the queue fills. Now your Anthropic and Gemini requests are also stuck behind the OpenAI backlog.

Bifrost uses provider-isolated worker pools. Each provider gets its own queue. OpenAI being slow does not affect Anthropic latency at all. This matters a lot under weighted load balancing, because the whole point is that you have multiple providers and they should operate independently.

Your App → Bifrost Gateway → [OpenAI Pool]    → OpenAI
                           → [Anthropic Pool]  → Anthropic
                           → [Gemini Pool]     → Gemini

Backpressure policies are configurable per provider: drop (discard), block (wait), or error (fail fast). So if OpenAI's pool fills up, you can choose to fail fast and let the failover logic route to the next provider, instead of waiting.

Setting Up With Your Stack

Bifrost is provider-agnostic on the client side. It exposes an OpenAI-compatible API, so any client that speaks OpenAI format works out of the box.

Python (OpenAI SDK):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-bifrost-virtual-key"
)

Claude Code:

export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
# Now Claude Code routes through Bifrost to any configured provider

Any OpenAI-compatible client:
Just change the base URL to point at your Bifrost instance. That is it. The multi-provider setup guide covers the full configuration for all 19 supported providers.

If you are using tools like Zed editor, LibreChat, or Gemini CLI, there are specific integration guides for each.

When Not to Use Weighted Load Balancing

Being honest here. Weighted routing adds complexity. If you are:

Running a prototype or hobby project: just use one provider directly
Under 100 requests per day: the reliability benefit is not worth the setup
Using only one model for a very specific task: routing adds no value

Weighted load balancing makes sense when you are in production, handling real traffic, and need either cost optimization, reliability guarantees, or both.

Getting Started

# Start Bifrost (zero config)
npx -y @maximhq/bifrost

# Open dashboard
open http://localhost:8080

# Configure providers and weights through the UI
# Or edit the JSON config directly

19 providers supported out of the box. You can compare available models and pricing on the model library before deciding your split. If you are evaluating gateways in general, the buyer's guide covers what to look for.

Source code: GitHub | Docs

We maintain Bifrost at Maxim AI. It is open-source, MIT licensed, and free to self-host. If you run into issues, open an issue on GitHub or check the docs.

DEV Community: Pranay Batta

MCP at Scale: Access Control, Cost Governance, and 92% Lower Token Costs

The Hidden Tax on Every MCP Request

I Tested Bifrost's Code Mode Approach

Benchmark Results: 3 Controlled Rounds

Access Control That Actually Works

Per-Tool Observability

Connection Flexibility

Honest Trade-offs

Who Should Care

Try It

How to Track LLM Costs and Rate Limits on AWS Bedrock with an AI Gateway

The Problem with AWS Native Cost Tracking

The Gateway Approach

Setting Up Bifrost with AWS Bedrock

Four-Tier Budget Hierarchy

Rate Limiting That Actually Works for LLMs

Observability at Sub-Millisecond Overhead

Honest Trade-offs

When to Use What

Quick Start

Best Claude Code Gateway for Multi-Model Routing

Why Multi-Model Routing Matters

The Setup: Bifrost as a Claude Code Gateway

Install and Connect

Weighted Routing Configuration

Automatic Failover

Model Pinning for Bedrock and Vertex AI

Budget Controls Across Providers

Observability Across All Providers

Honest Trade-offs

Performance

Quick Start Summary

Best MCP Gateway for 50% Token Cost Savings

The Problem with Classic MCP

How Bifrost Code Mode Changes This

What Code Mode Actually Does

The Latency Numbers

Agent Mode: The Other Side

Deployment

Who Should Use Code Mode

How to Connect Any Model with Gemini CLI Using Bifrost AI Gateway

The Problem with Single-Provider CLI Tools

How Bifrost Works with Gemini CLI

Setup

Step 1: Install Bifrost

Step 2: Configure Providers

Step 3: Point Gemini CLI to Bifrost

What You Get

Multi-Provider Routing

Automatic Failover

Budget Controls

Cost Tracking

Performance

The Overhead Question

When This Makes Sense

Links

Top 5 Enterprise AI Gateways for Dynamic Routing in 2026

Why dynamic routing matters

The five gateways I tested

1. Bifrost

2. LiteLLM

3. Kong AI Gateway

4. Cloudflare AI Gateway

5. Azure API Management

Comparison table

Honest trade-offs

Which one should you pick

How to Connect Non-Anthropic Models to Claude Code with Bifrost AI Gateway

The Problem

What Bifrost Does

Setup: 3 Minutes

Step 1: Install

Step 2: Configure a Provider

Step 3: Point Claude Code at Bifrost

Multi-Provider Routing

Automatic Failover

Bedrock and Vertex AI

Features Worth Mentioning

Budget Enforcement