DEV Community: Kamya Shah

Claude Code token usage is the input, output, and cache tokens each session sends to Claude, and it drives the bill. This guide compares native tracking with gateway monitoring and shows how Bifrost attributes tokens and cost per developer, exports metrics

Kamya Shah — Tue, 06 Oct 2026 09:36:06 +0000

Kamya Shah

Oct 6

Claude Code Token Usage: Monitor It with an AI Gateway

#claude #ai #apigateway #llm

13 min read

Claude Code Token Usage: Monitor It with an AI Gateway

Kamya Shah — Tue, 06 Oct 2026 09:35:48 +0000

TL;DR

Claude Code's /usage command (aliased as /cost and /stats) reports token usage and an estimated cost for one session on one machine; it cannot attribute spend across a team or stop a request.
Claude Code's OpenTelemetry export streams claude_code.token.usage and claude_code.cost.usage per user, but only from machines that carry the telemetry settings, and it enforces nothing.
An AI gateway on the request path records tokens and cost for every Claude Code request, tags it to a developer and team, and works the same whether Claude is served by Anthropic, Amazon Bedrock, or Vertex AI.
Bifrost meters Claude Code through per-developer virtual keys, exports Prometheus metrics and OpenTelemetry spans, and blocks requests with a 402 once a developer, team, or customer budget is spent.
Connecting Claude Code takes two settings: ANTHROPIC_BASE_URL pointed at Bifrost and ANTHROPIC_AUTH_TOKEN set to a Bifrost virtual key.

Claude Code token usage grows with every turn of a session because Claude Code resends the full conversation on each request, and across enterprise deployments it averages around $13 per developer per active day and $150 to $250 per developer per month, according to Anthropic's Claude Code cost guide. Bifrost, the open-source AI gateway built in Go by Maxim AI, sits between Claude Code and the model provider and records tokens, cost, model, and developer for every request. This guide covers what Claude Code reports natively, where those tools stop at team scale, and how to monitor, report on, and cap Claude Code token usage with Bifrost.

How to Check Claude Code Token Usage Today

Claude Code token usage can be checked in four native places: the /usage command, the status line, OpenTelemetry export, and the Claude Console or Claude admin analytics. Each answers a different question, and none of them attributes spend across every provider while also enforcing a per-developer budget at request time.

The table summarizes each option, with an AI gateway such as Bifrost listed last for comparison.

Where to check	What it shows	Scope	Enforces a budget
`/usage` (also `/cost`, `/stats`)	Session tokens per model, cache reads and writes, estimated cost at list price	One session, one machine	No
Status line cost field	Running session cost	One session	No
Claude Code OpenTelemetry	`claude_code.token.usage` by type (input, output, cacheRead, cacheCreation), `claude_code.cost.usage` in USD	Each machine that has telemetry enabled	No
Claude Console or Claude admin analytics	Spend per member, workspace or seat usage	Anthropic-billed usage only	Workspace or organization spend limits
Bifrost AI gateway	Tokens, cost, latency, model, provider per request, per virtual key	Every request routed through it, any provider	Yes, per virtual key, team, and customer

The /usage Session block is the fastest way to see Claude token usage on a single machine. Claude Code computes its dollar figure locally from token counts at list price, so Anthropic describes it as an estimate and points to the Console for authoritative billing. On Pro, Max, Team, and Enterprise plans, the plan usage breakdown in the same screen is computed from local session history, so usage from other devices is not included.

Claude Code telemetry goes further. Setting CLAUDE_CODE_ENABLE_TELEMETRY=1 and an OTLP or Prometheus exporter makes each machine emit token and cost metrics tagged with user.account_uuid, session.id, and model, as described in Claude Code's monitoring guide. Coverage depends on every laptop and CI runner receiving those settings, and the cost metric is an estimate computed on the client. For background on why a request-path layer behaves differently, the pillar guide on AI gateway architecture and why it matters covers the model in depth.

Why Claude Code Usage Is Hard to Track Across a Team

Claude Code usage is hard to track at team scale because cost per task varies by orders of magnitude, native reports are per machine or per billing surface, and the same developer may reach Claude through Anthropic, Bedrock, or Vertex AI. A platform team needs one ledger that covers all of it.

Three mechanics drive the variance, all documented by Anthropic:

Context is resent on every request. A one-line question late in a long session still carries the whole conversation, read at the cached-token rate when the cache is warm and at the full input rate after a cache miss.
Parallel work multiplies tokens. Subagents send their own requests, and agent teams use approximately 7x more tokens than standard sessions when teammates run in plan mode.
Billing surfaces split. Anthropic's analytics dashboards and the Claude Code Analytics API do not cover Claude Code usage billed through Amazon Bedrock, Google Cloud, or Microsoft Foundry.

Native tools alone cannot tell an engineering manager which developers drive the bill, what share goes to Opus, or which team will overrun its allocation. Anthropic's own LLM gateway guidance lists usage tracking by developer or team, regardless of which provider serves the request, as a core reason to put a gateway in front of Claude Code. The related guide on how teams manage Claude Code usage limits covers how plan limits interact with gateway limits.

What a Claude Code Usage Monitor Needs to Capture

A Claude Code usage monitor for a team needs per-request token counts and cost, attribution to a developer and team, a model and provider breakdown, export into existing monitoring tools, and enforcement on the same path. Without the last two, a monitor produces a report that arrives after the money is spent.

Requirement	Why it matters for Claude Code	Native tools	Bifrost
Tokens and cost per request	Long sessions and subagents hide inside daily totals	Per session or per day	Every request in request logs
Developer and team attribution	Spend reviews happen per team	Per user where telemetry or Console billing applies	Per virtual key, team, and customer
Provider coverage	Claude may be served by Anthropic, Bedrock, or Vertex AI	Split by billing surface	One view across providers
Standard export	Data belongs in Grafana or Datadog	OpenTelemetry from each machine	Prometheus and OTLP from the gateway
Enforcement	A cap must stop the next request	Workspace or org spend limits	Hierarchical budgets and rate limits

A Claude Code token usage monitor needs a credential per developer for attribution and a control point on the request path for enforcement. This is the core job of an AI gateway placed in front of model traffic, and Bifrost is built on exactly that pairing: virtual keys for attribution, budgets for enforcement.

How Bifrost Monitors Claude Code Token Usage

The Bifrost AI gateway monitors Claude Code token usage by receiving every request on its Anthropic-compatible endpoint, resolving the developer's virtual key, forwarding the request to the configured provider, and recording tokens, cost, and latency against that key. Because measurement happens on the request path, no developer machine needs its own telemetry pipeline.

Figure 1: Token and cost data is captured on the request path, so attribution does not depend on telemetry settings on each laptop.

As Figure 1 shows, three outputs come from the same request stream:

Built-in request logs and dashboard. Every request is logged with provider, model, input and output tokens, cost, latency, and status. The dashboard at http://localhost:8080 adds token and cost analytics, and the logs API filters by virtual key, model, token range, cost range, and time window.
Prometheus metrics. The /metrics endpoint exposes bifrost_input_tokens_total, bifrost_output_tokens_total, and bifrost_cost_total, labeled with virtual_key_name, team_name, customer_name, model, and provider. Multi-node deployments push to a Prometheus Push Gateway instead.
OpenTelemetry spans. The OTel plugin emits GenAI semantic-convention spans carrying gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, and gen_ai.usage.cost to Grafana Cloud, New Relic, Honeycomb, or any OTLP collector.

Bifrost covers Claude wherever it is served. The same virtual key can route Claude Code to Anthropic directly or to Claude models on AWS Bedrock through Bifrost and Vertex AI, and every one of those requests lands in the same ledger. The sub-hub guide on choosing an AI gateway for Claude Code covers routing, model aliasing, and the MCP gateway beyond monitoring. Teams also running OpenAI's agent can apply the same approach to Codex CLI token spend.

What Bifrost Records for Each Claude Code Request

For each Claude Code request, Bifrost resolves the virtual key, checks every applicable budget and rate limit, calls the provider, and then computes cost from token counts against its pricing catalog. Logging runs asynchronously, so recording adds no wait to the developer's session.

Figure 2: Enforcement happens before the provider call and recording happens after it, so blocked requests cost nothing and allowed ones are always attributed.

Cost accuracy depends on pricing. The model catalog syncs a pricing sheet every 24 hours by default, and custom pricing overrides catalog rates at global or virtual key scope, so negotiated Anthropic or Bedrock rates show up in reports. Native Claude Code figures use list price unless an administrator distributes a modelPricing managed setting.

Claude Code sends an x-claude-code-session-id header on every request, and session affinity keeps that session on the provider key that served it, so prompt cache hits continue. Any request header prefixed x-bf-lh- is copied into log metadata, which adds dimensions such as repository or cost center.

The request path stays fast. Bifrost adds 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks, and the logging plugin adds under 0.1 ms, which is negligible next to model response time.

Building a Claude Code Usage Dashboard and Token Usage Report

A Claude Code usage dashboard is built by scraping Bifrost's Prometheus metrics into Grafana, or by querying the logs API for a periodic Claude Code token usage report. Both paths use labels Bifrost already attaches, so no instrumentation is added to Claude Code.

Three PromQL queries cover most weekly reviews:

# Claude Code spend by team over the last 30 days
sum by (team_name) (increase(bifrost_cost_total[30d]))

# Input tokens by developer key and model over 7 days
sum by (virtual_key_name, model) (increase(bifrost_input_tokens_total[7d]))

# Requests refused because a budget was spent
sum by (virtual_key_name) (increase(bifrost_error_requests_total{error_type="policy_budget_exceeded"}[1d]))

For a written report, the logs API returns a stats object with total_requests, total_tokens, and total_cost for any filter, which makes a monthly per-developer export a single scheduled call:

curl 'http://localhost:8080/api/logs?virtual_key_ids=<vk-id>&start_time=2026-09-01T00:00:00Z&end_time=2026-09-30T23:59:59Z&limit=100'

Prometheus metrics and labels suit dashboards and spend-rate alerts. The OpenTelemetry plugin suits teams that want Claude Code traffic beside application traces, and Bifrost Enterprise adds a native Datadog connector.

Deeper patterns are covered in the guides to Prometheus dashboards for LLM traffic and OpenTelemetry traces and metrics for LLMs.

Controlling Claude Code Cost with Budgets and Alerts

Claude Code cost is controlled in Bifrost with hierarchical budgets on virtual keys, teams, and customers, plus token and request rate limits on virtual keys. Every applicable budget must pass before a request runs, and a spent budget returns HTTP 402 to Claude Code.

Figure 3: A runaway session exhausts the developer budget first, long before it can drain the team or organization allocation.

The budget model has three properties that matter for Claude Code:

Independent checks. Provider config, virtual key, team, and customer budgets are each checked, and the request cost is deducted from all of them.
Reset windows. Budgets reset on 1d, 1w, 1M, 1Q, or 1Y; setting calendar_aligned snaps them to calendar boundaries in UTC so they match finance periods.
Rate limits where bursts happen. Token and request limits apply at the virtual key and provider config levels, which is where a subagent fan-out shows up first. Teams and customers carry budgets only.

Bifrost Enterprise adds alerting on top: CEL rules such as budget_usage_percent > 80 are evaluated every 60 seconds against virtual key, team, or customer scopes, and notifications go to Slack, Microsoft Teams, PagerDuty, or a webhook. The governance resource page covers how budgets, RBAC, and SSO fit together, and the guide to governing Claude Code token usage per team walks through team structures in detail.

Rolling Out Claude Code Usage Monitoring

Rolling out Claude Code usage monitoring takes five steps: run Bifrost with provider keys, create teams and one virtual key per developer, distribute Claude Code settings, connect an exporter, then add budgets and alerts. Attribution begins with the first request carrying a virtual key.

Figure 4: Attribution starts at step three, the moment a developer's Claude Code sends its first request with a virtual key.

Start the Bifrost gateway with npx -y @maximhq/bifrost or the maximhq/bifrost Docker image and add an Anthropic, Bedrock, or Vertex key. Then create a virtual key per developer, attached to a team:

curl -X POST http://localhost:8080/api/governance/virtual-keys \
  -H "Content-Type: application/json" \
  -d '{
    "name": "claude-code-alice",
    "team_id": "team-platform",
    "calendar_aligned": true,
    "provider_configs": [
      {"provider": "anthropic", "weight": 1.0, "allowed_models": ["claude-sonnet-4-6", "claude-haiku-4-5", "claude-opus-4-8"], "key_ids": ["*"]}
    ],
    "budgets": [{"max_limit": 150.00, "reset_duration": "1M"}],
    "rate_limit": {"token_max_limit": 2000000, "token_reset_duration": "1h", "request_max_limit": 300, "request_reset_duration": "1m"}
  }'

Each developer's Claude Code then needs two values in the env block of settings.json, as shown in the Claude Code integration guide:

"env": {
  "ANTHROPIC_BASE_URL": "http://localhost:8080/anthropic",
  "ANTHROPIC_AUTH_TOKEN": "<bifrost-virtual-key>"
}

With ANTHROPIC_AUTH_TOKEN, no Anthropic account login is needed, and billing follows the virtual key. Since Claude Code 2.1.212, add the required Anthropic headers (including anthropic-version) to Bifrost's allowed headers under Client Settings. At fleet scale, distribute the base URL through Claude Code managed settings; setting allowedProviders to ["customEndpoint"] there makes Claude Code refuse sessions pointed anywhere other than the gateway. Developers who prefer a guided setup can run the Bifrost CLI with npx -y @maximhq/bifrost-cli, which configures Claude Code and stores the virtual key in the OS keyring.

Bifrost supports 25+ providers and 10,000+ models, and Bifrost Enterprise adds in-VPC deployment, clustering, RBAC, and SSO for regulated teams.

Teams pairing monitoring with reduction can follow the guide to reducing Claude Code token costs, and teams evaluating cheaper model tiers can review gateways that run Claude Code with non-Anthropic models.

Frequently Asked Questions

Is there a way to track Claude Code usage?

Yes. For one session, run /usage inside Claude Code to see tokens per model and an estimated cost. For a team, enable Claude Code's OpenTelemetry export on every machine, use the Console or admin analytics for Anthropic-billed usage, or route Claude Code through an AI gateway such as the Bifrost gateway, which records tokens and cost for every request by virtual key across Anthropic, Bedrock, and Vertex AI.

How is Claude Code usage measured?

Claude Code usage is measured in API tokens, split into input, output, cache read, and cache creation tokens, and converted to cost using per-model rates. Subscription plans also measure usage against a rolling five-hour window and a weekly window. Bifrost measures input and output tokens and cost per request at the gateway, so the figure reflects what actually passed through to the provider.

How do I check Claude Code token usage?

Run /usage (or its aliases /cost and /stats) to see the current session's input, output, and cache tokens by model with an estimated cost at list price. Running /clear resets those totals. For history across sessions and developers, query Bifrost's logs API or dashboard, which filter Claude Code requests by virtual key, model, cost, and time range.

What does token usage mean?

Token usage is the number of tokens a model reads and writes for a request. Input tokens cover the prompt, conversation history, files, and tool results Claude Code sends; output tokens cover the response, including thinking tokens. Because Claude Code resends the conversation each turn, Claude Code token usage grows with session length even when each new prompt is short.

How much does Claude Code cost?

Anthropic reports an average of around $13 per developer per active day and $150 to $250 per developer per month across enterprise deployments, with 90% of users below $30 per active day. Actual Claude Code cost depends on model choice, session length, codebase size, and use of subagents or agent teams, which is why per-developer budgets are usually set from a measured pilot baseline.

Does routing Claude Code through Bifrost change how developers work?

No. Developers keep the same commands, models, and workflow; only ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN change in settings.json. Session affinity keeps prompt caching intact, and Bifrost adds 11 microseconds of overhead per request. The visible difference is that a spent budget returns an error instead of continuing to bill.

Start Monitoring Claude Code Token Usage with Bifrost

Monitoring Claude Code token usage at team scale requires a layer that sees every request, attributes it to a developer and team, exports it to existing dashboards, and stops spend at a defined limit. Bifrost does all four with two settings per developer and no change to how Claude Code is used. The Bifrost Claude Code resource page and the complete Claude Code gateway guide cover routing and governance beyond usage monitoring.

To see how Bifrost can centralize Claude Code usage monitoring and cost control for your engineering organization, book a demo with the Bifrost team.

Codex token usage is the input, cached input, output, and reasoning tokens that Codex CLI consumes as it reads code, calls tools, and edits files. This guide shows how to check it, what drives Codex cost, and how Bifrost adds per-developer budgets, routing

Kamya Shah — Tue, 06 Oct 2026 08:32:15 +0000

Kamya Shah

Oct 6

Codex Token Usage: Track and Cap Spend with an AI Gateway

#codex #ai #gateway #llmcost

13 min read

Codex Token Usage: Track and Cap Spend with an AI Gateway

Kamya Shah — Tue, 06 Oct 2026 08:32:01 +0000

TL;DR

Codex token usage grows with context size, agent turns, reasoning effort, and model choice, so similar tasks can consume very different amounts of tokens.
Codex CLI's /status and /usage commands show token activity for one developer; neither attributes spend across a team or blocks a request when a budget runs out.
Bifrost enforces budgets at the provider config, virtual key, team, and customer levels, and rejects a Codex CLI request when any applicable budget is exhausted.
Bifrost adds 11 microseconds of overhead per request at 5,000 requests per second, so budget checks do not slow interactive Codex CLI sessions.
Connecting Codex CLI takes a /logout, an exported virtual key, and a named model_providers entry in ~/.codex/config.toml.

Codex token usage is the count of input, cached input, output, and reasoning tokens that OpenAI's Codex CLI consumes while it reads a repository, calls tools, and edits files, and it compounds quickly once a whole engineering team runs the agent daily. Bifrost, the open-source AI gateway built in Go by Maxim AI, sits between Codex CLI and the model provider to attribute every token to a developer, enforce budgets before spend happens, and log cost per request. This guide covers how to check Codex token usage, how Codex cost and usage limits work, and how to configure Bifrost so Codex CLI spend stays predictable.

What Drives Codex Token Usage

Four factors drive Codex token spend: the context the agent reads, the number of agent turns, reasoning effort, and the model's per-token rate. OpenAI's Codex pricing page states that model choice, context, reasoning, tool use, retrieval, and caching all affect usage, so prompt length alone is not a reliable estimate.

Repository context: files, chat history, and tool results are sent as input tokens, far more than the prompt the developer typed.
Agentic turns: plan, edit, test, and fix steps are separate requests that each replay the growing conversation.
Reasoning effort: higher effort produces more reasoning tokens, billed as output.
Model rate: on OpenAI's current Codex credit rate card, GPT-6 Astra costs 250 credits per million input tokens and 1,250 per million output tokens, while GPT-6 Luna costs 2.5 and 12.5. That is a 100x spread for the same token count.

Model choice is the factor a team controls most directly, which is why a shared control point matters. The pillar guide to AI gateway architecture explains that pattern for any set of clients and models.

How to Check Codex Usage

Codex CLI exposes token usage through two slash commands. /status displays session configuration and token usage, including the active model and remaining context capacity. /usage shows daily, weekly, or cumulative ChatGPT token activity. Both commands report on one developer's session or account and cannot stop a request.

OpenAI documents both on the Codex CLI slash commands page, and ChatGPT-plan users also get a usage dashboard with reset times. Codex CLI can export OpenTelemetry metrics, including a turn.token_usage metric split by input, cached input, output, and reasoning tokens; the export is off by default and enabled with an [otel] block.

Method	What it shows	Attributes spend across a team	Blocks requests at a limit
`/status` in Codex CLI	Session model, config, and token usage	No	No
`/usage` in Codex CLI	Daily, weekly, or cumulative ChatGPT token activity	No	No
ChatGPT usage dashboard	Plan limits and reset times for one account	No	No (plan limits apply)
Codex OpenTelemetry export	Per-turn token counts by type	Per machine, if every machine is configured	No
Bifrost request logs	Tokens, cost, latency, model, and virtual key per request	Yes	Yes, through budgets and rate limits

The difference in the last row is position. Bifrost sits on the request path, so its request logs count Codex token usage for every developer in one place, and its budgets reject the next request when a limit is reached. Client-side methods only report after the fact.

Codex Cost and Usage Limits

Codex cost depends on how a developer signs in. With a ChatGPT plan, Codex usage draws on the plan's included allowance and then on credits. With an API key, Codex usage is billed at standard API pricing for the model the key can access, with no plan allowance and no five-hour window.

OpenAI's Codex plans and rate card list the current options:

Access option	Price (US)	How Codex usage is limited
Free	$0/month	Basic access on GPT-6 Luna
Go	$8/month	Lightweight coding tasks
Plus	$20/month	Local messages per five-hour period, weekly limits may also apply, then credits
Pro	From $100/month ($100, $200, or $500 tiers)	No five-hour limit currently; higher allowance, then credits
Business	$20/user/month billed annually, $25 billed monthly	Five-hour allowance on Standard Business, workspace credits
Enterprise / Edu	Contact sales	Flexible pricing plans have no fixed rate limits
API key	Standard API pricing	Billed per token; models follow what the key can access; no cloud features such as GitHub code review or Slack

Two details matter for Codex usage limits planning. Codex credit billing has no separate cache-write charge, and GPT-5.5 retires from Codex on all ChatGPT plans on October 14, 2026, so budgets built on GPT-5.5 rates need updating.

For Codex CLI pricing in API-key mode, which is how teams route Codex through a gateway, the bill is every request's tokens at the provider's rate. The OpenAI model cost calculator gives a per-million-token baseline before rollout.

Why Codex Token Usage Needs Team-Level Controls

Codex token usage needs team-level controls because every built-in view is scoped to one developer, while the bill is scoped to the organization. Without a shared control point, a platform team cannot see which developer or model drives Codex cost, or stop a runaway agent loop before the invoice arrives. Teams typically try one of four approaches:

Approach	Visibility	Enforcement	Limitation
ChatGPT seats per developer	Per-account dashboard	Plan limits only	No budget per project; limits are set by OpenAI, not the team
One shared OpenAI API key	Organization-level billing	Provider rate limits only	No per-developer attribution
Separate API keys per developer	Per-key billing	Manual key revocation	Key sprawl; no model or provider policy
AI gateway with virtual keys	Per request, per developer, per model	Budgets, rate limits, routing rules	Requires running a gateway

Figure 1: Codex token usage becomes attributable and enforceable once every developer's traffic shares one gateway.

As Figure 1 shows, the Bifrost AI gateway gives each developer or CI job its own virtual key, so every Codex CLI request carries an identity the gateway can bill and limit. The same pattern applies to other coding agents; the guide to governing Claude Code and Codex CLI with one gateway covers running both under shared policy.

Setting Budgets and Rate Limits on Codex CLI with Bifrost

Bifrost enforces Codex CLI spend through a budget hierarchy: customer, team, virtual key, and provider config. When a request arrives, Bifrost checks every applicable budget independently, and any single exhausted budget blocks the request. The request's cost is then deducted from every level that applies.

Figure 2: One exhausted budget at any level stops the request, and its cost is deducted from every level that applies.

The budget and limits system maps onto a Codex rollout like this:

Level	Typical Codex CLI use	Budget	Rate limits
Customer	A business unit or external client	Yes	No
Team	Platform or frontend team monthly allocation	Yes	No
Virtual key	One developer, service account, or CI job	Yes	Requests and tokens
Provider config	Separate OpenAI and Anthropic spend on one key	Yes	Requests and tokens

Budgets reset on 1d, 1w, 1M, 1Q, or 1Y windows; with calendar_aligned: true, a monthly Codex budget resets on the first of the month in UTC. Rate limits use token_max_limit and request_max_limit with their own windows, such as one million tokens per hour on a provider config, which caps burst spend from an agent stuck in a retry loop.

A temporary budget override adds capacity for a set number of reset cycles without changing the base limit, which covers a release week. Bifrost Enterprise adds budget alerts to Slack, Microsoft Teams, PagerDuty, or a webhook when a budget crosses a threshold.

The Bifrost governance overview shows how these levels fit together, and the write-up on hierarchical spend controls with virtual keys covers budget design for larger organizations.

Cutting Codex Cost with Caching and Routing

Budgets cap Codex cost; caching and routing lower it. Bifrost reduces spend in three ways: it keeps provider prompt caches warm with session affinity, it can inject prompt-cache markers that Codex does not send, and it routes requests to lower-cost models when a provider budget nears its limit.

Session affinity and prompt caching

Codex CLI sends a session-id header on every request. Bifrost adopts that header for session affinity, so a session stays on the provider and API key that served it and keeps reading the same provider prompt cache, with nothing to configure. On the Codex credit rate card, cached input for GPT-6 Astra costs 25 credits per million tokens against 250 for fresh input.

Codex also sends no prompt-cache markers, so on Anthropic models nothing is cached and every turn pays full input price. Bifrost's auto prompt caching adds the marker on the first cacheable block for requests that arrive without one, so turn one writes the cache and later turns read it. Auto injection is off by default and is enabled per provider.

Semantic caching, with its limits

Bifrost semantic caching replays a stored response for an identical or similar request, so the provider is never called, and it covers the Responses API that Codex CLI uses. Caching engages only when a request carries an x-bf-cache-key header or a default_cache_key is set, and conversation_history_threshold (default 3) skips conversations with more messages than that. Long Codex sessions usually exceed it, so semantic caching suits short, repeated requests such as scripted CI prompts. The guide to semantic caching for cutting token spend covers tuning.

Cost-aware routing rules

Routing rules evaluate a CEL expression on each request. The budget_used variable reports the percentage consumed of the budget configured for the request's provider and model, including a provider config on the virtual key. A rule such as budget_used > 85, scoped to a team or virtual key, redirects Codex traffic to a lower-cost model before the hard cap is reached.

Figure 3: Shifting to a lower-cost model near the threshold keeps developers working instead of hitting a hard stop.

Routing to a different model family works because Bifrost translates Codex CLI's Responses API calls to 25+ providers and 10,000+ models in the supported providers matrix. The guide on how to route Codex CLI to any model covers provider choice and tool-calling requirements in detail.

Codex sessions that call MCP tools also pay for tool definitions on every request. The analysis of how the MCP gateway cuts token costs in Claude Code and Codex CLI quantifies that overhead and the effect of tool filtering per virtual key.

Codex CLI Config for Bifrost

Connecting Codex CLI to Bifrost takes five steps: start the gateway, run /logout in Codex CLI, export a Bifrost virtual key as OPENAI_API_KEY, add a named Bifrost provider to config.toml, and launch codex. The logout comes first because Codex CLI always prefers an existing OAuth session over a custom API key.

Figure 4: The logout step comes first because Codex CLI prefers an existing OAuth session over the gateway key.

Start the open-source Bifrost gateway with npx -y @maximhq/bifrost or docker run -p 8080:8080 maximhq/bifrost, as described in the gateway setup guide. Then, inside Codex CLI, run /logout, exit, and export the virtual key in the same terminal you will launch Codex from:

export OPENAI_API_KEY=<bifrost_virtual_key>

Add a named provider to ~/.codex/config.toml (or a project-level .codex/config.toml):

model = "openai/gpt-5.4"
model_provider = "bifrost"

[model_providers.bifrost]
name = "Bifrost"
base_url = "http://localhost:8080/openai/v1"
env_key = "OPENAI_API_KEY"
wire_api = "responses"
supports_websockets = false

Use a named model_providers entry rather than openai_base_url: with openai_base_url, Codex keeps its built-in OpenAI provider and sends a client-side web namespace tool that Amazon Bedrock rejects. supports_websockets = false keeps Codex CLI on HTTPS, which non-OpenAI models require. The full reference is on the Codex CLI integration page.

Models use the provider/model format, either at launch or mid-session with /model:

codex --model openai/gpt-5-codex
codex --model anthropic/claude-sonnet-4-5-20250929
codex --model gemini/gemini-2.5-pro

Non-OpenAI models must support tool calling, which Codex CLI relies on for file and terminal operations. To skip manual edits, the Bifrost CLI (npx -y @maximhq/bifrost-cli) sets OPENAI_BASE_URL to the gateway's /openai/v1 path, stores the virtual key in the OS keyring, and launches Codex CLI; the walkthrough on using Bifrost CLI with coding agents shows the flow.

Monitoring Codex Token Usage Across a Team

Bifrost logs every Codex CLI request with input and output tokens, cost, latency, provider, model, and virtual key. Logging runs asynchronously, so it adds no latency, and logs filter by model, provider, token range, or cost range.

The built-in request logs answer questions such as which developer ran the most expensive session this week. For dashboards, Bifrost exposes Prometheus metrics including bifrost_input_tokens_total, bifrost_output_tokens_total, and bifrost_cost_total, labeled by provider, model, and virtual_key_id, so a Grafana panel can chart Codex cost per developer. Traces can also go to any collector through OpenTelemetry.

Cost figures come from a model catalog that syncs provider pricing every 24 hours, and custom pricing applies negotiated rates. Bifrost benchmarks show 11 microseconds of overhead per request at 5,000 RPS with a 100% success rate, so none of this tracking slows a session.

Teams running both coding agents can compare this setup with the guide on monitoring Claude Code token usage, and the Claude Code gateway selection guide covers the Anthropic side of the rollout.

Frequently Asked Questions

How do I check Codex token usage?

Run /status in Codex CLI to see the session's model, configuration, and token usage, or /usage to see daily, weekly, or cumulative ChatGPT token activity. ChatGPT-plan users can also check the Codex usage dashboard for remaining limits. These views cover one developer. For team-wide Codex token usage by developer and model, route Codex CLI through Bifrost and read its per-request token logs or Prometheus metrics.

Is Codex pay to use?

Codex is included in ChatGPT Free, Go, Plus, Pro, Business, Edu, and Enterprise plans, with allowances that differ by plan, and usage beyond the allowance draws on credits. With an OpenAI API key, every request is billed at standard API pricing and cloud features such as GitHub code review are unavailable.

How much does Codex CLI cost?

Codex CLI cost follows the sign-in method. On a ChatGPT plan, it is the plan price plus any credits used past the included allowance; Plus is $20 per month and Pro starts at $100 per month. With an API key, Codex CLI cost is the per-token rate of the chosen model multiplied by the input, cached input, and output tokens each session consumes.

What happens when you hit Codex usage limits?

When a ChatGPT-plan user reaches a Codex usage limit during an active turn, the agent can finish that turn, subject to fair use. After that, Plus and Pro users can buy additional credits, and Business or Enterprise workspaces with flexible pricing can buy workspace credits. Limits reset on the schedule shown in the usage dashboard, including five-hour windows on Plus.

Can I set a spending limit on Codex CLI?

Yes. Routing Codex CLI through Bifrost lets a team attach budgets to each developer's virtual key, a team, a customer, and each provider config. When any applicable budget is exhausted, Bifrost rejects further requests until the budget resets on its daily, weekly, monthly, quarterly, or yearly window.

Can Codex CLI use non-OpenAI models through Bifrost?

Yes. Bifrost translates Codex CLI's Responses API requests to other providers, so codex --model anthropic/claude-sonnet-4-5-20250929 runs under the same virtual key and budget. The model must support tool calling, and Codex CLI must use HTTPS rather than WebSocket mode.

Start Managing Codex Token Usage with Bifrost

Codex token usage scales with every developer, agent turn, and model upgrade, while built-in Codex views report it per account. Bifrost, as the AI gateway layer in front of every model call, turns that usage into attributed, budgeted, and logged traffic through virtual keys, budgets, routing, and prompt-cache handling. For multi-model Codex setups, see the guide to running Codex CLI with Claude, Gemini, or local models.

To put budgets and spend visibility on Codex CLI across your engineering organization, book a demo with the Bifrost team.

A Claude Code gateway authenticates each developer, enforces budgets, and routes Claude Code traffic to Anthropic, Amazon Bedrock, or Vertex AI from one endpoint. This guide covers what to evaluate, how to configure Claude Code with Bifrost, and how to rol

Kamya Shah — Tue, 06 Oct 2026 08:30:45 +0000

Kamya Shah

Oct 6

Claude Code Gateway: A Complete Guide to Choosing One

#claude #ai #llm #apigateway

13 min read

Claude Code Gateway: A Complete Guide to Choosing One

Kamya Shah — Tue, 06 Oct 2026 08:30:34 +0000

TL;DR

A Claude Code gateway sits between Claude Code and model providers so developers hold gateway credentials instead of provider keys, and usage, budgets, and audit trails live in one place.
The deciding criteria are forwarding fidelity (streaming, beta headers, error bodies, prompt caching), governance depth, provider coverage, and where the gateway runs.
Bifrost connects to Claude Code with two settings.json values, ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN set to a virtual key, and needs no Anthropic account login.
Bifrost routes Claude Code to Claude on Amazon Bedrock, Vertex AI, or Azure through stable deployment names, and keeps each session on one provider key so prompt cache hits continue.
Enterprise rollout follows five steps: deploy the gateway, issue one virtual key per developer, distribute managed settings, verify, then lock the endpoint.

A Claude Code gateway is a service that sits between Claude Code and model providers, authenticating each developer, enforcing budgets, and forwarding every request to Anthropic, Amazon Bedrock, Google Vertex AI, or another provider from one endpoint. Bifrost, the open-source AI gateway built in Go by Maxim AI, fills that role for engineering organizations running Claude Code at scale, with 11 microseconds of overhead per request at 5,000 RPS. This guide explains what to evaluate in a Claude Code gateway, why teams searching for a "Claude Code proxy" usually need a gateway, and how to configure, govern, and roll one out.

What Is a Claude Code Gateway?

A Claude Code gateway is an Anthropic Messages-compatible endpoint that Claude Code calls instead of api.anthropic.com. Developers authenticate with a gateway-issued credential, and the gateway applies access rules, budgets, and routing before forwarding the request with the organization's provider credential. Provider keys never reach developer laptops.

Claude Code selects the gateway through ANTHROPIC_BASE_URL. Anthropic's guide to other LLM gateways lists what a gateway centralizes (credentials, usage tracking, cost controls, audit logging, provider switching) and states that Anthropic does not endorse, maintain, or audit third-party gateway products, and does not support routing Claude Code to non-Claude models through any gateway, so model substitution is a choice your team validates and owns.

Figure 1: Developers hold gateway credentials; provider keys, budgets, and routing decisions stay in one place the platform team operates.

The same gateway can serve Codex CLI; see Codex CLI routing across any model. The AI gateway architecture explainer covers the general pattern.

Claude Code Proxy vs Gateway: Which One Teams Need

Teams search for a "Claude Code proxy" for three reasons: running Claude Code against non-Anthropic models, sending traffic through a corporate HTTPS proxy, or adding cost and access control. Only the first two are proxy problems; cost control and access policy need identity, budgets, and logs, which is gateway work.

Open-source Claude Code proxy projects typically translate Anthropic-format requests into another provider's format for one developer. A corporate HTTPS_PROXY is a network control that sits between Claude Code and every server, including the gateway. Neither knows which developer sent a request or what it cost.

Need	Translation proxy	Corporate HTTPS proxy	Claude Code gateway (Bifrost)
Use other providers' models	Yes, usually one developer	No	Yes, 25+ providers
Per-developer credentials	No	No	Virtual key per developer
Budgets and rate limits	No	No	Virtual key, team, customer
Request logs with tokens and cost	Varies	No	Built in
Central provider switching	No	No	Yes, no laptop changes

Bifrost is an AI gateway rather than a proxy: it holds governance state, resolves models across providers, and records every request. The 5 best AI gateways in 2026 roundup compares self-hosted and managed options.

Why Claude Code Enterprise Rollouts Need a Gateway

Claude Code enterprise deployments need per-developer attribution, enforceable spend limits, provider redundancy, and one place to revoke access. Without a gateway, each developer authenticates to Anthropic directly and spend is visible only in aggregate on the invoice.

Cost scales with agent activity rather than seat count. Four problems appear at team scale:

Attribution: finance cannot tell which team or developer drove a spike.
Spend control: no per-developer cap exists until someone reads the invoice.
Offboarding: revoking access means rotating shared provider keys.
Provider dependency: a rate limit or outage on one account blocks everyone.

Bifrost addresses each through virtual keys: one per developer, revocable on its own, with budgets checked before every request. The Bifrost governance overview describes how keys, teams, and customers fit together, and the Bifrost Enterprise tier adds RBAC, SSO, and in-VPC deployment for regulated teams.

Key Criteria for Evaluating a Claude Code Gateway

Evaluate a Claude Code gateway on forwarding fidelity first, then governance, provider coverage, observability, and deployment model. Claude Code adds request fields and beta headers with each release, and Anthropic's gateway compatibility guide states that a gateway which strips or rewrites them breaks the matching feature.

Criterion	What to test	Failure symptom in Claude Code
Streaming	Events relayed as they arrive, SSE pings kept	Stalls, idle-timeout aborts
Beta header pass-through	`anthropic-beta` and `anthropic-version` forwarded verbatim	`400` errors naming unknown fields
Error bodies	Upstream errors forwarded unmodified	Automatic retry and recovery stop working
Prompt caching	`cache_control` preserved; sessions stay on one key	High `input_tokens`, low cache reads
Tool-call streaming	Tool arguments streamed intact by the upstream	Empty `arguments`, failed file edits
Governance	Budgets and rate limits per developer, team, customer	Spend found after the fact
Deployment	Self-hosted, in-VPC, or air-gapped	Source code leaves your network
Overhead	Published figure with instance type and load	Latency compounds over long sessions

Claude Code's retry logic matches on the upstream's error wording, so a gateway that wraps errors in its own envelope breaks recovery even when it keeps the status code. Prompt caching depends on routing too: a session that moves between provider keys misses the cache it warmed a turn earlier.

On overhead, Bifrost publishes sustained benchmarks of 11 µs per request at 5,000 RPS with a 100% success rate. The production-ready LLM gateway comparison scores general-purpose gateways on these criteria, and what an AI gateway does and why it matters covers the concepts.

How Bifrost Works as a Claude Code Gateway

The Bifrost AI gateway exposes an Anthropic-compatible endpoint at /anthropic, authenticates Claude Code with a virtual key, checks budgets and rate limits, resolves the model through routing rules, and pins the session to one provider key. Claude Code needs no code changes, only a base URL and a credential.

Figure 2: Policy checks run before any provider is called, and session affinity keeps a session on the provider whose prompt cache it already warmed.

The capabilities that matter most for Claude Code:

Session affinity: Claude Code sends x-claude-code-session-id on every request, and session affinity adopts it with no configuration, so a session and its subagents stay on one provider and key and keep hitting the same prompt cache.
Model aliasing: routing rules can rewrite an alias such as sonnet-model to any configured provider and model, matched on the claude-cli user agent.
Provider coverage: 25+ providers and 10,000+ models through one API, listed on the supported providers page.
Failover: automatic fallbacks rotate keys on 429 and retry 5xx errors with backoff.
Request logs: built-in observability records inputs, outputs, tokens, cost, and latency for each request, asynchronously.

Since Claude Code 2.1.212, Anthropic enforces anthropic-version and related headers. In Bifrost, set Settings > Client Settings > Allowed Headers to *, or add the list from the Bifrost Claude Code integration guide.

Claude Code Environment Variables for a Gateway

Claude Code reads gateway configuration from the env block of settings.json. With Bifrost, two variables are required: ANTHROPIC_BASE_URL pointing at the /anthropic endpoint and ANTHROPIC_AUTH_TOKEN holding a virtual key. The token is sent as a bearer header, so no Anthropic account login is needed.

Merge the block into the most specific settings file that applies: ~/.claude/settings.json (user), .claude/settings.json (project), or .claude/settings.local.json (personal override). Remove any top-level model field, because it overrides the environment-based model selection.

"env": {
  "ANTHROPIC_BASE_URL": "http://localhost:8080/anthropic",
  "ANTHROPIC_AUTH_TOKEN": "your-bifrost-virtual-key",
  "ANTHROPIC_DEFAULT_SONNET_MODEL": "claude-sonnet-4-6",
  "ANTHROPIC_DEFAULT_HAIKU_MODEL": "claude-haiku-4-5"
}

Variable	Value with Bifrost	What it does
`ANTHROPIC_BASE_URL`	`http://localhost:8080/anthropic`	Sends Claude Code traffic to the gateway
`ANTHROPIC_AUTH_TOKEN`	Bifrost virtual key	Bearer credential; replaces subscription login
`ANTHROPIC_CUSTOM_HEADERS`	`x-bf-vk: <key>`	Alternative; still requires an Anthropic account login
`ANTHROPIC_DEFAULT_SONNET_MODEL`, `_HAIKU_`, `_OPUS_`	`provider/model` or an alias	Maps each Claude Code tier to a model
`CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY`	`1`	Fills `/model` from the gateway's model list (2.1.129+)
`CLAUDE_CODE_ENABLE_FINE_GRAINED_TOOL_STREAMING`	`1`	Restores a default that is off behind custom base URLs

Gateway model discovery only lists model IDs beginning with claude or anthropic. The Bifrost CLI skips manual setup entirely: npx -y @maximhq/bifrost-cli configures the base URL, key, and model, stores the virtual key in the OS keyring, and launches Claude Code. Pinning tiers to GPT, Gemini, or local models is covered in the comparison of AI gateways for running Claude Code with non-Anthropic models.

Running Claude Code on Amazon Bedrock Through Bifrost

To run Claude Code on Bedrock through Bifrost, map Bifrost deployment names to Bedrock model IDs on the AWS Bedrock provider key, allow the bedrock provider on the developer's virtual key, and set Claude Code's tier variables to names such as bedrock/claude-sonnet-5. AWS credentials stay on the gateway.

Figure 3: Developers only ever see stable deployment names; the Bedrock model ID and AWS credentials live on the Bifrost provider key.

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://gateway.example.com/anthropic",
    "ANTHROPIC_AUTH_TOKEN": "your-bifrost-virtual-key",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "bedrock/claude-opus-4-8",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "bedrock/claude-sonnet-5",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "bedrock/claude-haiku-4-5-20251001"
  }
}

If the provider key restricts models, list every deployment name there; mappings do not expand the allowlist. Test with /model, then confirm in Bifrost Logs that the request resolved to the expected Bedrock model ID. The full steps and a troubleshooting table are in the Claude Code with Bedrock runbook, and a walkthrough lives in routing Claude Code through AWS Bedrock with Bifrost.

Compared with Claude Code's native Bedrock mode (CLAUDE_CODE_USE_BEDROCK=1), this path keeps per-developer keys, budgets, and logs, and lets one settings file switch a tier to Vertex AI or Azure. Through ANTHROPIC_BASE_URL, Claude Code sends its full capability set; if a Bedrock upstream rejects pre-release fields with 400 errors, set CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1.

Connecting Claude Code MCP Servers Through Bifrost

Bifrost also acts as an MCP gateway for Claude Code, aggregating every connected MCP server behind one /mcp endpoint. One claude mcp add command replaces a config entry per server, and the virtual key in the header scopes which tools each developer can call.

claude mcp add --transport http bifrost http://localhost:8080/mcp \
  --header "Authorization: Bearer your-virtual-key" \
  --scope user

Run /mcp inside Claude Code to confirm bifrost is connected. When inference also routes through Bifrost, turn off auto tool injection to keep the two paths separate. Per-key MCP tool filtering controls which tools appear.

Tool definitions matter for cost behind any gateway. Claude Code turns MCP tool search off when ANTHROPIC_BASE_URL points to a non-first-party host, per Anthropic's MCP documentation, so every tool definition loads upfront unless tool search is re-enabled. Bifrost Code Mode cut input tokens by 92.8% in a 508-tool benchmark by exposing tools through four meta-tools.

The Bifrost MCP gateway page and the guide to reducing MCP token costs for Claude Code cover the details.

Budgets and Rate Limits for Claude Code Teams

Bifrost enforces Claude Code budgets at four levels (customer, team, virtual key, and provider config) and rate limits at the virtual key and provider config levels. Every applicable budget is checked before a request is forwarded, and any single failure blocks it, so a developer cannot exceed a team cap through their own key.

Level	Budget	Rate limits	Typical Claude Code use
Customer	Yes	No	Business unit or cost center
Team	Yes	No	Platform, mobile, data teams
Virtual key	Yes	Requests and tokens	One per developer
Provider config	Yes	Requests and tokens	Cap Opus-tier spend on one provider

Budgets reset on 1d, 1w, 1M, 1Q, or 1Y windows and can be calendar-aligned to UTC boundaries. Virtual keys are deny-by-default for providers.

In Bifrost Enterprise, access profiles auto-issue a governed virtual key to each user from a role, and alerting notifies Slack, Microsoft Teams, PagerDuty, or a webhook when a budget crosses a threshold.

For dashboards and per-developer reporting, see monitoring Claude Code token usage with an AI gateway. Teams running both agents can apply the same keys to Codex CLI token spend.

Rolling Out a Claude Code Gateway Across the Organization

A Claude Code gateway rollout has five steps: deploy the gateway with provider credentials, issue one virtual key per developer, distribute the base URL and credential through managed settings, verify each machine, then restrict Claude Code to the gateway endpoint. The first four match Anthropic's rollout checklist.

Figure 4: Lock the endpoint last, after every machine has proven it reaches the gateway, so a distribution gap never blocks a developer.

Deploy: run Bifrost with npx -y @maximhq/bifrost or Docker, then move to clustering or an in-VPC deployment for production.
Issue keys: one virtual key per developer, or access profiles issued from SSO roles.
Distribute: push the env block through a managed settings file via MDM; a managed ANTHROPIC_BASE_URL cannot be overridden by a developer's shell.
Verify: have each developer run /status, then confirm their requests appear in Bifrost logs.
Lock: set allowedProviders to ["customEndpoint"] in managed settings (Claude Code 2.1.285+) so sessions pointed anywhere else are refused.

For machines where configuring each tool is impractical, AI Gateway + Bifrost Edge extends the same governance to the endpoint: Bifrost remains the policy engine, and Bifrost Edge, currently in alpha, routes Claude Code traffic from each laptop through it with no settings.json changes. The guide to governing coding agents in enterprise rollouts covers policy design across agents.

Frequently Asked Questions

What is the difference between a Claude Code proxy and a Claude Code gateway?

A Claude Code proxy typically forwards or translates requests for one developer, often to reach a non-Anthropic model. A Claude Code gateway adds identity, budgets, rate limits, request logs, and central provider switching for a whole organization. Bifrost is an AI gateway: each developer authenticates with their own virtual key, and every request is checked against budgets and recorded with tokens and cost.

What LLM does Claude Code use?

Claude Code uses Anthropic's Claude models by default: Sonnet as the main tier, Opus for complex tasks, and Haiku for lightweight work. Behind a gateway, each tier maps to whatever the ANTHROPIC_DEFAULT_*_MODEL variables specify, including Claude on Bedrock or Vertex AI. Anthropic does not support non-Claude models through gateways; running Claude Code with non-Anthropic models covers what teams test.

Does a gateway slow Claude Code down?

Bifrost adds 11 microseconds per request at 5,000 RPS in sustained benchmarks, which is negligible next to model response times. The larger performance factor is prompt caching: Bifrost session affinity keeps each Claude Code session on one provider key, so cache reads continue across turns instead of resetting when routing changes.

Do developers still need a Claude subscription behind a gateway?

No, not when the gateway credential is set. With ANTHROPIC_AUTH_TOKEN holding a Bifrost virtual key, requests carry that credential instead of a claude.ai login, and usage is billed to the provider account behind the gateway. Setting only ANTHROPIC_BASE_URL without a credential keeps the subscription login active, so its usage limits and billing still apply.

Can Claude Code use Amazon Bedrock through a gateway?

Yes. In Bifrost, map deployment names such as claude-sonnet-5 to Bedrock model IDs on the AWS Bedrock provider key, allow bedrock on the virtual key, and set Claude Code's tier variables to bedrock/claude-sonnet-5. AWS credentials stay on the gateway, and each request is logged with the resolved Bedrock model ID for verification.

How do you set budgets for a Claude Code team?

Issue each developer a virtual key with its own budget and rate limits, attach the keys to a team with a team budget, and optionally group teams under a customer budget. Bifrost checks every applicable budget before forwarding a request. Alerting can notify the team before a cap is reached.

Start Running Claude Code Through Bifrost

A Claude Code gateway gives platform teams per-developer credentials, enforceable budgets, routing to Bedrock and Vertex AI, and request-level logs without changing how developers use Claude Code. Bifrost does this with two settings.json values and 11 µs of overhead. The Bifrost Claude Code resource page has setup details, and to plan a Claude Code enterprise rollout with the team, book a Bifrost demo.

An LLM gateway is the layer that authenticates, routes, governs, and logs every model call through one API. This comparison ranks Bifrost, LiteLLM, Kong AI Gateway, Cloudflare AI Gateway, and AWS Bedrock on overhead, failover, governance depth, MCP support

Kamya Shah — Tue, 06 Oct 2026 08:29:29 +0000

Kamya Shah

Oct 6

Best LLM Gateways in 2026: A Production-Ready Comparison

#llm #ai #gateway #api

15 min read

Best LLM Gateways in 2026: A Production-Ready Comparison

Kamya Shah — Tue, 06 Oct 2026 08:29:18 +0000

TL;DR

An LLM gateway is production-ready when it adds negligible latency under load, fails over across providers without application code, enforces budgets per team, governs MCP tool calls, and deploys where compliance requires.
Bifrost adds 11 microseconds of overhead per request at 5,000 requests per second with a 100% success rate, and enforces budgets at the customer, team, virtual key, and provider-config levels.
LiteLLM Proxy covers 100+ LLMs in Python, but production deployments run PostgreSQL and Redis alongside it, and SSO, audit logs, and Prometheus metrics require an enterprise license.
Kong AI Gateway adds AI plugins to Kong Gateway; its AI MCP Proxy plugin needs Kong Gateway 3.12 and an AI Gateway Enterprise license.
Cloudflare AI Gateway and AWS Bedrock are managed services, so neither runs inside your own network or spans every cloud from one control plane.

An LLM gateway is the layer between applications and model providers that authenticates, routes, governs, and logs every model call through one API. Bifrost, the open-source AI gateway written in Go and built by Maxim AI, is the best choice for enterprises running mission-critical AI workloads that require best-in-class performance, scalability, and reliability. This guide compares the five LLM gateways most often shortlisted for production in 2026 (Bifrost, LiteLLM, Kong AI Gateway, Cloudflare AI Gateway, and AWS Bedrock) on gateway overhead, failover, governance depth, MCP support, deployment model, and compliance. Every competitor capability below was checked against that vendor's own documentation, and anything a vendor does not publish is marked "Not published."

What Is an LLM Gateway?

An LLM gateway is a control layer that sits between AI applications and model providers, exposing one API while handling authentication, routing, failover, spend limits, and logging for every request. Applications call the gateway instead of each provider SDK, so changing a model or adding a provider becomes configuration rather than a code change.

Menlo Ventures estimated enterprise spend on foundation model APIs at $12.5 billion in 2025. Without a shared layer, each team that drives that spend writes its own retry logic, stores its own provider keys, and reports cost in its own format.

Figure 1: Every model call and tool call crosses the same layer, so access, spend, and failover are decided once instead of in each application.

As Figure 1 shows, a production gateway now carries tool calls over the Model Context Protocol as well as model calls. The broader architecture is covered in what an AI gateway is and why it matters, and teams choosing their first gateway by use case can start with the roundup of AI gateways for self-hosted and managed teams.

LLM gateway vs LLM proxy vs LLM router

The three terms overlap, and vendors use them loosely. In practice they describe different scopes:

LLM proxy: forwards requests to one or more providers behind a unified API. Scope is translation and forwarding.
LLM router: decides which model or provider serves each request, based on cost, latency, content, or rules. Scope is the routing decision.
LLM gateway: combines both and adds identity, budgets, rate limits, guardrails, audit trails, and observability. Scope is control of all AI traffic.

Production teams need the gateway scope, because forwarding and routing alone do not answer who spent what or what happens when a provider returns errors for an hour. The same distinction applies to tools, where an MCP gateway differs from an MCP proxy and an MCP server.

How to Evaluate an LLM Gateway for Production

Evaluate an LLM gateway on six production criteria: overhead under sustained load, failover behavior, governance depth, MCP and agent support, deployment model, and observability. Feature checklists hide the differences that matter, so ask each vendor for a measured number, a documented mechanism, or a deployment option rather than a yes or no.

Criterion	Why it decides production readiness	What to verify
Gateway overhead	Added latency compounds on every request and every agent step	A published figure with instance type, request rate, and what the number excludes
Failover	Provider 429s and 5xx errors are routine at scale	Retries per provider, fallback chains across providers, and whether apps need retry code
Governance depth	Budgets and access belong to the platform, not each app	Budget levels, virtual keys, rate limits, RBAC, SSO, and which tier they sit in
MCP and agent support	Agents route tool calls as well as model calls	Tool filtering per key, MCP auth types, and execution control
Deployment model	Data residency rules out some options entirely	Self-hosted, in-VPC, on-prem, air-gapped, or managed only
Observability and compliance	Debugging, cost attribution, and audits depend on it	Request logs, Prometheus, OpenTelemetry, audit logs, and log export targets

Figure 2: Policy checks run before any provider is called, and failover happens inside the gateway, not in application code.

Figure 2 is the request path to test every candidate against. The LLM Gateway Buyer's Guide expands these criteria into a procurement checklist, and evaluating an LLM gateway for enterprise scalability covers load-testing method.

Best LLM Gateways Compared at a Glance

The best LLM gateway for production covers all six criteria in one control plane. Bifrost is the only option here that combines an open-source core, published microsecond-level overhead, four budget levels, and model plus MCP traffic in one deployment you can host in your own VPC.

	Bifrost	LiteLLM	Kong AI Gateway	Cloudflare AI Gateway	AWS Bedrock
Form	Go gateway, open source (Apache 2.0)	Python proxy, open source with enterprise tier	AI plugins on Kong Gateway	Managed service on Cloudflare's network	Managed AWS service
Published overhead	11 µs at 5,000 RPS (t3.xlarge)	Not published	Not published	Not published	Not published
Deployment	Self-hosted, in-VPC, on-prem, air-gapped	Self-hosted; production runs PostgreSQL and Redis	Self-hosted, or Konnect control plane with your data planes	Managed only	Managed only, in AWS regions
Failover	Retries plus cross-provider fallback chains	Load balancing, routing, fallbacks	Routing and load balancing across providers	Retries and model fallbacks	Cross-Region inference within AWS
Budgets	Customer, team, virtual key, provider config	Per virtual key and user	Token rate limits and spend caps	Budget and rate limits per key in Dynamic Routing	Not published
MCP	MCP client and server, six auth types, Code Mode	MCP Gateway	AI MCP Proxy (3.12+, enterprise license)	Not published	AgentCore Gateway (separate service)

"Not published" means the vendor documentation read for this comparison does not state the figure or feature. Gateway overhead deserves the most scrutiny: in a 500 RPS benchmark against LiteLLM, Bifrost measured 0.99 ms of gateway overhead against 40 ms, and a P99 latency of 1.68 s against 90.72 s, on the same t3.medium instance.

1. Bifrost: Lowest-Overhead LLM Gateway with Built-In Governance

The Bifrost AI gateway is open source, written in Go, and unifies 25+ providers and 10,000+ models behind one OpenAI-compatible API. Bifrost adds 11 microseconds of overhead per request at 5,000 RPS with a 100% success rate in sustained benchmarks, and it governs model calls and MCP tool calls from the same deployment.

Performance. The benchmarking suite measured 11 µs of Bifrost overhead on a t3.xlarge and 59 µs on a t3.medium, both at 5,000 RPS with zero failed requests. Existing OpenAI, Anthropic, and Google GenAI clients switch over by changing only the base URL, as a drop-in replacement.

Failover and routing. Retries and fallbacks work in two layers: retries rotate keys on 429 and auth errors and back off on 5xx, then fallbacks move the request to the next provider once retries are exhausted. Routing rules use Common Expression Language to route on headers, parameters, capacity, and organizational scope, evaluated from virtual key to team to customer to global.

Governance. Virtual keys are the unit of access: each one restricts providers and models and carries its own budget and rate limits. The budget hierarchy checks every applicable level before a request proceeds, as Figure 3 shows.

Figure 3: Any single budget failure blocks the request, so a team cap holds even when an individual virtual key still has room.

MCP gateway. Bifrost acts as both an MCP client and an MCP server, with six MCP authentication types (None, Headers, OAuth 2.0, Per-User OAuth, Per-User Headers, and Token Exchange) and tool filtering per request, per client, or per virtual key. Tool calls are not executed automatically unless Agent Mode is configured. Code Mode cut input tokens by 58.2% at 96 tools and 92.8% at 508 tools in benchmark rounds, with around 40% faster execution in large deployments, a pattern detailed in how Code Mode cuts agent token costs.

Enterprise deployment and compliance. Bifrost Enterprise is a strict superset of the open-source gateway and adds the following:

Clustering with gossip-based state sync, six service-discovery methods, and zero-downtime rolling updates
Adaptive load balancing that shifts weights using live error rate and latency per model and key
In-VPC deployments on AWS, GCP, Azure, and Cloudflare, plus on-prem and air-gapped installs
Guardrails with native secrets detection and custom regex, plus AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, and Microsoft Presidio
Audit logs of administrative activity, signed with an HMAC key and archivable to S3 or GCS
RBAC, OIDC single sign-on through Okta, Microsoft Entra, and Keycloak, and secret storage in HashiCorp Vault, AWS Secrets Manager, or GCP Secret Manager

Observability ships in the open-source gateway: request logs with tokens and cost, Prometheus metrics, and OpenTelemetry traces.

Coding agents can share the same deployment; see choosing an AI gateway for Claude Code and routing Codex CLI to any model.

Best for: Bifrost is built for enterprises running mission-critical AI workloads that require best-in-class performance, scalability, and reliability. It serves as a centralized AI gateway to route, govern, and secure all AI traffic across models and environments with ultra low latency. Bifrost unifies LLM gateway, MCP gateway, and Agents gateway capabilities into a single platform. Designed for regulated industries and strict enterprise requirements, it supports air-gapped deployments, VPC isolation, and on-prem infrastructure. It provides full control over data, access, and execution, along with robust security, policy enforcement, and governance capabilities.

2. LiteLLM Proxy: Python LLM Gateway with a Wide Provider Catalog

LiteLLM Proxy is an open-source, Python-based LLM gateway that exposes 100+ LLMs through an OpenAI-compatible interface. It is a common starting point for Python teams, and its production footprint and enterprise licensing are the two factors that most affect a production decision.

What LiteLLM provides:

Virtual keys with budgets per key and per user, plus rate limiting
Load balancing, routing, and fallbacks across deployments
Caching, logging, alerting, and spend tracking
An MCP Gateway and an Agent Gateway for A2A traffic
Self-hosted deployment through Docker, and a Rust AI Gateway in beta

Production considerations:

LiteLLM's production reference architecture places PostgreSQL and Redis alongside the proxy in every deployment mode.
SSO for the admin UI, audit logs, secret manager integrations, Prometheus metrics, and per-key guardrail control are listed as enterprise features.
Overhead under load is not published by LiteLLM. In Bifrost's 500 RPS test on a t3.medium, LiteLLM completed 88.78% of requests with a P99 latency of 90.72 s.

Best for: Python-centric teams that want a self-hosted proxy with a broad provider catalog and can operate PostgreSQL and Redis alongside it. Teams that outgrow it can compare options on the LiteLLM alternative page.

3. Kong AI Gateway: AI Plugins on an Existing API Platform

Kong AI Gateway adds AI-specific plugins to Kong Gateway, so organizations already running Kong for REST traffic can apply the same control plane to LLM traffic. Capabilities arrive as individual plugins, which suits existing Kong operators more than teams adopting Kong only for AI.

What Kong AI Gateway provides:

Routing and load balancing across AI providers, including OpenAI, Anthropic, Azure AI, Amazon Bedrock, and Gemini
AI Semantic Cache and AI Prompt Compressor for token cost reduction
Token rate limiting and spend caps, with AI Consumer Groups scoping model access by team
AI Prompt Guard, AI Semantic Prompt Guard, and AI Sanitizer for PII redaction, plus integrations with AWS, Azure, and GCP guardrail services
Audit logs, metrics exporters, and OpenTelemetry spans through Konnect

Production considerations:

The AI MCP Proxy plugin, which bridges MCP and HTTP and exposes REST APIs as MCP tools, requires Kong Gateway 3.12 and is available only with an AI Gateway Enterprise license.
Deployment is self-hosted Kong Gateway or a Konnect-managed control plane with data planes in your environment.
Gateway overhead for the AI plugin chain is not published.

Best for: Organizations already standardized on Kong that want LLM traffic governed by the same platform team and tooling. A side-by-side view is in Kong AI Gateway alternatives.

4. Cloudflare AI Gateway: Managed Gateway on Cloudflare's Network

Cloudflare AI Gateway is a managed service that routes LLM calls through Cloudflare's global network with one line of code. Core features (dashboard analytics, caching, and rate limiting) are free, but the service does not run inside your own network.

What Cloudflare AI Gateway provides:

Response caching, rate limiting, and request retries with model fallbacks
Analytics on requests, tokens, and cost, plus request logging
Dynamic Routing with conditional branches, percentage splits, and per-key budget and rate limits that switch to a fallback when exceeded
DLP scanning, plus guardrails billed as Workers AI token usage

Production considerations:

Cloudflare AI Gateway runs only on Cloudflare's network, so it does not meet requirements that request data stay inside your own VPC or on-prem environment.
Logpush for exporting logs requires the Workers Paid plan.
MCP governance and published gateway overhead are not stated on the AI Gateway documentation pages.

Best for: Teams already building on Cloudflare Workers that want managed caching, analytics, and basic routing without operating infrastructure. Self-hosted options are compared in the Cloudflare AI Gateway alternative guide.

5. AWS Bedrock: Managed Model Access and AgentCore Gateway on AWS

AWS Bedrock is a managed, serverless platform that provides access to hundreds of foundation models inside the AWS account boundary. It is not a cross-cloud gateway: its controls apply to models served through Bedrock, and MCP lives in a separate service, AgentCore Gateway.

What AWS Bedrock provides:

Bedrock Guardrails for content filtering and PII detection
Cross-Region inference that routes requests across AWS Regions within a geography or globally
Intelligent Prompt Routing, prompt caching, and model distillation for cost control
Compliance coverage including ISO, SOC, GDPR, and FedRAMP High, with HIPAA eligibility; Bedrock does not use customer data to train models
AgentCore Gateway, a fully managed gateway that converts APIs, Lambda functions, and existing services into MCP-compatible tools

Production considerations:

Providers outside Bedrock's catalog, such as OpenAI's own API or self-hosted models, need a separate routing layer.
Cross-Region inference profiles do not support Provisioned Throughput.

Best for: Organizations standardized on AWS whose model traffic stays inside Bedrock. Teams that want Bedrock plus other providers can put Bifrost in front of it, keeping native Bedrock SDK calls while adding budgets and cross-provider failover, as described on the Bifrost for AWS Bedrock page and in AWS Bedrock gateway alternatives.

How to Choose an LLM Gateway for Your Constraints

Choose an LLM gateway by eliminating options on hard constraints first: where data may travel, which platforms you have already standardized on, and how many clouds and providers you use. Feature comparisons only matter among the options that survive those three questions, and the AI gateway architecture and core features guide explains what each remaining feature does.

Figure 4: Data residency and existing platform commitments narrow the choice faster than feature lists do.

Constraint	Best fit	Why
Request data must stay in your VPC, on-prem, or air-gapped	Bifrost	Self-hosted with in-VPC and air-gapped installs, governance included
Multi-cloud or multi-provider with strict latency budgets	Bifrost	11 µs overhead at 5,000 RPS and cross-provider fallback chains
Model calls and MCP tool calls under one policy set	Bifrost	MCP client and server with per-key tool filtering in the same deployment
Kong already runs your API platform	Kong AI Gateway	Reuses the existing control plane and operators
All model traffic stays in AWS Bedrock	AWS Bedrock	IAM, Regions, and compliance stay inside one account boundary
Workers-based app needing managed caching	Cloudflare AI Gateway	No infrastructure, free core features

Teams whose apps and agents use more than one provider land in the first three rows. The security side of that decision is covered in LLM gateway security for prompt injection, PII, and audit, and the Bifrost Enterprise page covers clustering and private deployment for regulated teams.

Frequently Asked Questions

What is the best LLM gateway?

For production enterprise workloads, Bifrost is the best LLM gateway in this comparison. It adds 11 microseconds of overhead at 5,000 RPS, fails over across 25+ providers, enforces budgets at four levels, governs MCP tool calls, and deploys self-hosted, in-VPC, or air-gapped. Kong AI Gateway suits existing Kong users.

Does AWS have an LLM gateway?

AWS offers two related services. Bedrock provides managed access to foundation models with Guardrails and Cross-Region inference, and AgentCore Gateway is a managed gateway that converts APIs and Lambda functions into MCP-compatible tools. Neither routes to providers outside AWS, so multi-provider teams usually place a gateway such as Bifrost in front of Bedrock to add budgets and cross-provider failover.

What is the Bifrost LLM gateway?

Bifrost is an open-source AI gateway built in Go that gives applications one OpenAI-compatible API for 25+ providers and 10,000+ models. It handles retries, fallbacks, virtual keys, hierarchical budgets, semantic caching, and MCP tool governance, adding 11 microseconds of overhead per request at 5,000 RPS. Bifrost runs self-hosted, in-VPC, on-prem, or air-gapped.

Is LiteLLM an LLM gateway?

Yes. LiteLLM Proxy is a Python-based LLM gateway that exposes 100+ models through an OpenAI-compatible API, with virtual keys, budgets, fallbacks, and an MCP Gateway. Production deployments run PostgreSQL and Redis alongside it, and features such as admin SSO, audit logs, and Prometheus metrics require an enterprise license. Teams comparing migration paths can review migrating from LiteLLM to Bifrost.

What is the difference between an MCP gateway and an LLM gateway?

An LLM gateway governs model calls: which caller may use which model, at what cost, with what failover. An MCP gateway governs tool calls: which agent may reach which MCP server and tool, under which credentials. Bifrost runs both in one deployment, so a single virtual key carries model budgets and MCP tool permissions together, as covered on the MCP gateway page.

How much latency does an LLM gateway add?

Bifrost measured 11 microseconds of overhead at 5,000 RPS on a t3.xlarge, and 0.99 ms against LiteLLM's 40 ms in a separate 500 RPS test with a 60 ms mock upstream. Ask every vendor for instance type, request rate, and what the figure excludes, because numbers from different methods are not comparable.

Try Bifrost as Your Production LLM Gateway

A production LLM gateway has to add almost no latency, recover from provider failures on its own, enforce budgets per team, govern MCP tool calls, and run where your compliance rules allow. Bifrost covers all five with an open-source core and Bifrost Enterprise for clustering and private deployment. Measured results are on the Bifrost performance benchmarks page, and the governance model is outlined in how Bifrost governs AI usage. To run Bifrost against your own workload and plan an enterprise AI gateway rollout, book a demo with the Bifrost team.

AI gateways put one API in front of many model providers and centralize routing, failover, budgets, and logging. This roundup compares Bifrost, Cloudflare AI Gateway, Vercel AI Gateway, LiteLLM, and OpenRouter by use case, from regulated enterprises to pro

Kamya Shah — Tue, 06 Oct 2026 08:27:42 +0000

Kamya Shah

Oct 6

5 Best AI Gateways in 2026: Picks by Use Case

#ai #llm #apigateway #aigateway

15 min read

5 Best AI Gateways in 2026: Picks by Use Case

Kamya Shah — Tue, 06 Oct 2026 08:27:18 +0000

TL;DR

AI gateways put one API in front of many model providers and centralize routing, failover, budgets, and logging, so applications stop integrating with each provider separately.
Bifrost is the best overall choice for enterprise and platform teams: open source under Apache 2.0, 11 µs of overhead at 5,000 RPS, and deployable in a VPC, on-prem, or air-gapped.
Cloudflare AI Gateway and Vercel AI Gateway are managed services that fit teams already building on those platforms and willing to route prompts through a vendor network.
LiteLLM is a self-hosted Python gateway with an MIT-licensed core; SSO, audit logs, and several governance features require an enterprise license.
OpenRouter is a hosted model marketplace with one key for hundreds of models, best for prototyping and model exploration rather than governed production traffic.

AI gateways are the infrastructure layer between applications and LLM providers: they expose one API and handle routing, failover, budgets, and logging for every model call. Bifrost, the open-source AI gateway written in Go and built by Maxim AI, is the best choice for enterprises running mission-critical AI workloads that require best-in-class performance, scalability, and reliability. This roundup compares five AI gateways by use case (self-hosted versus managed, developer versus platform teams) for teams choosing a first gateway.

What AI Gateways Do and Why Teams Adopt Them

AI gateways are a single control point that receives model requests from applications, applies authentication and spend policy, routes each call to a provider, retries or fails over on errors, and records tokens, cost, and latency. Teams adopt them once more than one application, provider, or team is calling models, because those concerns stop fitting inside application code.

Multi-provider use is now common. Menlo Ventures' 2025 State of Generative AI in the Enterprise report puts enterprise AI spend at $37 billion in 2025, with Anthropic, OpenAI, and Google together accounting for 88% of enterprise LLM API usage. Running two or more of them multiplies SDKs, credentials, rate limits, and invoices.

Figure 1: Every application integrates once with the gateway; provider choice, budgets, and logging move out of application code.

As Figure 1 shows, the gateway turns many provider integrations into one. It removes the same problems for most teams:

Provider-specific code: each provider has its own SDK, request format, and error codes.
Outages and rate limits: a 429 or 5xx becomes a user-facing failure without retries and fallbacks.
Invisible spend: costs surface on invoices instead of per team or per key.
Credential sprawl: raw provider keys end up in services, notebooks, and laptops.

For the full component breakdown, see the cluster guide on what an AI gateway is and how its architecture works.

How to Choose the Best AI Gateway

The best AI gateway is the one whose deployment model matches where your data is allowed to go, and whose governance depth matches how many teams share it. Settle those two questions first; provider coverage, failover, and observability are table stakes that most of the five tools below cover in some form.

The criteria below separate the five tools in practice; a weighted scoring matrix is in the LLM gateway buyer's guide.

Criterion	What to check	Why it matters
Deployment model	Self-hosted, in-VPC, on-prem, air-gapped, or managed only	Decides whether prompts and logs leave your network
Governance depth	Scoped keys, budgets per team or customer, rate limits, RBAC	Decides whether one gateway can serve many teams safely
Reliability	Retries, key rotation, provider and model fallback chains	Keeps provider incidents from becoming application incidents
Overhead	Published per-request latency under sustained load	Gateway latency compounds in agent loops with many calls
Agent support	MCP tool routing, coding agent integrations	Agent traffic needs the same policy as chat traffic
Licensing	Open source license and which features are paywalled	Determines lock-in and the cost of governance features

Teams evaluating gateways for production SLAs, compliance, and MCP depth should also read the production-focused comparison of the best LLM gateways, which scores Bifrost against Kong AI Gateway and AWS Bedrock on those criteria. Policy design itself is covered on the Bifrost governance resource page.

AI Gateways Compared at a Glance

The five AI gateways split into two groups: Bifrost and LiteLLM are self-hosted, while Cloudflare AI Gateway, Vercel AI Gateway, and OpenRouter are managed services. Bifrost is the only one of the five that combines an Apache 2.0 license, a published microsecond-level overhead figure, and governance features that ship in the open-source build.

Gateway	Deployment	License	Model access	Failover	Budgets and keys	Published overhead
Bifrost	Self-hosted, in-VPC, on-prem, air-gapped	Apache 2.0	25+ providers, 10,000+ models	Retries, key rotation, fallback chains	Virtual keys, hierarchical budgets, rate limits	11 µs at 5,000 RPS
Cloudflare AI Gateway	Managed (Cloudflare network)	Managed service	Workers AI, OpenAI, Anthropic, Google Gemini, Replicate, more	Request retry and model fallback	Spend limits, rate limiting	Not published
Vercel AI Gateway	Managed (callable from any infrastructure)	Managed service	Models across providers via AI SDK, OpenAI, and Anthropic APIs	Provider and model fallbacks	Budgets per team, project, key, member	Not published
LiteLLM	Self-hosted (Docker)	MIT core, enterprise license for some features	100+ LLMs	Load balancing and fallbacks	Virtual keys, budgets, rate limits	Not published
OpenRouter	Managed (hosted only)	Managed service	Hundreds of models, one endpoint	Automatic fallbacks	Not published	Not published

Bifrost's overhead figure comes from its published performance benchmarks. "Not published" means the vendor page reviewed carries no figure, not that the capability is absent.

1. Bifrost: The Open Source AI Gateway for Enterprise Teams

The Bifrost AI gateway is open source and gives applications one OpenAI-compatible API to 25+ providers and 10,000+ models, adding 11 microseconds of overhead per request at 5,000 RPS. Governance, failover, caching, and MCP tool routing run in the same Go binary, which runs on a laptop or Kubernetes, with Bifrost Enterprise adding in-VPC and air-gapped deployment.

Figure 2: Governance and cache checks run before any provider call, and the fallback chain engages only after retries on the primary are exhausted.

Performance and deployment

Bifrost adds 11 µs of overhead per request at a sustained 5,000 RPS with a 100% success rate on a t3.xlarge instance, as documented in the benchmarking methodology. The gateway starts with zero configuration, and providers are added through the web UI, API, or config file:

npx -y @maximhq/bifrost
# or
docker run -p 8080:8080 maximhq/bifrost

Existing code moves over as a drop-in replacement: change the SDK base URL and swap the provider key for a Bifrost virtual key.

client = openai.OpenAI(
    base_url="http://localhost:8080/openai",
    api_key="<bifrost-virtual-key>",
)

The supported providers matrix covers OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, Azure OpenAI, Mistral, Groq, Ollama, vLLM, and OpenRouter, among others.

Routing, failover, and caching

Bifrost handles provider errors in two layers. Retries and fallbacks first retry transient 5xx errors with exponential backoff and rotate to another API key on 429 or auth failures; when retries are exhausted, the request moves to the next provider in the fallback chain, which gets its own retry budget.

Semantic caching runs in two modes: direct hash matching replays identical requests, and semantic matching serves a cached answer for similar prompts using embeddings. Caching engages only for requests that carry a cache key and requires a vector store such as Redis, Valkey, Weaviate, or Qdrant, so measure hit rates on your own traffic. The trade-offs are covered in the guide to cutting token spend with semantic caching.

Governance and security

Virtual keys are the primary governance entity in Bifrost: each key carries its own model and provider allow-list, budget, and rate limits. Hierarchical budgets stack at the virtual key, team, and customer levels, with reset windows from one day to one year. These controls ship in the open-source build.

Bifrost Enterprise adds guardrails (built-in secrets detection and custom regex, plus external providers such as AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, and Patronus AI), clustering, RBAC, and signed audit logs of administrative activity. Regulated teams can run it through in-VPC deployments on AWS, Google Cloud, Azure, or Cloudflare.

MCP and coding agents

Bifrost acts as both a client and a server for the Model Context Protocol, so tool calls pass through the same keys and logs as model calls; the MCP gateway resource page covers the architecture. Code Mode replaces large tool catalogs with four meta-tools and cut input tokens by 92.8% at 508 tools in Bifrost's benchmark.

For coding agents, Bifrost exposes OpenAI, Anthropic, and Gemini-compatible endpoints, so Claude Code, Codex CLI, and Gemini CLI point at it with a base URL change. The Bifrost CLI launches those agents against the gateway with keys and models preconfigured.

2. Cloudflare AI Gateway

Cloudflare AI Gateway is a managed gateway that runs on Cloudflare's network and is available on all Cloudflare plans. It adds caching, rate limiting, request retry and model fallback, analytics, and logging in front of providers including Workers AI, OpenAI, Anthropic, Google Gemini, and Replicate, without any infrastructure for the team to operate.

Cloudflare's feature list also covers spend limits, dynamic routing, an auto router that picks a lower-cost model per request, content-moderation guardrails, data loss prevention, and bring-your-own-key storage.

Strengths: no servers to run, plus request, token, and cost analytics.
Trade-offs: the gateway runs only as a managed service, so prompts and logs pass through Cloudflare's network; no self-hosted option is published.

Best for: Teams already building on Cloudflare Workers that want caching, rate limiting, and cost analytics in front of a few providers with nothing to operate, and whose compliance posture allows prompts to transit a third-party network.

Teams that outgrow it usually want self-hosting or per-team governance; see the best Cloudflare AI Gateway alternative.

3. Vercel AI Gateway

Vercel AI Gateway is a managed gateway that gives applications one endpoint to models across providers, with provider and model fallbacks, request logs, budgets, and bring-your-own-key support. Vercel states that it adds zero markup to provider token prices, including with BYOK, and that calling applications do not need to run on Vercel.

Applications call it through the AI SDK, the OpenAI Chat Completions and Responses APIs, or the Anthropic Messages API. Request logs record provider attempts, latency, tokens, and cost per request.

Strengths: tight AI SDK integration, multimodal coverage (text, image, video, speech, embeddings), and budgets scoped to a team, project, API key, or team member.
Trade-offs: budgets are soft caps that cover spend billed through Vercel system credentials; BYOK spend is metered separately and does not count toward those limits. The gateway is managed only.

Best for: Frontend and full-stack teams building with Next.js and the AI SDK that want model access, fallbacks, and spend visibility from the same vendor that hosts their application.

Teams that need the gateway inside their own network compare options in Vercel AI Gateway alternatives for 2026.

4. LiteLLM

LiteLLM is a self-hosted Python gateway, the LiteLLM Proxy, that calls 100+ LLMs through one OpenAI-format interface. Its MIT-licensed core lists virtual keys, budgets, rate limits, spend tracking, load balancing, fallbacks, caching, guardrails, and an admin UI, deployed with Docker on infrastructure the team runs.

Several governance features sit behind LiteLLM's enterprise license, including SSO for the admin UI, audit logs with retention policies, JWT authentication, secret manager integrations, and custom guardrails per API key. Teams should price that license into any comparison against gateways that ship those controls in the open-source build.

Performance is the other consideration. In Bifrost's head-to-head benchmark at 500 RPS on a t3.medium instance, LiteLLM recorded a P99 latency of 90.72 seconds against 1.68 seconds for Bifrost, with roughly 9.5x lower throughput and about three times the memory.

Strengths: large provider catalog and a familiar Python SDK.
Trade-offs: Python runtime overhead under concurrency, and SSO and audit logs require a paid license.

Best for: Python-first teams and prototypes that want a self-hosted gateway with broad provider coverage and moderate request volumes.

Teams searching for LiteLLM alternatives usually want lower overhead or governance without a separate license; the LiteLLM alternative resource page compares the two and maps the migration. Bifrost also offers LiteLLM SDK compatibility, so existing LiteLLM client code can point at it.

5. OpenRouter

OpenRouter is a hosted service that gives developers access to hundreds of AI models through a single OpenAI-compatible endpoint. It handles provider fallbacks automatically and picks a cost-effective provider for each request. Inference is billed at provider rates with no markup, while credit purchases carry a fee (5.5% with a $0.80 minimum by card).

Bring-your-own-key usage is free up to a monthly allowance, after which OpenRouter charges a 5% fee. Prompts and completions are not logged by default. Configurable spend limits per team or per key are not addressed in the FAQ reviewed for this comparison, and no self-hosted option is published.

Strengths: fastest way to try many models with one account and one key, with unified billing across providers.
Trade-offs: hosted only, and per-team governance, guardrails, and audit controls are not part of its published feature set.

Best for: Individual developers, demos, and model evaluation work where trying many models quickly matters more than self-hosting or per-team policy.

The most common OpenRouter alternatives are self-hosted gateways that keep the convenience of one API; the best OpenRouter alternative for production covers them, and a three-way OpenRouter vs LiteLLM vs Bifrost comparison breaks down the LiteLLM vs OpenRouter question. Teams can also keep OpenRouter as one upstream provider behind Bifrost and apply virtual key budgets on top.

Self-Hosted vs Managed AI Gateways

Self-hosted AI gateways run inside your network, so prompts, responses, and logs stay in infrastructure you control until the provider call. Managed AI gateways remove operational work but route every prompt through the vendor's network first. For regulated data, that difference usually decides the shortlist before any feature comparison starts.

Figure 3: With a self-hosted gateway, prompts and logs stay inside infrastructure you control until the provider call; with a managed one, they pass through a vendor network.

The practical trade-offs:

Data residency: a self-hosted gateway such as Bifrost keeps request logs in your own database; Bifrost also supports on-premise and air-gapped deployment for environments with no cloud identity federation.
Operations: managed gateways need no patching; Bifrost self-hosts as a single binary or container.
Cost model: managed services charge platform or credit fees; self-hosted gateways cost only their compute.
Failure domain: a managed gateway adds the vendor's availability to the request path.

A dedicated comparison of the best self-hosted AI gateway options goes deeper on this path.

Which AI Gateway Fits Your Team

The right pick follows from the team's constraints, not feature counts. Regulated enterprises and platform teams that serve many internal consumers need Bifrost; product teams already on Cloudflare or Vercel can start with those platforms' gateways; and individual developers exploring models can start with OpenRouter.

Figure 4: Data residency and team type settle most gateway decisions before any feature comparison starts.

Use case	Recommended gateway	Reason
Regulated enterprise (finance, healthcare, public sector)	Bifrost	In-VPC, on-prem, and air-gapped deployment with audit logs and guardrails
Platform team serving many internal teams	Bifrost	Virtual keys and hierarchical budgets per team and customer
Coding agents (Claude Code, Codex CLI)	Bifrost	Anthropic and OpenAI-compatible endpoints, any model behind them, per-developer budgets
App already running on Cloudflare Workers	Cloudflare AI Gateway	Managed caching and analytics on the same platform
Next.js product team on Vercel	Vercel AI Gateway	AI SDK integration and budgets per project
Python prototype with modest traffic	LiteLLM	Self-hosted, Python SDK, broad provider catalog
Model exploration and demos	OpenRouter	One key for hundreds of hosted models

Coding-agent rollouts need their own planning because token volume per developer is high. The guide to choosing an AI gateway for Claude Code covers settings, model slots, and budgets, and the companion guide on how to route Codex CLI to any model covers the config.toml provider setup.

Whichever gateway you choose, the core layers stay the same; the pillar guide to AI gateway architecture and core features explains each one.

Frequently Asked Questions

What are AI gateways?

AI gateways are middleware that sit between applications and model providers, exposing one API while handling authentication, routing, retries, failover, caching, budgets, and logging. Applications send every model request to the gateway instead of calling providers directly. Bifrost, for example, exposes OpenAI, Anthropic, and Gemini-compatible endpoints that route to 25+ providers, so switching models becomes a configuration change rather than a code change.

What are the top AI gateways?

The top AI gateways in 2026 are Bifrost, Cloudflare AI Gateway, Vercel AI Gateway, LiteLLM, and OpenRouter. Bifrost leads for enterprise and platform teams because it is open source, self-hostable, and adds 11 µs of overhead at 5,000 RPS. Cloudflare and Vercel suit teams already on those platforms, LiteLLM suits Python prototypes, and OpenRouter suits model exploration.

Which AI gateways are open source?

Of the five gateways in this list, Bifrost and LiteLLM are open source. Bifrost is licensed under the Apache License 2.0 and ships virtual keys, budgets, rate limits, and fallbacks in the open-source build. LiteLLM's core is MIT-licensed, with SSO, audit logs, and some governance features under an enterprise license, while Cloudflare AI Gateway, Vercel AI Gateway, and OpenRouter are managed services. The open-source AI gateway roundup covers more options.

What are the best OpenRouter alternatives?

The best OpenRouter alternatives keep a single API across many models while adding self-hosting and per-team governance. Bifrost is the strongest option for production: it routes to 25+ providers, including OpenRouter itself, and adds virtual keys, hierarchical budgets, and fallback chains inside your own network. Vercel AI Gateway is a managed alternative for teams on Vercel, and LiteLLM is a self-hosted Python option.

How much latency does an AI gateway add?

Overhead depends mainly on the runtime. Bifrost adds 11 microseconds per request at a sustained 5,000 RPS in its published benchmark results, which is negligible next to model response times measured in seconds. Python-based gateways add more under concurrency, and most managed gateways do not publish an overhead figure, so ask vendors for sustained-load numbers before committing.

Should you self-host or use a managed AI gateway?

Self-host when prompts or logs cannot leave your network, when many teams share the gateway, or when request volume makes per-request platform fees expensive. Use a managed gateway when operational simplicity matters most and your compliance posture allows third-party data transit. Bifrost supports both paths for enterprises through Bifrost Enterprise deployments in a VPC, on-prem, or air-gapped.

Try Bifrost as Your AI Gateway

Among the AI gateways compared here, Bifrost is the one built for enterprise and platform teams: open source, self-hosted inside your network, 11 µs of overhead at 5,000 RPS, and governance through virtual keys and hierarchical budgets in the open-source build. Teams moving from OpenRouter or LiteLLM keep one OpenAI-compatible API and gain control over their data. To see how Bifrost fits your infrastructure, book a demo with the Bifrost team.

[Boost]

Kamya Shah — Tue, 06 Oct 2026 08:25:21 +0000

Kamya Shah

Oct 6

What Is an AI Gateway? Architecture, Features, and Why It Matters

#ai #apigateway #gateway #llm

17 min read

An AI gateway is the control layer between your apps and every LLM provider. Learn its architecture, request flow, core features, and how to choose one.

Kamya Shah — Tue, 06 Oct 2026 08:25:03 +0000

Kamya Shah

Oct 6

What Is an AI Gateway? Architecture, Features, and Why It Matters

#ai #apigateway #gateway #llm

17 min read