DEV Community: Swapnoneel Saha

Best Open-Source MCP Gateways for Secure AI Agent Access (2026)

Swapnoneel Saha — Wed, 30 Sep 2026 10:48:55 +0000

TL;DR

An open-source MCP gateway is the essential control plane for securing, governing, and observing AI agents that interact with external tools and APIs.
Key evaluation criteria for 2026 include performance overhead, governance capabilities (access control, budgets), security features, and deployment flexibility (self-hosted, air-gapped).
Bifrost is the best overall open-source MCP gateway for teams requiring microsecond latency, unified LLM and MCP governance, and enterprise-grade security in a self-hostable binary.
Other strong open-source options include Docker MCP Gateway for container-native workflows and Obot for teams seeking a comprehensive, self-hosted MCP platform.
Self-hosting an MCP gateway provides full control over data, credentials, and audit trails, which is a critical requirement for regulated industries and production AI workloads.

As AI agents become more autonomous, the need to govern their access to internal systems has become a primary security concern for engineering teams. The Model Context Protocol (MCP) has emerged as the open standard for this agent-to-tool communication, but the protocol itself doesn't solve for security or governance. This is where an MCP gateway becomes critical infrastructure. Bifrost, an open-source AI gateway from Maxim AI, is one of several solutions designed to provide this control layer.

An MCP gateway acts as a centralized proxy between AI clients (like Claude Desktop or custom agents) and the MCP servers that expose tools, APIs, and data. Instead of letting agents connect directly to a sprawling landscape of tools, the gateway enforces access control, manages credentials, and creates a complete audit trail for every tool call. This article compares the best open-source MCP gateways available in 2026 for teams that need to secure agent access while maintaining full control over their infrastructure.

Key Criteria for Evaluating Open-Source MCP Gateways

When selecting an MCP gateway, the focus has shifted from basic proxying to enterprise-grade governance and performance. A production-ready gateway must provide robust security without becoming a bottleneck.

Criterion	Why It Matters
Performance & Overhead	Every tool call an agent makes passes through the gateway. High latency here directly impacts the end-user experience of agentic applications. Look for gateways with overhead measured in microseconds.
Security & Governance	The core function of the gateway. This includes role-based access control (RBAC), per-user or per-agent virtual keys, budget and rate-limit enforcement, and the ability to filter which tools are accessible.
Authentication & Identity	Must support modern authentication protocols like OAuth 2.0 and integrate with enterprise identity providers (IdPs) like Okta or Entra ID to manage user and agent identities.
Observability	A centralized gateway is the ideal place to log and trace every tool interaction. Native support for OpenTelemetry (OTLP) and Prometheus metrics is essential for monitoring and auditing agent activity.
Deployment Model	For security and compliance, self-hosting is non-negotiable. The gateway must be deployable in any environment, including private clouds (VPC), on-premises data centers, and air-gapped networks.
Unified LLM & MCP Control	Agents interleave model calls and tool calls. A gateway that can govern both types of traffic through a single control plane simplifies operations, unifies cost tracking, and provides a complete trace of an agent's actions.

The Top 3 Open-Source MCP Gateways in 2026

Based on the criteria above, here is an assessment of the leading open-source options for securing AI agent access.

1. Bifrost

Bifrost is a high-performance, open-source gateway written in Go that unifies LLM routing and MCP governance in a single binary. It is designed for mission-critical workloads where performance and security are paramount.

Best for: Enterprise teams and performance-sensitive applications that need a single, self-hostable control plane for all AI traffic (both model and tool calls) with microsecond latency.

Key Features:

Unified Gateway: Bifrost manages both LLM requests and MCP tool calls, providing a single point of governance. This simplifies infrastructure and provides a holistic view of agent behavior.
High Performance: Adds less than 100 microseconds of overhead at 5,000 requests per second, ensuring that the gateway is never a bottleneck for agentic workflows.
Granular Governance: Uses virtual keys to enforce per-consumer budgets, rate limits, and access policies. It supports MCP tool filtering, allowing administrators to control precisely which tools are available to specific agents or users.
Advanced Security: Integrates with enterprise identity providers via OIDC and supports federated authentication for turning existing enterprise APIs into secure MCP tools.
Deployment Flexibility: Can be deployed as a single binary, in a Docker container, or in a clustered configuration for high availability within a VPC or on-premise environment.
Code Mode & Agent Mode: Features an optimized "Code Mode" that can reduce token consumption and latency for complex tool orchestration tasks.

Beyond the gateway, Bifrost's governance and security can be extended to the endpoint. Bifrost Edge ensures that AI traffic from desktop apps and coding agents on employee machines is routed through the gateway, applying the same security policies everywhere and preventing shadow AI.

2. Docker MCP Gateway

The Docker MCP Gateway is an open-source solution from Docker that excels at orchestrating MCP servers as containerized workloads. It is a natural fit for development teams already standardized on the Docker ecosystem.

Best for: Development teams and organizations that use Docker for container orchestration and want a secure, isolated environment for running each MCP server.

Key Features:

Container-per-Server Isolation: Runs each MCP server in its own sandboxed Docker container with restricted network and filesystem access, providing a strong security posture at the infrastructure layer.
Docker Ecosystem Integration: Deeply integrated with Docker Desktop and Docker Compose, offering a seamless local development experience.
Credential Management: Leverages Docker's built-in secrets management for handling credentials securely.
OCI-Based Catalog: Manages MCP servers as OCI images, enabling versioning and supply-chain security practices.

While excellent for development and container-native teams, it is less focused on being a unified, multi-user enterprise control plane for both LLM and MCP traffic compared to Bifrost.

3. Obot

Obot is an open-source MCP platform that includes a gateway, a server catalog, and agent orchestration tools. It is designed for teams who want to self-host their entire MCP stack and avoid vendor lock-in.

Best for: Organizations looking for a comprehensive, end-to-end open-source MCP platform to run on their own Kubernetes infrastructure.

Key Features:

Full Platform: Provides more than just a gateway; it's a complete system for managing the lifecycle of MCP servers and agents.
Kubernetes-Native: Designed to run on Kubernetes, making it a good fit for teams with existing cloud-native expertise.
Enterprise Identity: The enterprise version supports integration with IdPs like Okta and Microsoft Entra for centralized user authentication.
Self-Hosted Control: Gives teams complete control over their data, security, and infrastructure without relying on a hosted service.

Obot offers a broader platform scope but requires more operational overhead to manage the full Kubernetes-based deployment compared to a standalone gateway like Bifrost.

Recommendation

For the majority of teams seeking a secure, performant, and flexible open-source solution, Bifrost is the leading choice in 2026. Its unique ability to govern both LLM and MCP traffic through a single, low-latency control plane addresses the reality of modern agentic systems. The focus on enterprise-grade governance, comprehensive security features, and the ability to deploy anywhere—from a local machine to an air-gapped cluster—makes it the most versatile and production-ready option.

While Docker MCP Gateway and Obot are strong contenders in their respective niches (container-native workflows and full-platform self-hosting), Bifrost provides the best balance of performance, security, and operational simplicity for securing AI agent access at scale.

Frequently Asked Questions

What is an MCP gateway?

An MCP gateway is a centralized control layer that manages how AI agents interact with external tools, APIs, and data sources via the Model Context Protocol (MCP). It acts as a secure intermediary, handling authentication, access control, routing, auditing, and policy enforcement for all tool calls.

Why is an open-source MCP gateway important?

Open-source gateways provide transparency, customization, and full control over your infrastructure. For security-conscious organizations, this means data and credentials never have to pass through a third-party service, ensuring data sovereignty and simplifying compliance with regulations like GDPR and HIPAA.

How does an MCP gateway improve AI agent security?

It improves security by centralizing control. Instead of agents holding credentials directly, the gateway manages access. It enforces policies on which agents can use which tools, creates immutable audit logs of all activity, and can be integrated with enterprise identity systems to ensure only authorized users and agents can invoke sensitive tools.

Can an MCP gateway reduce costs?

Yes, by providing centralized observability and budget controls. A gateway like Bifrost can enforce spending limits per agent, user, or project. By logging every tool and model call, it provides the necessary data to identify and optimize costly agent behaviors.

What is the difference between an LLM gateway and an MCP gateway?

An LLM gateway manages requests to large language models (routing, failover, caching, etc.), while an MCP gateway manages requests from AI agents to external tools. Bifrost is unique in that it combines both functions into a single, unified gateway, recognizing that agentic workflows involve a constant mix of both traffic types.

Next Steps

Securing agent access is a foundational step in building reliable and enterprise-ready AI applications. Teams evaluating open-source MCP gateways can request a Bifrost demo to see how unified governance works in practice or review the open-source repository on GitHub.

Best AI Governance Tools for Responsible AI Usage in 2026

Swapnoneel Saha — Wed, 30 Sep 2026 10:44:30 +0000

TL;DR

AI governance tools are now a mandatory component for enterprises, driven by regulations like the EU AI Act and the need to manage risks from shadow AI and autonomous agents.
The market is split between two main categories: runtime enforcement platforms that control live AI traffic, and policy management platforms for compliance, risk assessment, and audit.
Runtime enforcement tools like Bifrost and Fiddler AI are critical for preventing policy violations in real-time, while platforms like Credo AI and IBM watsonx.governance excel at lifecycle management and documentation.
For teams that need a high-performance, open-source, and self-hosted control plane, the Bifrost AI gateway provides the strongest combination of access control, cost management, audit logging, and security.

The adoption of AI is no longer a question for most enterprises; the central challenge has shifted to control. As employees use third-party AI tools and engineering teams deploy autonomous agents, organizations face a growing governance gap. Without a dedicated framework, they risk data leakage, compliance violations, and operational incidents. AI governance platforms are the tools enterprises use to close this gap, providing the structure to enforce policies, manage risk, and ensure AI operates safely and responsibly.

This has become a strategic necessity, with frameworks like the NIST AI Risk Management Framework (AI RMF) and ISO/IEC 42001 now guiding enterprise strategy. The market for governance tools has matured to address these needs, but it has also split into distinct categories. Choosing the right tool depends on whether the primary goal is documenting policy for compliance or enforcing it on live traffic. This guide compares the top AI governance tools of 2026, with a focus on solutions for responsible enterprise AI usage.

Runtime vs. Policy: The Two Halves of AI Governance

AI governance tools generally fall into one of two categories, and many large enterprises use a tool from each.

Policy & Compliance Platforms: These are systems of record for an organization's AI governance program. They provide tools for creating an inventory of AI models, conducting risk assessments, mapping controls to regulatory requirements (like the EU AI Act), and generating audit-ready documentation. They help answer the questions: "What AI are we using?" and "Is it compliant with our policies?"
Runtime Enforcement Platforms: These tools sit in the path of live AI requests and enforce governance policies in real time. They act as a control plane for AI traffic, applying rules for access, data security, content safety, and cost before a request reaches a model and before a response reaches a user. They answer the question: "Can we stop a policy violation before it happens?"

A comprehensive governance strategy needs both. A policy is ineffective without enforcement, and enforcement is arbitrary without a clear policy. This review focuses on the tools that provide the technical controls necessary for a complete governance program.

Key Criteria for Evaluating AI Governance Tools

When assessing AI governance platforms, enterprises should look for a core set of capabilities that address the full lifecycle of AI usage.

Criterion	Description
Access Control & Identity	Granular control over who (users, teams, services) can access which models, with what budgets, and under which conditions. Support for SSO and RBAC is critical.
Policy Enforcement	The ability to define and enforce rules at runtime. This includes routing, rate limits, and guardrails for content, data (PII), and security (secrets).
Audit & Observability	Immutable, detailed logs of all AI requests, responses, and policy decisions. This is non-negotiable for compliance with SOC 2, HIPAA, or ISO 27001.
Shadow AI & Endpoint Governance	The ability to discover and govern unmanaged AI usage on employee devices (desktop apps, browser AI) to close compliance gaps.
Deployment & Integration	Flexibility to deploy in any environment (cloud, VPC, on-prem) and integrate with existing identity providers, observability stacks, and security tools.
Regulatory Alignment	Features that directly support compliance with major frameworks like the EU AI Act, NIST AI RMF, and ISO 42001.

The Best AI Governance Tools of 2026 at a Glance

This table compares the leading AI governance tools across the key evaluation criteria.

Tool	Primary Focus	Runtime Enforcement	Policy Management	Shadow AI Governance	Deployment Model
Bifrost	Runtime Enforcement	✅ Yes	➖ No	✅ Yes	Self-Hosted (Cloud/VPC/On-Prem)
Credo AI	Policy & Compliance	➖ No	✅ Yes	➖ No	SaaS
IBM watsonx.governance	Model Lifecycle	✅ Yes	✅ Yes	➖ No	SaaS (IBM Cloud) / Hybrid
OneTrust AI Governance	Privacy & GRC	➖ No	✅ Yes	➖ No	SaaS
Microsoft Purview	Data Governance	✅ Yes	✅ Yes	➖ No	SaaS (Azure)
Fiddler AI	AI Observability & Security	✅ Yes	➖ No	➖ No	SaaS / VPC

A Deeper Look at the Top AI Governance Platforms

1. Bifrost

Bifrost is a high-performance, open-source AI gateway from Maxim AI that acts as a centralized runtime enforcement point for all AI traffic. It unifies access to over 1000 models through a single API and applies governance policies to every request. Its position in the network allows it to enforce controls that policy-only platforms cannot.

Best for: Enterprises that need a high-performance, self-hosted, and extensible platform for runtime AI governance, security, and cost control.

Key Capabilities:

Unified Access & Cost Control: Bifrost uses virtual keys to manage access. Teams and services get unique keys with their own budgets, rate limits, and model permissions, providing granular control over spending and usage.
Runtime Policy Enforcement: As a gateway, Bifrost enforces routing rules, provider failover, and load balancing on every request, ensuring reliability and compliance with architectural policies.
Security and Guardrails: The platform integrates with security tools like AWS Bedrock Guardrails and Azure Content Safety to apply content moderation, PII redaction, and prompt injection defense at the gateway layer.
Immutable Audit Logs: Bifrost generates comprehensive, signed audit logs for every transaction, providing the evidence required for SOC 2, HIPAA, and ISO 27001 compliance.
Endpoint Governance with Bifrost Edge: Beyond the gateway, Bifrost Edge extends the same governance and security controls to AI traffic on employee machines. It provides visibility into and control over desktop apps and browser-based AI, addressing the "shadow AI" problem directly with endpoint enforcement.

2. Credo AI

Credo AI is a leading AI governance platform focused on policy, risk, and compliance management. It serves as a central registry for an organization's AI systems, helping teams document use cases, assess risks against established frameworks, and generate compliance reports.

Best for: GRC (Governance, Risk, and Compliance) teams and Chief AI Officers who need a system of record for managing AI policies and demonstrating regulatory compliance.

Key Capabilities:

AI Registry: Provides a centralized inventory to catalog all AI models, applications, and use cases across the enterprise.
Risk Assessment: Offers workflows to evaluate AI systems against fairness, performance, transparency, and security risks.
Policy Packs: Translates regulations and standards like the EU AI Act and NIST AI RMF into actionable assessment requirements and report templates.

3. IBM watsonx.governance

IBM watsonx.governance is an enterprise platform designed to govern the entire AI model lifecycle. It provides tools for tracking models from development to deployment, monitoring for bias and drift, and ensuring transparency and explainability.

Best for: Large enterprises, particularly those in regulated industries like finance and healthcare, that need to manage the lifecycle of a diverse portfolio of AI models.

Key Capabilities:

Model Lifecycle Management: Tracks model development, validation, deployment, and performance in a central inventory.
Bias and Drift Detection: Continuously monitors production models for fairness, bias, and performance degradation.
Explainability: Generates explanations for model predictions, helping teams understand and document how AI systems make decisions.

4. OneTrust AI Governance

OneTrust is a well-established platform in the privacy and trust management space, and its AI Governance solution extends these capabilities to AI systems. It helps organizations inventory AI use cases, assess them against privacy and ethical risks, and connect AI governance to broader compliance programs.

Best for: Organizations that already use OneTrust for privacy management and want to integrate AI governance into their existing GRC workflows.

Key Capabilities:

AI Inventory and Discovery: Helps organizations create a comprehensive inventory of their AI projects and understand where AI is being used with their data.
Risk Assessment: Provides templates and workflows for assessing AI projects against privacy regulations and ethical guidelines.
Regulatory Intelligence: Keeps teams updated on evolving AI laws and helps automate compliance workflows.

5. Microsoft Purview

For organizations heavily invested in the Microsoft ecosystem, Microsoft Purview provides unified data and AI governance. It extends Microsoft's data classification, lineage, and security capabilities to cover AI workloads running in Azure and Microsoft 365.

Best for: Enterprises using Azure Machine Learning, Azure OpenAI, and Microsoft Copilot that need integrated data security and governance.

Key Capabilities:

Unified Data Governance: Combines data discovery, classification, and policy enforcement across an organization's data estate.
AI-Specific Controls: Applies data security and compliance policies to AI interactions, including retaining prompts and responses for audit purposes.
Deep Microsoft Integration: Offers native policy enforcement and metadata synchronization for AI services within the Microsoft ecosystem.

6. Fiddler AI

Fiddler AI is an AI observability platform with strong capabilities for runtime governance and security. Its Fiddler Guardrails product provides real-time content moderation to detect and block harmful content, PII leaks, and hallucinations before they reach users.

Best for: ML engineering and security teams that need a real-time defense layer for their LLM applications.

Key Capabilities:

Real-Time Guardrails: Detects and blocks safety risks, PII, and leaked secrets in prompts and responses with low latency.
AI Observability: Provides deep monitoring of model performance, data drift, and explainability for production AI systems.
Hallucination Detection: Includes specific checks to identify and flag unfaithful or fabricated responses in RAG applications.

Recommendation: Choosing the Right Tool for the Job

The best AI governance tool depends entirely on the problem an organization is trying to solve.

For teams whose primary need is to document policies, create an AI inventory, and generate compliance reports for auditors, a policy and compliance platform like Credo AI or OneTrust is the logical starting point. These tools provide the necessary framework for establishing a formal governance program.

For teams whose primary need is to enforce those policies on live AI traffic, prevent data leaks, control costs, and create an immutable audit trail, a runtime enforcement platform is essential. In this category, Bifrost stands out as the most powerful and flexible option. Its open-source foundation, high-performance architecture, and comprehensive feature set—from virtual keys for cost control to Bifrost Edge for endpoint governance—provide a complete control plane for enterprise AI. While other tools offer components of runtime governance, Bifrost unifies them in a single, self-hostable gateway that gives organizations full control over their AI infrastructure.

Ultimately, a mature enterprise AI program needs both layers: a system of record for policy and a system of enforcement for control. Teams evaluating AI governance solutions can request a Bifrost demo or review the open-source repository to see how a runtime gateway can serve as the foundation for a responsible AI strategy.

Frequently Asked Questions

What is AI governance?

AI governance is the framework of policies, processes, and controls an organization uses to ensure its AI systems operate ethically, securely, and in compliance with laws and regulations. It covers the entire AI lifecycle, from development to deployment and monitoring.

Why is AI governance important in 2026?

With the rapid adoption of powerful AI and the introduction of binding regulations like the EU AI Act, governance is no longer optional. It is a requirement for managing significant financial, reputational, and operational risks, and for building trust with customers and regulators.

What is the difference between AI governance and AI security?

AI governance is a broad discipline focused on oversight, policy, and risk management. AI security is a subset of governance that focuses specifically on protecting AI systems from threats like data leakage, model theft, and prompt injection attacks. Effective governance tools often include strong security features.

How do you govern "shadow AI"?

Shadow AI—the unapproved use of AI tools by employees—is a major governance challenge. The most effective way to govern it is with an endpoint agent, like Bifrost Edge, that can discover, monitor, and apply policies to AI applications running on employee devices, regardless of how they are connected.

What are the most important features of an AI governance tool?

The most critical features include granular access control (who can use what), real-time policy enforcement (guardrails for content and data), comprehensive audit logging, and the flexibility to deploy in your own environment to maintain data control.

Sources

Top 5 Enterprise AI Gateways for Multi-Model Routing in 2026

Swapnoneel Saha — Wed, 30 Sep 2026 10:40:22 +0000

TL;DR

An AI gateway is essential infrastructure for production AI, providing a unified control plane for routing, failover, governance, and observability across multiple LLM providers.
Multi-model routing optimizes for cost, latency, and capability by sending each request to the most appropriate model, rather than relying on a single, expensive one.
Bifrost is a top choice for enterprises needing high performance and comprehensive governance in a self-hosted or in-VPC deployment.
LiteLLM offers the broadest provider support, making it ideal for teams prioritizing model variety in a self-hosted, open-source environment.
Kong, Cloudflare, and Vercel are strong contenders for teams already invested in their respective ecosystems, offering integrated AI routing at the API or edge layer.

As enterprise AI moves from single-model prototypes to production systems, engineering teams face a common set of challenges: provider outages, unpredictable costs, and the operational complexity of managing dozens of models and API keys. A single-model strategy is no longer viable; no one model is the cheapest, fastest, and most capable for every task. This reality has made the AI gateway a critical piece of infrastructure.

An AI gateway, or LLM gateway, is a dedicated layer that sits between applications and AI providers. It unifies access to thousands of models through a single, consistent API and centralizes the logic for multi-model routing. Instead of embedding provider-specific SDKs and routing rules in every service, teams point their applications at the gateway and manage policy from one place. This article compares the top five enterprise AI gateways for 2026, focusing on their capabilities for multi-model routing.

Key Criteria for Evaluating Enterprise AI Gateways

Before comparing platforms, it's important to establish the criteria that matter for enterprise use cases. Beyond a simple unified API, a true enterprise gateway must provide robust controls across several key areas.

Capability	Description	Why It Matters for Enterprises
Routing Strategy	The logic used to select a model for each request. Common strategies include failover, load balancing, cost-based, latency-based, and semantic routing.	Determines an organization's ability to optimize for cost, user experience, and reliability without application code changes.
Deployment Model	How the gateway is hosted: fully managed SaaS, self-hosted in a public cloud, in a VPC, on-premise, or in an air-gapped environment.	Enterprises in regulated industries often have strict data residency and security requirements that mandate self-hosted or in-VPC deployments.
Performance	The latency overhead the gateway adds to each request, especially under load.	For real-time, user-facing AI applications, gateway latency can be a significant bottleneck. Low-overhead gateways are critical for production performance.
Governance	Features for managing access, cost, and compliance, such as virtual keys, per-user budgets, RBAC, and audit logs.	Provides the financial and security controls necessary to operate AI safely at scale, preventing budget overruns and unauthorized access.
Observability	The ability to log, trace, and monitor every request with detailed information on token usage, cost, latency, and provider choice.	Essential for debugging, performance tuning, and providing finance and compliance teams with a clear view of AI usage and spend.
MCP Support	Native support for the Model Context Protocol (MCP) to manage and govern connections to external tools and agents.	As AI becomes more agentic, governing tool use becomes as important as governing model use. A gateway with MCP support provides a unified control plane.

The Top 5 Platforms Compared

Based on these criteria, here is a detailed comparison of the leading enterprise AI gateways.

1. Bifrost

Bifrost is a high-performance, open-source AI gateway from Maxim AI, written in Go. It is designed for enterprise-grade workloads that demand low latency and comprehensive governance. Bifrost unifies access to over 1,000 models from more than 25 providers through a single OpenAI-compatible API.

Its primary differentiator is performance; published benchmarks show it adds only 11 microseconds of overhead per request at a sustained load of 5,000 requests per second. This makes it suitable for latency-sensitive applications like real-time copilots and conversational agents.

Routing Capabilities:

Automatic Failover: Bifrost provides robust fallback chains. When a request to a primary provider fails, it automatically retries with exponential backoff before failing over to the next provider or model in a pre-configured sequence.
Load Balancing: It can perform weighted load balancing across multiple API keys for the same provider or across different providers to distribute traffic and avoid rate limits.
Routing Rules: Administrators can define rules to direct specific types of requests to certain models or providers based on request metadata.
Semantic Caching: Reduces costs and improves latency by caching responses based on the semantic meaning of a query, not just an exact string match.

Best for: Enterprises that require a self-hosted, high-performance gateway with a full suite of governance and security features. Its ability to be deployed in a VPC or an air-gapped environment makes it a strong choice for organizations in finance, healthcare, and other regulated industries.

2. LiteLLM

LiteLLM is a popular and mature open-source AI gateway known for its extensive provider support, covering over 140 providers and 1,800+ models. Built in Python, it has a large community and is often the first gateway to support new and emerging models.

Routing Capabilities:

Flexible Strategies: LiteLLM offers multiple routing strategies out of the box, including simple-shuffle, least-busy, latency-based, and cost-based routing.
Auto Routing: A beta feature allows for more intelligent routing by classifying request complexity and routing to different tiers of models accordingly.
Granular Control: Routing strategies, fallbacks, and timeouts can be configured on a per-key or per-team basis, allowing for highly customized behavior across an organization.

Best for: Teams that prioritize the breadth of model compatibility and want a flexible, open-source solution they can self-host and customize. Its Python base makes it highly accessible for many development teams.

3. Kong AI Gateway

The Kong AI Gateway extends the widely adopted Kong API Gateway with AI-specific capabilities. For organizations already using Kong to manage their microservices and APIs, this provides a familiar control plane for also managing LLM traffic.

Routing Capabilities:

Model-Based Routing: Kong can inspect incoming prompts and route them to different upstream providers based on complexity. A common pattern is routing simple requests to a fast, cheap model and complex reasoning tasks to a more powerful, expensive one.
Plugin Architecture: AI features are delivered via plugins, such as ai-proxy-advanced, which handle credential injection, format translation, and routing logic.
Unified Observability: AI traffic metrics are integrated into Kong Konnect's analytics dashboards, alongside standard API traffic.

Best for: Enterprises already invested in the Kong ecosystem. It allows platform teams to apply their existing API management policies (authentication, rate limiting, logging) to AI workloads with minimal friction.

4. Cloudflare AI Gateway

Cloudflare's AI Gateway is a managed service that leverages Cloudflare's global edge network to provide observability, caching, and routing for AI applications. It's designed for ease of use and integrates tightly with other Cloudflare products like Workers AI.

Routing Capabilities:

Dynamic Routing: Allows configuration of rules to automatically failover between providers when one is unavailable. If a model is available on multiple providers, the gateway can route around an outage without manual intervention.
Model-First Routing: Cloudflare is moving towards a model where developers request a specific model (e.g., Llama 3), and the gateway decides the best provider (including Cloudflare's own Workers AI) to serve it from based on availability and capacity.
Caching at the Edge: Caching responses directly on Cloudflare's edge network can significantly reduce latency for users globally.

Best for: Teams building applications on the Cloudflare stack or those who prefer a fully managed, zero-ops solution. Its strength lies in its global performance and seamless integration with Workers.

5. Vercel AI Gateway

Similar to Cloudflare, the Vercel AI Gateway is a managed service tightly integrated into the Vercel ecosystem, primarily serving applications built with Next.js and the Vercel AI SDK. It focuses on providing developers with a simple way to access hundreds of models without managing credentials or complex routing logic.

Routing Capabilities:

Automatic Failover: The gateway can be configured with ordered fallbacks, ensuring that if a request to a primary model fails, it transparently retries with an alternative.
Cost-Aware Routing: Vercel provides patterns for developers to build their own classifiers that route simple, low-value requests to cheap models and escalate complex ones to more capable frontier models, all through a single gateway endpoint.
Firewall-Style Rules: Administrators can set rules to restrict which models the team is allowed to use, enforcing governance at the gateway level.

Best for: Development teams building on Vercel and Next.js. The tight integration with the Vercel AI SDK and platform provides a streamlined developer experience for building AI-powered frontends.

Comparison at a Glance

Feature	Bifrost	LiteLLM	Kong AI Gateway	Cloudflare AI Gateway	Vercel AI Gateway
Deployment	Self-hosted, In-VPC, Air-gapped	Self-hosted	Self-hosted, Managed (Konnect)	Managed	Managed
Core Tech	Go	Python	NGINX (Lua)	Managed Service	Managed Service
Performance	Very High (<15 µs overhead)	Moderate	High	High (Edge Network)	High
Routing	Failover, Weighted LB, Rules	Cost, Latency, Usage-based	Prompt-aware, Plugin-driven	Dynamic Failover	Failover, Cost-aware
Governance	Virtual Keys, Budgets, RBAC, Audit Logs	Virtual Keys, Budgets, Teams	Consumer Auth, RBAC	Rate Limiting, Logs	Budgets, Allow-lists
MCP Support	Yes (Native)	Yes	Yes (Enterprise)	No	No
Open Source	Yes (Apache 2.0)	Yes (MIT)	Yes (Core Gateway)	No	No

Conclusion

Choosing an enterprise AI gateway is a foundational architectural decision. The right choice depends less on a raw feature list and more on your organization's specific needs for deployment, performance, and governance.

For enterprises requiring maximum control, performance, and security within their own infrastructure, a self-hosted solution like Bifrost provides the most comprehensive feature set, from granular governance to native MCP support. For teams prioritizing the widest possible model support in an open-source package, LiteLLM is a strong and flexible choice. For organizations already committed to an existing platform, the AI gateways from Kong, Cloudflare, and Vercel offer a compelling path to unified AI routing and observability with lower operational overhead.

Ultimately, the goal of a multi-model routing strategy is to build more resilient, cost-effective, and capable AI applications. The gateways listed here provide the essential infrastructure to achieve that goal at enterprise scale.

Sources

Top Enterprise LLM Observability Tools in 2026

Swapnoneel Saha — Wed, 30 Sep 2026 10:35:33 +0000

TL;DR

LLM observability platforms are essential for debugging, evaluating, and monitoring production AI applications, which fail in ways traditional APM tools can't detect.
The market has three main approaches: traditional APM vendors adding LLM features, AI-native tracing tools, and AI gateways that provide observability at the infrastructure layer.
An AI gateway is the most effective point for enterprise observability, as it captures telemetry from every model, agent, and application without requiring code-level instrumentation.
Bifrost, an open-source AI gateway, provides comprehensive, low-overhead observability by default, emitting LLM-native metrics, traces, and logs compatible with existing enterprise monitoring stacks.
When evaluating tools, key criteria include data ownership, OpenTelemetry support, performance overhead, multi-agent tracing, and integration with governance and security controls.

The use of large language models (LLMs) in production is no longer a question of if but how. As enterprises deploy AI agents and copilots, they face a new class of operational challenges. Models hallucinate, agentic workflows get stuck in loops, and a single bad prompt can cause costs to skyrocket. Traditional application performance monitoring (APM) tools, built for a world of deterministic software, are blind to these failures. An API call returning a 200 status code says nothing about whether the response was accurate, safe, or helpful.

This is the gap LLM observability tools fill. They provide the deep visibility needed to trace, debug, and evaluate the non-deterministic behavior of AI systems. For enterprises, where reliability, security, and cost control are non-negotiable, choosing the right observability strategy is critical. This guide compares the leading enterprise LLM observability tools in 2026, with a focus on where and how telemetry is captured. While several approaches exist, capturing observability signals at the AI gateway layer offers the most comprehensive and scalable solution. Bifrost, an open-source AI gateway from Maxim AI, exemplifies this approach, providing rich, contextual telemetry for every AI request that passes through it.

Why the Gateway is the Right Place for Observability

LLM observability can be implemented at multiple levels: SDKs, application code, or a centralized gateway. For an enterprise, the gateway is the superior choice for three reasons:

Universal Coverage: A gateway sees every request from every application, team, and agent. Instrumenting the gateway once provides visibility into all AI traffic, including "shadow AI" usage from uninstrumented coding agents or desktop apps. This eliminates the need to add and maintain observability code in every single application.
Standardized Telemetry: The gateway emits a consistent set of metrics, traces, and logs regardless of the originating application or the destination model provider. This creates a unified source of truth for cost attribution, performance analysis, and error tracking across the entire organization.
Zero Application Overhead: Observability at the gateway has a negligible impact on application performance. High-performance gateways like Bifrost add mere microseconds of latency, ensuring that monitoring doesn't slow down production systems.

Key Criteria for Evaluating Enterprise LLM Observability Tools

When assessing solutions, enterprises should look beyond dashboards and focus on foundational capabilities:

Deployment & Data Control: Can the tool be self-hosted in a VPC or on-premises for maximum data control and to meet compliance requirements?
Performance Overhead: What is the latency impact on production AI requests? Solutions should have minimal, predictable overhead under load.
Open Standards Support: Does the platform natively support OpenTelemetry for metrics and traces? This avoids vendor lock-in and ensures compatibility with existing stacks like Prometheus, Grafana, and Datadog.
Agent & Tool (MCP) Tracing: Can the tool trace multi-step agent workflows, including tool calls made via the Model Context Protocol (MCP)? A simple prompt-response log is insufficient for modern agents.
Integration with Governance: Does the observability data connect directly to governance controls like virtual keys, budgets, rate limits, and security guardrails? Visibility without control is an incomplete solution.

The Top LLM Observability Tools for Enterprises in 2026

The market offers several strong contenders, each with a different architectural philosophy.

1. Bifrost: Gateway-Native Observability

Bifrost is a high-performance, open-source AI gateway that provides deep observability as a built-in, first-class feature. Because it sits at the intersection of all AI traffic, it is the natural control point for capturing comprehensive telemetry.

Bifrost's approach is unique in that observability is not an add-on; it's part of the core infrastructure. It automatically captures detailed metadata for every request and response, including tokens, costs, latency, and provider details, all with negligible performance impact.

Key Capabilities:

Built-in Telemetry: Bifrost emits native Prometheus metrics and distributed traces via OpenTelemetry (OTLP), making it compatible with virtually any modern observability backend, including Grafana, Jaeger, and Datadog.
AI-Native Signals: It generates telemetry that standard tools miss, such as per-request token counts, cost data, provider fallback events, and MCP tool-call spans for agentic workflows.
Asynchronous Logging: A powerful logging plugin captures full request/response payloads asynchronously, ensuring that detailed tracing has zero impact on request latency.
Unified Governance: Observability is tied directly to Bifrost's governance model. All telemetry is tagged with the corresponding virtual key, allowing for precise cost attribution and usage monitoring by team, project, or user.

Best for: Enterprises that require a scalable, secure, and high-performance solution for unifying observability and governance at the infrastructure layer. Its ability to be self-hosted and its open-standards support make it ideal for regulated industries and organizations with existing monitoring stacks.

2. Datadog LLM Observability

Datadog has extended its market-leading APM platform to include LLM-specific observability. For organizations already invested in the Datadog ecosystem, this provides a single pane of glass for monitoring infrastructure, applications, and AI models.

Datadog excels at correlating LLM performance with underlying infrastructure metrics. It traces LLM calls from within application code using its libraries and provides dashboards for tracking tokens, latency, and errors. Bifrost also features a native Datadog connector, allowing teams to combine the benefits of gateway-level capture with Datadog's analysis and visualization tools.

Best for: Companies already standardized on Datadog for their APM and infrastructure monitoring who want to add LLM visibility within the same platform.

3. Langfuse

Langfuse is an open-source LLM engineering platform that combines tracing, prompt management, and evaluation capabilities. It is developer-centric and provides detailed, session-based replays that are useful for debugging complex agent conversations.

Langfuse requires instrumenting application code with its SDK to capture traces. It offers a clean UI for exploring traces and allows teams to create datasets for fine-tuning or evaluation from production data.

Best for: Development teams looking for an open-source, all-in-one platform to debug, evaluate, and manage prompts during the development lifecycle.

4. Arize AI (Phoenix)

Arize AI focuses on ML monitoring and has a strong open-source offering called Phoenix. Phoenix is particularly effective at detecting model drift and evaluating the quality of RAG (Retrieval-Augmented Generation) pipelines. It provides tools to analyze embeddings and visualize how retrieval quality impacts final responses.

Like Langfuse, Phoenix generally relies on in-app instrumentation to collect data. Its strength lies in post-production analysis and evaluation rather than real-time gateway control.

Best for: ML engineering teams that need to diagnose and troubleshoot complex RAG systems and monitor for subtle drifts in model quality over time.

Comparison at a Glance

Feature	Bifrost	Datadog LLM Observability	Langfuse	Arize Phoenix
Capture Point	AI Gateway	Application SDK	Application SDK	Application SDK
Deployment	Self-hosted (OSS), Cloud	SaaS	Self-hosted (OSS), Cloud	Self-hosted (OSS), Cloud
OpenTelemetry	Native Support	Yes	Yes	Yes
Agent Tracing	Yes (LLM + MCP)	LLM Tracing	Yes	RAG Evaluation
Performance	<15µs overhead	Low (SDK-dependent)	Low (SDK-dependent)	Low (SDK-dependent)
Governance	Integrated	Separate	Separate	Separate

Recommendation

For enterprises, LLM observability cannot be an isolated function; it must be an integrated part of the AI infrastructure stack, connected to security, governance, and cost management. While AI-native and APM tools offer valuable insights, they often miss traffic from uninstrumented systems and lack direct control mechanisms.

An AI gateway provides the most robust and comprehensive foundation for enterprise-grade observability. By capturing standardized telemetry from every AI request at the source, it delivers universal visibility without compromising performance.

Bifrost stands out as the top choice for enterprises in 2026. Its combination of high performance, open-source flexibility, native OpenTelemetry support, and integrated governance makes it the most effective platform for understanding and controlling production AI systems at scale. Teams can adopt Bifrost to centralize observability and then feed that rich, gateway-level data into downstream systems like Datadog or Grafana, getting the best of both worlds.

To learn more, teams can review the Bifrost documentation or request a demo.

Sources

Top 5 Enterprise LLM Observability Platforms in 2026

Swapnoneel Saha — Wed, 30 Sep 2026 10:30:16 +0000

TL;DR

LLM observability provides visibility into the performance, quality, cost, and security of AI applications, which traditional monitoring tools cannot capture.
Enterprise platforms are evaluated on tracing depth, evaluation capabilities, security and governance features, cost management, and deployment flexibility.
AI Gateways like Bifrost provide observability as a function of centrally routing all AI traffic, capturing complete data without SDK-level changes.
APM extensions from vendors like Datadog and New Relic integrate LLM tracing into existing infrastructure monitoring stacks.
AI-native platforms like Arize AI and Langfuse offer deep, purpose-built tracing and evaluation features, often with open-source options.

LLM observability has become a critical component of the enterprise AI stack. Traditional application performance monitoring (APM) can confirm that an API returned a 200 OK status, but it cannot tell you if the model's response was a hallucination, if it leaked sensitive data, or if a minor prompt change just increased costs by 300%. LLM observability platforms are designed to answer these questions by tracking the inputs, outputs, and internal states of AI applications.

This article compares the top five enterprise LLM observability platforms for 2026, assessing them on the features that matter most for production AI: tracing, evaluation, cost control, security, and integration. The platforms fall into three categories: AI gateways, APM extensions, and AI-native tracing tools.

Key Criteria for Evaluating Enterprise LLM Observability

Before comparing the tools, it's important to establish a framework. For enterprise use, an LLM observability solution must go beyond basic logging and provide robust, scalable, and secure insights.

Criteria	Description	Why It Matters for Enterprises
Full Request & Response Tracing	Captures the entire payload, including prompts, responses, model parameters, tool calls, and retrieval steps.	Essential for debugging, auditing, and understanding model behavior. Incomplete data leaves blind spots.
Evaluation & Quality Scoring	Capabilities to assess outputs against defined metrics (e.g., faithfulness, relevance, toxicity) via automated evaluators or human feedback.	Moves beyond operational metrics (latency, errors) to measure the actual quality and safety of AI responses.
Cost & Token Tracking	Granular visibility into token consumption and cost per request, aggregated by user, project, or model.	Enables cost control, budget enforcement, and identification of expensive outliers before they impact the bottom line.
Security & Governance	Features like PII/secrets redaction, access controls (RBAC), audit logs, and policy enforcement (guardrails).	Critical for compliance, protecting sensitive data, and managing risk in regulated industries.
Integration & Deployment	Support for OpenTelemetry, compatibility with existing stacks (APM, data lakes), and flexible deployment options (cloud, VPC, self-hosted).	Reduces vendor lock-in, integrates with existing workflows, and meets enterprise data residency and security requirements.

The Top 5 Platforms Compared

This list assesses the leading platforms, highlighting their strengths and ideal use cases for enterprise teams.

1. Bifrost: Observability Through a Unified AI Gateway

Bifrost is a high-performance, open-source AI gateway that provides observability as a native function of its architecture. By centralizing all LLM traffic through a single point, Bifrost captures complete, structured data for every request and response automatically, without requiring developers to instrument each application with an SDK.

Bifrost's approach is powerful for enterprises because it guarantees 100% visibility. Since all requests—from any team, application, or model—must pass through the gateway, nothing is missed. This makes it a strong foundation for governance and cost control.

Key Features:

Built-in, Asynchronous Logging: Bifrost logs the full request context—inputs, outputs, tokens, cost, latency, provider, and model—with negligible impact on performance.
OpenTelemetry & Prometheus Exports: Natively exports traces and metrics to existing monitoring systems like Datadog, New Relic, Grafana, and Honeycomb, integrating seamlessly into established enterprise stacks.
Integrated Governance and Security: Because observability is part of the gateway, the same platform enforces governance controls. Virtual keys, budgets, rate limits, and guardrails are applied to the same traffic being observed.
Enterprise-Grade Guardrails: Connects to a wide range of content safety providers, including AWS Bedrock Guardrails, Azure Content Safety, and Google Model Armor, to enforce security policies on live traffic.
Endpoint Governance with Bifrost Edge: A key differentiator is Bifrost Edge, which extends the gateway's observability and security policies to AI usage on employee machines, covering desktop apps and browser-based AI to mitigate shadow AI risks.

Best for: Enterprises that need a centralized control plane for all AI traffic, combining comprehensive observability with robust security, governance, and cost management in a single platform. Its high-performance, deployment flexibility (including in-VPC and air-gapped environments) makes it ideal for regulated industries and mission-critical applications.

2. Datadog Agent Observability

Datadog is a dominant player in the traditional APM and infrastructure monitoring space. Its Agent Observability (formerly LLM Observability) product extends its existing platform to cover AI applications. For companies already invested in the Datadog ecosystem, this offers a path to LLM monitoring with minimal vendor sprawl.

Datadog's core strength is its ability to correlate LLM traces with the rest of the application stack. Teams can see how a model's latency impacts overall service performance, linking AI behavior to infrastructure metrics, logs, and user experience data in one place.

Key Features:

Unified Monitoring: Views LLM performance alongside application and infrastructure metrics.
Auto-Instrumentation: Provides libraries for popular frameworks to automatically capture trace data.
Built-in Quality & Safety Checks: Includes out-of-the-box evaluations for metrics like toxicity and topic relevance.
Cost and Token Tracking: Monitors usage to help manage expenses.

Best for: Organizations already standardized on Datadog for their primary monitoring needs. It provides a single pane of glass for teams who want to add LLM visibility to their existing APM workflows without onboarding a new vendor.

3. New Relic AI Observability

Similar to Datadog, New Relic has extended its established APM platform to include AI observability. It leverages the OpenTelemetry standard, which gives teams more flexibility and reduces vendor lock-in compared to proprietary instrumentation.

New Relic focuses on providing a holistic view of the AI stack, from the application layer down to the infrastructure. It offers pre-built dashboards and integrations for popular AI frameworks and providers, allowing teams to get started quickly.

Key Features:

OpenTelemetry Native: Built on the open standard for telemetry, ensuring data portability.
Full-Stack Visibility: Connects AI layer performance with application and infrastructure health.
Real-time Insights: Offers dashboards for tracking performance, errors, and cost issues.
Model Context Protocol (MCP) Monitoring: Provides visibility into the entire MCP request lifecycle for agentic applications.

Best for: Companies that use New Relic as their APM and want to add LLM monitoring within that ecosystem, especially those who prioritize open standards like OpenTelemetry.

4. Arize AI

Arize AI is an AI-native observability platform that offers deep capabilities for both LLM evaluation and production monitoring. It is split into two main products: Phoenix, a popular open-source library for tracing and evaluation during development, and AX, the enterprise-scale monitoring platform.

Arize's strength lies in its comprehensive, model-centric approach. It's designed from the ground up to handle the nuances of AI systems, with strong features for monitoring embedding drift, evaluating RAG pipeline performance, and debugging complex agent workflows.

Key Features:

Open-Source Foundation: Phoenix allows teams to start with a powerful, self-hostable tool for tracing and evaluation.
Enterprise-Scale Monitoring: Arize AX is built for high-volume production environments, processing trillions of events per month for its customers.
Deep Evaluation Tooling: Offers advanced evaluation capabilities, including support for LLM-as-a-judge and integration with various evaluation frameworks.
OpenTelemetry-Based: Uses the OpenInference standard, which is built on OpenTelemetry, for vendor-agnostic instrumentation.

Best for: ML-focused teams and enterprises that require deep, AI-native tracing and evaluation capabilities across the entire model lifecycle. The open-source entry point makes it accessible, while the enterprise platform provides the scale and features needed for large-scale production deployments.

5. Langfuse

Langfuse is an open-source LLM engineering platform that combines observability, prompt management, and evaluation in one integrated system. It has gained significant traction for its developer-friendly workflow and comprehensive feature set.

Langfuse provides detailed, hierarchical traces that capture every step of an LLM application's execution, from model calls to tool use and retrieval steps. Its open-source nature allows for self-hosting, giving enterprises full control over their data—a critical requirement for privacy and compliance.

Key Features:

Open Source and Self-Hostable: Gives teams complete data sovereignty and control.
Integrated Platform: Combines tracing, prompt management, and evaluation in a single workflow.
Detailed Tracing: Offers hierarchical views of complex agent interactions.
Framework Agnostic: Integrates with various LLM frameworks and models.

Best for: Teams looking for a powerful, open-source platform that covers the entire LLM development lifecycle. Its self-hosting capability makes it a strong choice for organizations with strict data residency requirements or those who prefer to build on an open-source stack.

Frequently Asked Questions

What is the difference between LLM observability and traditional APM?

Traditional Application Performance Monitoring (APM) tracks operational metrics like latency, error rates, and resource usage. LLM observability goes further by analyzing the quality and content of AI responses, tracking things like hallucinations, relevance, token costs, and potential data leakage—issues that traditional APM cannot detect.

Why is OpenTelemetry important for LLM observability?

OpenTelemetry is an open standard for instrumenting, generating, and collecting telemetry data (traces, metrics, logs). Using an OpenTelemetry-native platform prevents vendor lock-in, as the instrumentation in your code is not tied to a specific vendor's SDK. This allows enterprises to switch observability backends without re-instrumenting their applications.

How does an AI gateway provide observability?

An AI gateway acts as a central proxy for all LLM API requests. Because every request and response flows through it, the gateway can log the complete data payload for every transaction automatically. This provides 100% visibility without requiring developers to add observability code to each individual application.

Can these platforms detect security issues like prompt injection?

Yes, advanced LLM observability platforms can help detect security threats. By monitoring prompts for anomalous patterns and applying guardrails, they can flag potential prompt injection attacks or attempts to leak sensitive data. Platforms with integrated guardrail systems provide an active defense layer, not just passive monitoring.

How do I choose the right platform for my organization?

The right choice depends on your existing infrastructure and primary needs.

If you need a unified control plane for governance, security, and observability, an AI gateway like Bifrost is the most comprehensive solution.
If you are already heavily invested in an APM platform, using the LLM module from Datadog or New Relic is the path of least resistance.
If your primary need is deep, AI-native evaluation and tracing, a purpose-built platform like Arize AI or Langfuse is likely the best fit.

Next Steps

Choosing an enterprise LLM observability platform is a strategic decision that impacts reliability, security, and cost. Gateways provide a holistic control plane, APM extensions offer integration with existing stacks, and AI-native tools deliver specialized depth. Teams evaluating these options can request a Bifrost demo to see how gateway-based observability works or explore the open-source repository to get started.

5 Best MCP Gateways in 2026

Swapnoneel Saha — Tue, 15 Sep 2026 11:44:59 +0000

If you’re building AI agents in 2026, the difficult part is no longer making a model call.

It’s making the model safely use real systems.

GitHub access. Databases. CRMs. Internal APIs. Cloud infrastructure. File systems. SaaS applications. Every MCP server introduces another transport, credential, tool catalog, session, permission model, and operational surface.

Among the options below, Bifrost by Maxim AI stands out for pairing an Apache 2.0 open-source core with a production-focused control plane. Its GitHub repository is available for teams that want to inspect, run, and extend the gateway themselves.

At small scale, connecting an agent directly to a few MCP servers works fine.

At production scale, teams start dealing with:

Credentials scattered across agent configurations
Hundreds of tools bloating the model context
Destructive tools exposed to the wrong users
No unified audit trail for tool calls
MCP servers using different transports
OAuth tokens expiring during long-running workflows
No reliable way to attribute tool usage to a user, team, or customer

That’s where MCP gateways become core infrastructure.

An MCP gateway sits between AI clients and MCP servers, giving teams one place to manage discovery, authentication, authorization, routing, observability, and policy enforcement.

This article compares the 5 best MCP gateways in 2026 based on real production concerns: security, transport support, governance, reliability, deployment flexibility, developer experience, and cost control.

We’ll start with a quick comparison table, then examine how each gateway behaves in production.

What Is an MCP Gateway (And Why Does It Matter in Production)?

MCP is an open protocol for connecting AI applications with external data sources and tools.

An MCP server can expose:

Tools that an AI model can call
Resources that provide context or data
Prompts and reusable workflows
Authentication and session handling

An MCP gateway sits in front of one or more MCP servers and presents a controlled interface to AI clients such as Claude, Cursor, ChatGPT, VS Code, or custom agents.

A production MCP gateway should provide:

One endpoint for multiple MCP servers
Centralized authentication and credential management
Tool-level filtering and access control
User, team, agent, or tenant-level permissions
Session-aware routing
Rate limits and usage budgets
Per-call logs, metrics, and traces
Support for remote and local MCP transports
Safe handling of OAuth tokens and secrets

Without a gateway, every agent becomes its own integration hub.

That means each agent must independently manage server URLs, tool schemas, credentials, retries, permissions, and logs. The result is an N×M integration problem: N agents connected directly to M tools.

An MCP gateway reduces that complexity to a governed control point.

How We Evaluated These MCP Gateways

This comparison focuses on production readiness rather than feature-count marketing.

The main criteria were:

Security and identity: OAuth, SSO, RBAC, delegated access, and secret handling
Tool governance: allowlists, virtual servers, approvals, scopes, and policy enforcement
Transport compatibility: STDIO, HTTP, SSE, and Streamable HTTP
Observability: audit logs, latency, errors, traces, and usage attribution
Reliability: session handling, retries, scaling, and high availability
Deployment: SaaS, VPC, on-premises, Kubernetes, and air-gapped options
Context efficiency: discovery, schema loading, and tool-token reduction
Pricing and operating model

The ranking is based on production breadth and architectural fit, not on one synthetic cross-vendor benchmark.

Quick Comparison: Top 5 MCP Gateways

Gateway	Delivery	Open Source	Strengths	Tradeoffs	Pricing Model	Best For
Bifrost	Self-hosted single binary and Enterprise	✓ Apache 2.0 core	Unified MCP, LLM, and agent control plane; governance; air-gapped deployment; high performance	Enterprise features require a paid tier	Free self-hosted; Enterprise custom	Production AI systems that need one control plane
IBM ContextForge	Self-hosted Python, Docker, and Kubernetes	✓ Apache 2.0	MCP, A2A, REST/gRPC federation; registry; plugins; observability	Broad architecture requires more operational ownership	Free software; infrastructure costs apply	Platform teams that want open-source federation
Kong AI Gateway	Konnect-managed or self-hosted	Varies	API-to-MCP mapping; aggregation; authentication; policies; metrics and audit logs	Heavier API-management setup	Commercial and enterprise pricing	Existing Kong and API-platform teams
Azure API Management AI Gateway	Managed Azure AI gateway and API Management	✗	One governed endpoint for models and MCP; OpenAPI-to-MCP; connectors; runtime keys; policies	Public preview; regions, limits, and pricing may change	Preview pricing to be announced; Azure costs apply	Azure-native platform teams
Arcade MCP Gateways	Managed Cloud, VPC, and air-gapped Enterprise	✗	Tool curation; user-scoped OAuth; remote MCP federation; action-focused integrations	Remote MCP support is Streamable HTTP and tools-focused	Free tier; Team from $25/month plus usage; Enterprise custom	Applications needing user-authorized actions

These gateways overlap, but they solve slightly different problems. Some are full AI control planes. Others focus on tool federation, API management, managed connectors, or user-level authorization.

1. Bifrost (by Maxim AI)

Many MCP gateways focus on aggregating tools.

Bifrost takes a broader approach: MCP traffic is part of the same AI infrastructure as model calls and agent activity.

According to Maxim AI’s official Bifrost reference, Bifrost is an enterprise AI gateway and control plane for LLMs, MCP tools, and agents. The open-source core is licensed under Apache 2.0 and can run inside a company’s own infrastructure.

That broader scope is what puts Bifrost at number one.

One control plane for MCP, models, and agents

Bifrost can connect to upstream MCP servers over:

STDIO
HTTP
SSE

It can also expose connected tools through a single MCP endpoint for clients such as Claude Desktop, Cursor, Claude Code, and custom agents.

The result is one place to:

Register MCP servers
Discover and synchronize tools
Filter tools by client, request, or virtual key
Create curated virtual MCP servers
Apply user, team, customer, or business-unit policies
Authenticate upstream servers
Record every tool call

Bifrost supports server-level and per-user OAuth 2.0, automatic token refresh, shared headers, and token exchange. Credentials remain in the gateway instead of being copied into every agent configuration.

Governance is built into the request path

Bifrost combines tool governance with broader AI governance controls.

Teams can use:

Role-based access control
Data access policies
Virtual keys
Budgets and rate limits
Tool groups and access profiles
Guardrails for requests and responses
Personal-data and secret detection
Signed audit logs
Prometheus metrics and OpenTelemetry tracing

This matters because MCP tools are not just read-only information sources. They can create tickets, modify infrastructure, send messages, move money, or delete data.

The gateway needs to understand which tool is being called, who is calling it, and whether that action is allowed.

MCP Code Mode reduces context pressure

Large MCP installations can expose hundreds of tools. Sending every tool definition to the model on every request increases token usage and makes tool selection less reliable.

Bifrost’s MCP Code Mode lets the model orchestrate tools through sandboxed code and load schemas only when needed.

The official reference reports input-token reductions of up to 92.8% across large tool sets. Maxim AI also reports a published example where a 508-tool workflow reduced token cost from $377 to $29 while maintaining a 100% task pass rate.

This is one of Bifrost’s most interesting advantages for agent-heavy systems: the gateway is not only routing tool calls; it is also reducing the cost of deciding which tools to call.

Performance and deployment

Bifrost is written in Go and is designed for the request path. Its architecture keeps configuration in memory, reuses pooled memory, and maintains provider connections ahead of time.

Maxim AI’s official reference reports approximately 11 microseconds of gateway overhead at 5,000 requests per second on a single t3.xlarge instance. That is a vendor-published benchmark, so teams should validate results against their own MCP server latency and workload patterns.

Deployment is flexible:

Virtual machines
Kubernetes
Private cloud
In-VPC environments
On-premises infrastructure
Fully disconnected or air-gapped networks

Bifrost runs inside the customer’s infrastructure, and the official reference states that prompts, responses, files, and telemetry do not leave that environment.

Teams can also add AI guardrails and route operational data into existing monitoring systems through AI observability integrations.

Pricing

Bifrost’s open-source version is free forever for self-hosted use under Apache 2.0.

The Enterprise tier is custom priced and adds features such as:

High availability
Cluster mode
Identity-based governance
Advanced security
Compliance controls
Enterprise support

Bifrost Enterprise includes a 14-day trial. See current Bifrost pricing before making a purchasing decision.

Best for

Teams running high-traffic, customer-facing, regulated, or multi-tenant AI systems where MCP tools need to follow the same policies as LLM and agent traffic.

If you are evaluating it, start with the Bifrost MCP Gateway documentation and the open-source repository.

2. IBM ContextForge

IBM ContextForge is one of the broadest open-source options in this category.

It acts as an MCP registry, proxy, and federation layer, but also supports A2A, REST, and gRPC services.

That makes it useful for organizations whose internal tool ecosystem is not uniformly MCP-based yet.

What stands out

ContextForge supports:

Federation across multiple MCP servers
REST-to-MCP adaptation
gRPC-to-MCP translation
HTTP, JSON-RPC, WebSocket, SSE, and Streamable HTTP
Virtual servers containing selected tools
JWT, Basic Auth, and custom authentication
Rate limiting
OAuth and user-scoped access
Admin UI and real-time logs
Prometheus metrics
OpenTelemetry tracing
Redis-backed caching and session storage
Multi-cluster Kubernetes deployments
Plugin extensibility

It can wrap legacy REST or gRPC services as MCP-compatible tools, which can reduce the need to rewrite existing internal APIs.

Tradeoffs

ContextForge’s breadth is also its main tradeoff.

This is a platform to operate, not just a lightweight reverse proxy. Production deployments may require PostgreSQL, Redis, Kubernetes configuration, secret management, backups, upgrades, and monitoring.

Teams also remain responsible for securing the gateway and validating downstream tool behavior. The software provides the control surface, but operating a reliable self-hosted control plane still requires platform engineering effort.

Pricing

ContextForge is open source under Apache 2.0. There is no gateway license fee for self-hosting, but teams pay for infrastructure, operations, and any commercial support they choose to add.

Best for

Platform teams that want an open-source, self-hosted gateway capable of federating MCP with REST, gRPC, and A2A services.

3. Kong AI Gateway

Kong AI Gateway brings MCP into an established API-management platform.

It supports three traffic types through a unified control plane:

LLM traffic
MCP traffic
Agent-to-agent traffic

Kong’s MCP layer can map REST API endpoints into MCP-compatible tools, aggregate multiple MCP servers, and expose those tools to clients such as Claude Desktop and Cursor.

What stands out

Kong provides:

AI MCP Server entities
REST-to-MCP mapping
Multiple MCP server aggregation
Key-based and OpenID Connect authentication
OAuth protected-resource metadata
ACLs and rate-limiting policies
MCP audit logs
Latency, response-size, and error metrics
OpenTelemetry integrations
Konnect-based management
Self-hosted deployment options

For companies already operating Kong, this can be a natural extension of an existing API governance model.

MCP tools can be managed alongside other APIs rather than becoming a completely separate infrastructure category.

Tradeoffs

Kong is powerful, but it assumes familiarity with API management, control planes, data planes, policies, and Kong configuration.

It is usually more infrastructure than a small team needs for a handful of MCP servers. The MCP registry functionality is also marked as a technology preview in the current documentation.

Pricing

Kong AI Gateway is commercially positioned through Kong Konnect and enterprise deployments. Pricing depends on the selected control-plane, data-plane, support, and deployment model.

Best for

Enterprises already standardized on Kong or teams that want MCP governance integrated into an existing API platform.

4. Azure API Management AI Gateway

Azure API Management AI Gateway is Microsoft’s managed gateway for AI models and MCP tools. The AI Gateway tier is currently in public preview.

It gives platform teams one place to publish, secure, govern, and observe AI traffic.

A single MCP server can federate backends from:

Remote MCP servers
OpenAPI specifications
Built-in SaaS connectors

That makes it possible to bring existing APIs and MCP services behind one governed endpoint without rewriting every backend.

What stands out

Azure AI Gateway provides:

Runtime access keys for client applications
Backend authentication through API keys, OAuth 2.0, or managed identity
Content-safety policies
IP filters
Request and token rate limits
Azure Monitor and Application Insights integration
OpenTelemetry support
A self-service model and tool catalog
Private networking options

The gateway keeps backend credentials away from client applications. Agents authenticate to the gateway, while the gateway handles authentication to the underlying model or tool service.

Azure API Management can also expose REST API operations as MCP tools, which makes it attractive for enterprises with a large existing API estate.

Tradeoffs

The main limitation is preview maturity.

Microsoft currently lists the AI Gateway tier in East US 2 and Sweden Central, with availability, quotas, telemetry, and APIs subject to change. The preview has no service-level agreement, so teams should use controlled pilots and maintain a rollback plan for critical workloads.

Azure API Management’s MCP capabilities currently focus on tools rather than MCP resources or prompts.

Pricing

Microsoft states that pricing and the business model for the AI Gateway tier will be announced later. Related Azure services, observability resources, networking, and backend providers may still generate costs.

Best for

Azure-native organizations that want managed governance for models, APIs, and MCP tools through Microsoft identity, networking, and monitoring.

5. Arcade MCP Gateways

Arcade MCP Gateways are designed around managed tool access and user-authorized actions.

A gateway can combine tools from multiple MCP servers into one endpoint. Administrators choose exactly which tools are exposed, and different gateways can provide different tool bundles for different projects or workflows.

What stands out

Arcade supports:

Tool federation from multiple MCP servers
Per-gateway tool selection
Server instructions for agents
Remote MCP server registration
OIDC-based user identity
OAuth-based authentication
Project and organization scopes
Dashboard-based gateway creation
AI-assisted gateway configuration
Streamable HTTP connections
Managed tool and connector catalogs

The user identity model is particularly useful for applications where an agent must act on behalf of an end user rather than using one shared service account.

Arcade supports Arcade Auth, an external OIDC User Source, or API-key-style headers for clients that cannot complete a browser-based OAuth flow.

Tradeoffs

Arcade’s remote MCP support currently focuses on Streamable HTTP servers that are reachable from Arcade.

Its documentation also states that remote MCP gateways expose tools, while prompts, resources, and sampling are not supported for remote servers today.

That makes Arcade a strong fit for action-oriented integrations, but it is not the same type of full AI infrastructure control plane as Bifrost or Kong.

Pricing

Arcade publishes:

Free: $0 per month, including 2,000 auth events and 2,000 tool calls per month
Team: $25 per month plus usage
Auth events: $0.10 each
Tool calls: $0.01 each
Enterprise: Custom pricing with VPC, air-gapped deployment, SSO, RBAC, audit logs, and private registry access

See the Arcade pricing page for current limits and terms.

Best for

Product teams that need user-scoped authorization, managed integrations, and a simple way to expose carefully selected tools to agents or IDE clients.

How to Choose the Right MCP Gateway

There is no universal winner for every MCP architecture.

Ask yourself:

Do MCP calls need to follow the same policy as model calls?

If your organization is managing LLMs, agents, and MCP tools as one production system, a unified AI gateway is usually the cleanest model.

Bifrost is especially strong here because its control plane covers LLM, MCP, and agent traffic, with self-hosted and air-gapped deployment options.

Kong is a strong alternative for teams that already manage APIs through its platform.

Do you need open-source federation?

If you need to connect MCP servers with REST, gRPC, or A2A services while keeping the gateway in your own infrastructure, IBM ContextForge is worth serious consideration.

Bifrost is also a strong option when performance, unified AI governance, and a simpler deployment model matter more than broad protocol federation.

Are you primarily trying to roll out approved connectors?

Azure API Management AI Gateway is a strong fit when your connectors and APIs already live in Azure. It can federate remote MCP servers, expose operations from OpenAPI definitions as tools, and connect to built-in SaaS integrations through one governed endpoint.

Arcade is compelling when the main requirement is exposing user-authorized actions through curated gateways.

Which MCP transports do your servers use?

Transport compatibility matters more than many comparison tables suggest.

If you depend on local STDIO servers, verify support before choosing a managed gateway. If your servers are already exposed through Streamable HTTP, options such as Arcade and Kong become easier to evaluate.

Can you enforce tool-level permissions?

Server-level access is not enough.

A gateway should let you expose read-only tools while hiding destructive operations. For example, an agent may be allowed to read a GitHub repository but not merge a pull request, query a database but not modify rows, or inspect cloud resources but not delete them.

Can you explain every tool call later?

For production systems, logs should answer:

Which user or agent initiated the call?
Which MCP server handled it?
Which tool was executed?
What authorization policy applied?
How long did it take?
Did it fail, retry, or time out?
What did it cost?
What model request caused it?

If the gateway only shows server uptime, it is not providing enough operational visibility.

How large will your tool catalog become?

A gateway should help agents discover tools without forcing every schema into every prompt.

For large MCP installations, evaluate lazy discovery, virtual servers, tool groups, schema loading, and approaches such as Bifrost MCP Code Mode.

Final Thoughts

MCP is becoming the standard interface between AI agents and the systems they need to use.

That also means MCP gateways are becoming more than connection managers. They are becoming policy boundaries.

The right gateway determines:

Which tools an agent can see
Which actions it can execute
Which identity is attached to the call
Where credentials are stored
How sessions are routed
How failures are handled
What gets logged
How tool usage is controlled and optimized

If you want open-source federation across MCP, REST, gRPC, and A2A, IBM ContextForge is a strong choice.

If you want MCP governance integrated with an established API-management platform, Kong is compelling.

If you want managed Azure governance for models, APIs, and MCP tools, Azure API Management AI Gateway is worth evaluating, subject to its preview limitations.

If your priority is user-authorized actions through curated tool bundles, Arcade is a good fit.

But for teams building high-traffic production AI systems, Bifrost is the strongest overall option because it combines MCP, LLM, and agent governance in one control plane. Its Apache 2.0 core, self-hosted deployment, air-gapped support, tool-level policies, Code Mode, observability, and performance focus give it a particularly strong foundation for serious agent infrastructure.

Start with the Bifrost MCP Gateway, review the documentation, explore the Bifrost resources, and validate the architecture against your own tools and security requirements.

Thanks for reading! 🙏

What Is Granular Access Control? RBAC, ABAC, and AI

Swapnoneel Saha — Wed, 09 Sep 2026 17:25:41 +0000

"Can this person use the AI API?" is one question.

"Can this service call gpt-5, at most 200 times an hour, with a 50 dollar cap this month, with no file system tools attached, and only until Friday?" is a completely different question.

Granular access control is what turns the first question into the second one. It means defining permissions at the smallest useful unit, so an identity can take exactly one action, on exactly one resource, within exactly one budget, under exactly one set of conditions, and nothing else.

That idea has a name, the principle of least privilege, and it just means giving each caller the minimum access it needs. Granular access control is how you actually write that down.

And the reason it stopped being a nice-to-have this year is that most of the things asking for permission are no longer people.

Access control is a dial, not a switch

Strip a permission down and it is three things: a subject (who is asking), an action (what they want to do), and a resource (what they want to do it to). Every access control system you have ever used is just a way of writing those three things down and checking them.

Coarse-grained access control leaves all three parts wide open. "Engineers can use the database" is one permission covering forty people, six verbs, and every table you own. It is one line of config, and it is fast to write, which is exactly why it survives so long.

Granular access control narrows the same three parts instead. "The reporting service can run SELECT on the orders table, and nothing else." One caller, one verb, one table.

Think of a house key against a hotel keycard. The house key opens everything you own, forever, and the only way to take it back is to change the lock. The keycard opens room 402, on floor 4, until Friday at 11 in the morning, and the front desk can kill it from the lobby without touching a single door.

Both are access control. The keycard is granular.

So granularity is not a product you buy or a model you adopt. It is how far down you are willing to turn the dial, and how much extra work you are willing to do in exchange. Turn it too far and you get a permissions table nobody can read. Leave it too coarse and one leaked credential owns the whole system.

The five dimensions you can narrow a permission on

Most explanations of granular access control stop at "give people less access", which is true and completely useless as advice. The useful version is that a permission can be narrowed along five separate axes, and you can turn each one independently.

Here they are, from the one everybody already does to the one almost nobody does.

1. Who is asking

The identity. This is the axis every system starts with, and the one that changed the most recently.

For a long time a subject meant a person, or a service account somebody created once and forgot about. Now a subject is just as likely to be a coding agent, a CI job, a background worker, or a chain of three agents where the last one has no idea which human started the request.

These are called non-human identities, and they are the ones quietly holding most of the over-permissioned access. Sonrai's cloud access research, published in May 2026, found that 92% of identities with sensitive permissions did not use them once in 90 days, and 87% of that group were machine identities. That is a vendor scanning its own customers and not an independent audit, so take the exact number lightly.

The fix here is easy to say and annoying to do: one identity per caller, never one shared credential per team.

2. What action they want to take

Read, write, update, delete. List versus fetch. Call a model versus look at what models exist.

Coarse systems have two levels here, usually named something like read and admin. Granular systems split up the verbs that can do different amounts of damage. Being able to view a log line and being able to export the whole log table are not the same permission, even though both are "reading".

3. Which resource, exactly

This is where granularity gets real, because a resource is rarely one flat thing.

A database can be scoped at the server, the database, the table, the column, or the row. An AI setup can be scoped at the provider, the model, or a single tool that a model is allowed to invoke. Each step down that ladder means less damage if the key leaks, and one more thing to maintain.

Row-level scoping is the one worth knowing by name. It answers "which of these records can you see", which is a different question from "can you see this table", and it is the difference between a support agent seeing their own tickets and a support agent seeing everybody's.

4. How much they can consume

Here is the axis classic access control mostly ignores, and it is the one that bites hardest with AI.

Permission has traditionally been a boolean. You can call this endpoint, or you cannot. But when a single call can cost real money and a loop can make ten thousand of them, "yes" without a ceiling is not a permission, it is an open tab.

So the granular version attaches quantity to the grant. This key may spend 200 dollars a month. This key may burn 10,000 tokens an hour. This key may make 100 requests a minute. Developers have been asking providers for exactly this for years, and OpenAI's own developer forum carries a feature request for per-key spending limits with people piling into it.

5. Under what conditions

The context around the request. Time of day, source IP, environment, device posture, whether the key has expired yet.

Expiry is the underrated one. A permission with no end date is a permission you will forget you granted, and every "how did this old key still work" incident starts there.

Is granular access control the same as RBAC?

No, and this trips up a lot of people, so it is worth separating properly.

RBAC (role-based access control) groups permissions into roles and hands roles to identities. It is the most common model in the world because it maps onto how companies actually think. You are a developer, developers get the developer role, done.

RBAC can be granular or coarse depending entirely on how you write the roles. That is the part people miss. A role called admin with every permission attached is RBAC and it is not granular at all.

The trouble shows up when you try to push RBAC down the dial. Every new condition needs its own role, so you get developer-staging, developer-staging-eu, developer-staging-eu-readonly, and pretty soon nobody can tell you what any of them do. The industry name for this is role explosion, and it is the standard failure mode of a team that discovered granularity and only had roles to express it with.

ABAC (attribute-based access control) fixes that by evaluating attributes at request time instead of pre-baking roles. Department, clearance, resource classification, time, location. It gets you much finer control without the role count exploding, and the price you pay is that when a request gets denied, you have to trace which attribute failed instead of just reading a role name.

ReBAC (relationship-based access control) asks a third question: how is this subject related to this resource? You can edit the doc because you created it. You can see the profile because you manage that person. Google Drive works this way, and so does every app where ownership is the real rule.

Well, which one should you use? Honestly, most real systems end up with RBAC for the broad strokes and one of the other two layered on for the cases roles cannot express. Granularity comes from the rules you write, and not from the model you picked. You can write a terrible coarse ABAC policy. Plenty of people have.

Why AI made this urgent

I run Claude Code and GPT Codex daily, and both of them do things a normal API consumer never did. They read files and call tools. And they chain several model calls together off one instruction I typed half-awake.

That breaks the two assumptions most permission systems were built on: that whoever is asking is a person, and that every request is something a person actually asked for.

The numbers back this up. 1Password surveyed 1,000 security and engineering staff at large US firms in late May and early June 2026, and found that agents in production reached roughly twice as much data as had actually been approved. In the same survey, 33% of developers running agents reported a breach or security incident tied to an over-privileged non-human identity, and 40% said they leave agents holding persistent access to systems and secrets after the task is finished.

None of that is an AI-specific vulnerability. It is the same over-permissioning problem as always, running at a speed and volume that humans never generated.

Where granular access control breaks in practice

Everybody agrees with least privilege. Almost nobody has it. So it is worth being specific about where the plan falls apart, because the reason is usually the setup, and not laziness.

A provider key cannot be narrowed. Your OpenAI or Anthropic key is a single credential with your whole account behind it. There is no version of it that means "gpt-4o only, 50 dollars, no tool calls". So the moment more than one service needs model access, you either share one key and lose all attribution, or you mint several and lose all central control.

Enforcement lives in the wrong place. If the permission check is inside your application code, then every new service, notebook, cron job, and intern's side script has to re-implement it correctly. One of them will not. And the ones that skip it will not show up in any policy list, because they never registered with the policy system in the first place.

You cannot enforce what you cannot attribute. At Zonko Labs I built an internal tool that captured our AI product's data logs and generated reports on latency and probable slowdowns, and the thing that made it useful was that every log line could be traced back to a specific caller. Without that, a spend spike is just a number going up. You cannot tighten a permission when you do not know which caller needs loosening.

So the pattern behind all three is the same. Granular access control needs a place in the request path that sees every call, knows who made it, and can say no before the call leaves your network.

Where the gateway comes in

Let me be honest about where this actually helps first. If you are one person with one API key on one machine, you do not need any of this, and a gateway will not make your side project safer. It also cannot fix a permission model you have not thought about. A gateway enforces the rules you write, so if you write "allow everything", you get everything, faster.

But the moment there are several services, several models, and a few agents in the mix, the check has to sit in the traffic path. That is what an AI gateway is: a proxy that every model call goes through. Which makes it the one place you can check a permission once, instead of in every service.

Bifrost is the open-source one I keep pointing people at, partly because the whole thing sits in a public GitHub repo, so you can read exactly what it enforces rather than trusting a feature list.

The thing you hand out is a virtual key. Your real provider credentials sit inside the gateway, and each caller gets its own scoped key instead. And the things you can scope that key on are almost exactly the five dimensions above.

Take the rule in plain words first:

This key belongs to the support team. It can only use OpenAI, only the gpt-4o model, and it stops working on the first of next year.

Written out in Bifrost's governance config, that is:

{
  "id": "vk-support-bot",
  "is_active": true,
  "allow_all_providers": false,
  "expires_at": "2027-01-01T00:00:00Z",
  "provider_configs": [
    {
      "provider": "openai",
      "allowed_models": ["gpt-4o"],
      "blacklisted_models": []
    }
  ],
  "team_id": "team-support"
}

Reading that back: allow_all_providers set to false means any provider not listed here is denied, including ones added to the gateway later. allowed_models narrows dimension three to a single model. expires_at is dimension five, so the key dies on its own without anybody remembering to revoke it. And team_id attaches it to a team, which is where the budget lives.

A few more things worth knowing:

Spending and rate caps sit on the key itself. A budget is max_limit plus a reset_duration of 1m, 1h, 1d, 1w, 1M, 1Q or 1Y, and rate limits are separate token_max_limit and request_max_limit counters. Blowing through them returns real status codes rather than a generic failure: 402 for budget_exceeded, 429 for token_limited, 403 for model_blocked. That distinction sounds small until you are the one reading the error at 2 in the morning.

Tool access is deny-by-default. If you have wired up MCP servers, a virtual key with no MCP configuration gets no tools at all. Where tools are granted, they are listed explicitly in tools_to_execute, and the key's list acts as a ceiling that a request cannot widen. That is dimension three pushed all the way down to individual tools, which matters a lot given how much damage one file system tool can do inside an agent loop! (If MCP is new to you, I wrote a beginner's guide to MCP servers first.)

Seeing the dashboard is a separate grant from calling a model. Bifrost ships three system roles, Admin, Developer and Viewer, carrying 42, 27 and 14 permissions, and you can build custom ones by toggling resource and operation pairs. On top of that sits data access control, which decides which rows a user sees at all: own-data, team-data, or all-data. Being allowed to open the logs page and being allowed to see everybody's logs are two different grants.

Write the policy once, not once per person. Access profiles let you define one policy and auto-issue a per-user virtual key from it, each with its own budget counter. Edit the template, propagate, and every key follows. Keys can rotate on a schedule from 1h up to 365d. This is the part that decides whether any of it survives the team getting bigger. If every new engineer needs a hand-written policy, somebody will quietly hand out a shared key by month three.

Audit logs are signed. Administrative events can be HMAC-signed and exported as JSON, JSON Lines, or syslog in the RFC 5424 format that SIEM tools ingest. Knowing who did something and being able to prove it later are the same job, and homegrown setups usually only manage the first half.

Bifrost publishes its own benchmarks (roughly 20 microseconds of added latency at 5,000 requests per second), and those are vendor benchmarks run on a vendor harness, so measure your own. The point holds whichever gateway you use: the check belongs on the path, and not copy-pasted into fifteen codebases.

If governance at this layer is the actual problem you are solving, I also went through the tools in this space in more detail.

Where should you set the dial?

The floor is not RBAC, and it is not ABAC either. The floor is being able to answer one question about your own system right now:

If a credential leaked this minute, what exactly could someone do with it, and how much could they spend before anyone noticed?

If the honest answer is "everything, and I have no idea", the fix is not a bigger access control model. It is a first cut at all five dimensions, in this order: one identity per caller, then a spending cap on each one, then an expiry date, then a model or resource allowlist, then conditions. That order is deliberate, because knowing who called and capping what they spend buy you the most safety for the least work, and conditions buy you the least.

You will not get to per-row, per-tool, per-hour granularity, and you should not try. Nobody is running the theoretically correct permission model. The teams doing well are the ones who pushed the dial two notches past a shared key and then actually maintained it.

Anyway, I am curious where other people drew this line. If you have a permission setup that survived a team getting bigger, or one that collapsed into a shared key by month three, tell me in the comments, I want to hear the failure stories more than the success ones.

You can find me on X, or read the rest of my writing on my site.

Three Insights for Building Great Claude Skills

Swapnoneel Saha — Tue, 08 Sep 2026 18:32:06 +0000

Most Claude skills that never fire are not broken. Their description is.

That one fact is behind almost every "why won't my skill trigger" post you'll find. Three things separate a working skill from one that sits there ignored: a description written like a routing rule, a SKILL.md kept short on purpose, and pushing fragile steps into scripts instead of long instructions. Get those three right, and everything else gets much easier.

First, what a Claude skill actually is

A skill is a folder with a SKILL.md file in it. At the top of that file, you have a small YAML header with a name and a description, followed by instructions written in markdown. That's the whole concept. Anthropic launched skills in October 2025 and made the format an open standard in December, so Claude Code, the API, and claude.ai all use the exact same folder structure.

The interesting part is how skills load. Claude doesn't read your entire skill upfront. When it starts up, it reads only the name and description of every installed skill, costing around 100 tokens each. Claude loads the main instructions only when it decides your skill fits what you asked for. And it reads extra helper files only when those instructions specifically point to them. Anthropic calls this progressive disclosure. Two of the three insights below come down to working with this design instead of fighting it.

This means skills are cheap to keep around, costing you tokens only when they actually run. That single design choice shapes what makes a skill good or bad.

Insight 1: Your description is a routing rule, not a summary

Here is where people get confused. Your description isn't a friendly summary for humans browsing your repository. It is a routing rule Claude uses to pick your skill out of the pile.

Remember what Claude actually sees at startup: only the name and description of each skill. Nothing else. When you send a prompt, Claude decides whether to open your skill's full text based purely on that description. If the description is vague, Claude never reads the instructions. The best instructions in the world won't help if your skill never triggers.

That's why a skill that won't trigger is almost never a code bug. It just means your description doesn't explain when to trigger using the words you actually type.

A good description handles two jobs: what the skill does, and when Claude should run it. Write it in the third person (like "Reviews pull requests" instead of "I review" or "Review this"). Because Claude reads this inside its system prompt, mixing perspectives can confuse the model. You have up to 1,024 characters, so use that space for the exact phrases you would normally type.

---
name: reviewing-pull-requests
description: "Reviews pull request diffs against the team's checklist and flags risky changes. Use when the user asks to review a PR, review a diff, or check staged changes before merging."
---

Notice that the description names the triggers out loud: "review a PR", "review a diff", "check staged changes". Compare that to "Helps with code review", which competes with everything Claude already knows about code review and loses.

The name matters less than the description, but it has strict rules that catch people off guard. It must use only lowercase letters, numbers, and hyphens, capped at 64 characters. It cannot include the words "anthropic" or "claude". Anthropic recommends names ending in -ing (reviewing-pull-requests, analyzing-spreadsheets) to clearly show what action the skill takes. A name like claude-pr-helper breaks two rules at once, while a generic name like helper gives no useful information.

There is real data behind this. One developer ran 650 activation trials on the same skills. They found that passive descriptions triggered far less reliably than direct instructions starting with "Use this skill when." While this isn't an official benchmark from Anthropic, the takeaway matches both the official documentation and real-world use: name the specific situation where Claude should trigger, not just what it can do.

Also, keep your scope realistic. If it's too broad ("helps with frontend"), Claude will pick its own built-in knowledge instead. If it's too narrow ("fixes React 18 hydration errors in Next.js 14 app router"), it won't run if you phrase your prompt slightly differently. You are describing a general task, not a specific bug ticket.

Insight 2: Progressive disclosure is a token budget you spend

Once a skill triggers, its SKILL.md body loads into the same context window as your conversation, your files, and every other skill's metadata. Anthropic makes a great point in their authoring guide: the context window is shared space. Every token you waste in your instruction file is memory the model can't use for your actual work.

This means your main skill file should be smaller than you might expect. The official recommendation is to keep SKILL.md under 500 lines. Move anything longer into separate files that load only when Claude needs them. Loading happens in stages, and each stage has a different cost:

Level	When it loads	Rough cost
Name and description	Always, at startup	~100 tokens per skill
SKILL.md body	When the skill triggers	Under 5k tokens
Bundled files and scripts	Only when referenced	Nothing until read

That third row is the real secret. You can include a 2,000-line API guide, datasets, or code examples, and they cost zero tokens until Claude actually needs them. So the best approach is a lean SKILL.md that acts like a table of contents, pointing Claude to larger files when necessary:

reviewing-pull-requests/
  SKILL.md          # short: the workflow and when to read what
  checklist.md      # loaded only when reviewing
  examples.md       # loaded only when Claude wants a sample
  scripts/
    diff_stats.py   # executed, never loaded into context

The documentation highlights two practical rules here. First, keep linked files only one level deep and link them directly inside SKILL.md. If you bury files inside subfolders, Claude often previews just the first 100 lines and misses the rest. Second, don't explain concepts Claude already understands. Anthropic uses a straightforward example: a good instruction to extract PDF text is about 50 tokens of code. A bad one uses 150 tokens because it starts by explaining what a PDF is. Claude already knows what a PDF is!

Notice what I'm doing here, too. I'm not explaining what a token is or how context windows work. If you're building skills, you already understand those basics. That's the exact same mindset your skills need: only write what the model doesn't already know.

I care about this point a lot because of my work building ANRL. It's a compact data format created to cut token waste and context fragmentation, and it reduced formatting overhead by more than 40%. When you spend weeks fighting to save 40% on tokens, watching someone waste 3,000 tokens in SKILL.md re-explaining JSON is painful. Keeping your skills lean isn't just about clean formatting—it protects the model's working memory.

Insight 3: Let code do the deterministic work

The last one separates skills that feel reliable from skills that mostly work.

If a step is deterministic (parse this file, validate this schema, sort these rows), do not write instructions asking Claude to be careful. Write a script and tell Claude to run it. When Claude runs a script through bash, only the output comes back into context, not the code. So a bundled validate.py is cheaper than asking Claude to generate the same validation every time, and it does the exact same thing on every run instead of a slightly different thing.

Anthropic uses a great phrase for writing these helper scripts: "solve, don't defer." If a file might be missing, handle that check right inside the script. Don't let the script crash and force Claude to guess a fix, because improvised fixes break reliability. The same rule applies to unexplained settings: setting a 30-second timeout with an explanatory comment is much better than leaving a random number like 47 that no one can explain later.

The bigger idea is giving Claude freedom only when the task allows it. Anthropic's docs share a helpful comparison:
Some tasks are like a narrow bridge with steep drops on both sides, like running database migrations in a strict order. For those, give Claude a single script and tell it not to alter the command. Other tasks are like an open field with no dangers, like reviewing code where the best feedback depends on what Claude finds. For those, give general guidelines and trust the model to choose the right path. Making mistakes here hurts in both directions: adding too many rigid rules in an open field makes the skill inflexible, while giving too little direction on a narrow bridge leads to errors.

For bulk edits or risky actions, the documentation suggests a three-step pattern: plan, validate, and execute. Have Claude write its planned changes to a file first. Next, run a script that checks that plan for mistakes. Finally, apply the changes only if the check passes. It might sound like extra work, but it's the difference between catching an error in a safe file and catching it after it breaks your database.

When one skill becomes twenty

Those three insights help you build a solid individual skill. But developers rarely stop at just one. Once your team starts collecting dozens of skills, two quiet problems start showing up.

First, skills can connect to external tools through MCP (if you haven't used MCP yet, check out my beginner's guide to MCP servers). The easiest way to think about the difference: MCP provides the tool, while the skill teaches Claude how to use it for your specific workflow. But pay close attention to Anthropic's security warning: a skill can tell Claude to run tools in ways the author never mentioned. That's why they recommend checking every file inside any skill you didn't write yourself.

Second, it's best practice to test skills across different models like Haiku, Sonnet, and Opus. A skill is only as reliable as the model running it. In a team environment, that means your skills end up calling different models across different API providers.

Combine that with several teammates installing community skills from GitHub, and you aren't just managing simple markdown files anymore. You have unmonitored tool calls and model requests running across your team. That's an entirely different problem.

Do you need a gateway for your skills?

If you're running just two skills on your laptop, you don't. Seriously, skip this section and go write your skills instead of setting up infrastructure. A gateway won't make an individual skill better, and it won't fix a bad description.

A gateway starts to matter when you can no longer answer simple questions: Who is allowed to run which tool? Which skill triggered this request? Whose API budget is paying for it? When you hit that point, routing all that traffic through one place makes life much easier. That place is an AI gateway, and the one I keep returning to is Bifrost (by Maxim AI), an open-source AI gateway and control plane for LLMs, MCP tools, and agents (repo here).

Two of its core features line up directly with the scaling problems above.

To secure tool calls, Bifrost acts as an MCP gateway with deny-by-default tool filtering. Without explicit permissions, zero tools are accessible. You define what to unlock across client settings, request headers, and virtual key configurations. This means a skill could ask for a filesystem MCP server, but its key can restrict execution strictly to read_file, completely blocking destructive actions like delete_file. You don't have to depend on plain text instructions in a prompt to keep your systems safe. (It even has an MCP Code Mode that runs tool workflows inside a sandbox instead of dumping every tool schema into your context, cutting token usage by over 92% across large tool sets.)

To manage model access, Bifrost provides virtual keys and governance rules. Each key gets its own list of approved models (across 25+ providers and 10,000+ models), an automatic budget cap with reset periods, and custom rate limits. So when you want your team to test a skill across Haiku, Sonnet, and Opus, you give them one key with an allow-list and a spending cap instead of juggling five API keys in a spreadsheet. Provider fallback and retries are built in, so a skill doesn't fall over if one provider has an outage.

Performance is where this actually holds up. If a gateway is slow, developers route around it. Bifrost is written in Go with memory pooling and pre-warmed connections, keeping added gateway overhead to about 11 microseconds at 5,000 requests per second on a single t3.xlarge instance. In Maxim's published benchmarks against LiteLLM (tested with 500 concurrent users on AWS), Bifrost delivered 9.5x higher throughput (424 req/s vs 44.84 req/s) and roughly 50x lower P50 latency (804 ms vs 38.65 s). Those are vendor benchmarks, so treat them as such, but the main point holds: the gateway manages your traffic without becoming the slow part.

None of this is about writing an individual skill. It's just what helps once your skills work well and you start scaling up.

So what actually makes a Claude skill great?

It's not about making a giant SKILL.md file. It's much simpler than that.

Write your description like a clear routing rule in the third person, using the exact words you'd naturally type in a prompt. Keep the main instructions short and let extra reference files handle the heavy lifting, because context memory isn't free. And move fragile, repetitive tasks into code scripts so they run consistently every single time instead of guessing. Those three habits make the difference between a skill that runs reliably and one that sits forgotten in a folder.

Using an AI gateway matters as well, but only later on. Don't worry about it until you have more skills and tools than you can easily keep track of in your head. You'll know when you cross that line. Until then, setting up a gateway is just extra work you don't need.

If you've built skills that trigger reliably, I'd love to hear how you wrote your descriptions to make them work. That's still the area where most people spend the least effort. Come share your thoughts on X, or check out my other posts at my site.

How to Cut Time to First Token (TTFT) in LLM Apps

Swapnoneel Saha — Fri, 04 Sep 2026 15:49:58 +0000

Whenever you type something into an AI app like ChatGPT or Gemini, and hit enter, you have to stare at nothing for half a second, one second, and sometimes even more. Then the words start pouring out fast. The wait time that you always experience in between these two events is called time to first token, or TTFT, and it is the single thing your users actually feel.

Time to first token (TTFT) is the gap between sending a request and the first token coming back. To cut it, shorten and cache your prompt so the model reads less, stream the reply, keep connections warm, and route around slow providers. The biggest lever is prompt length, since most of that wait is the model reading your input.

Now let me show you where the time actually goes, and how we can fix this. So, let's begin!

What Time to First Token Actually Is

When you send a prompt, the model does two very different jobs.

First it reads everything you sent. Every token of your system prompt, your chat history, your retrieved documents, your question. It runs all of that through the network in one big forward pass and builds an internal memory of it. This step is called prefill, and nothing comes back to you while it runs.

Then it starts writing, one token at a time. That second phase is where you see text stream in.

TTFT covers the first phase plus everything around it: the network trip to the provider, any time your request spends waiting in a queue, and the prefill itself. Here is the part that matters. On any prompt that is not tiny, prefill dominates, and the more tokens you send, the more prefill work the model does before it can start. A short prompt starts answering quickly. A prompt stuffed with 8,000 tokens of context makes the reader sit and watch the cursor.

What Is a Good Time to First Token?

It depends on what the app is for.

The Modular LLM Inference Handbook puts it simply: a chatbot usually needs TTFT under 500 milliseconds to feel responsive, and a code-completion tool needs it below 100 milliseconds or it feels broken. A batch job that emails a summary once a day can take five seconds and nobody cares.

So before you optimize anything, decide which of these you are. A chat UI and a background report have completely different budgets, and chasing 100ms on a nightly job is wasted effort.

The Fixes, Biggest Wins First

Here is the order I actually work through. The early steps cost you almost nothing and pay back the most. The later ones matter, but only after the cheap wins are done. Do not skip to the bottom.

1. Measure TTFT on its own, before you touch anything

You cannot cut a number you are not looking at. And most teams are not looking at this one, because their logging records total response time and stops there. Total time hides TTFT completely, since a slow first token and a slow-typing model both show up as one big number.

So split them. Log the time from request-sent to first-token-received as its own field, separate from the time to finish. Do it at the p50 and the p99, because the average will lie to you and the tail is where users rage-quit.

This is not theory for me. At Zonko Labs I built the internal tool that captured the data logs for Luffy, an AI co-worker that lived in Slack, and turned them into reports on latency and probable slowdowns. The first thing that tool taught everyone was that our "the bot is slow" complaints were almost always a first-token problem, not a typing-speed problem. You only learn that if you measure the two apart.

If your traffic runs through an AI gateway, you can track this without building custom timers in your app. For example, Bifrost exports a metric called bifrost_stream_first_token_latency_seconds. It measures TTFT automatically across OpenAI, Anthropic, or any other provider, so you get clean p50 and p99 charts right out of the box.

2. Shorten the prompt

This is the biggest lever you fully control, and it is the one no tool can pull for you.

Prefill scales with input length, so every token you delete is prefill work the model never has to do. Look hard at what you are actually sending. Most apps ship a system prompt that grew over months and never got trimmed, a full chat history when the last four turns would do, and RAG retrieval that dumps ten documents when three answer the question.

Cut the system prompt to what the model needs, not what makes you feel safe. Truncate or summarize old history instead of resending all of it. Retrieve fewer, better chunks. None of this is glamorous, and all of it moves TTFT more than the fancy stuff.

Be honest about the tradeoff, though. Cutting context can cost you answer quality, so trim, then check your evals, then trim again. This is the one place where going too far actually hurts, so it earns the care.

3. Cache the stable part of your prompt

Most of what you send is the same every single call. The system prompt does not change. The tool definitions do not change. Only the user's question at the end changes. Prefill does not know that, so it re-reads the whole thing every time, unless you turn on caching.

There are two flavors, and they are not the same thing.

Prompt caching, also called prefix caching, is offered by the model providers themselves. It stores the processed form of a stable prefix so the next call reuses that work instead of redoing it, which is what prefix caching buys you. Put the parts that never change at the very front of your prompt and the changing question at the end, so the cache has the longest possible prefix to hit.

Semantic caching is the bigger swing. Instead of caching the prompt's prefill, it caches the whole answer, and serves it when a new question means the same thing as an old one. On a hit, you skip the model entirely, so TTFT collapses from "prefill a big prompt" to "look up a vector," which is milliseconds. I wrote a full explainer on semantic caching if you want the mechanics.

Let me make the caching point concrete with a small test you can predict.

Call A sends a 3,000-token system prompt that never changes, then a 40-token question at the end. Call B folds the same information into a fresh 3,000-token prompt that gets reworded on every request, then the same 40-token question. Both send roughly the same number of tokens. On the hundredth request, which one has the lower TTFT?

Call A wins, and it is not close. Call A's 3,000-token prefix is identical each time, so after the first call it lives in the prompt cache and its prefill is skipped. Call B looks different to the cache every time, so it pays the full prefill on all one hundred calls. Same token count, wildly different first-token time, decided entirely by whether the front of your prompt holds still. That is the whole game with caching: keep the front stable.

4. Stream the response, and know its one limit

Streaming sends each token to the user the instant it is generated, instead of waiting for the full answer. Turn it on. Every serious chat UI streams, and it is a one-line change in most SDKs.

Now the honest part, because it trips people up. Does streaming reduce TTFT? No. Streaming does not make the first token arrive any sooner, since prefill still has to finish first. What it changes is everything after that first token, and it changes how fast the whole thing feels. People read a streaming reply as faster than a non-streaming one even when the total time is identical, which is why faster first tokens matter more than total speed.

So streaming is a perception win, not a TTFT win, and the two stack. Streaming only pays off if TTFT is already short, because a three-second blank screen followed by a burst still reads as three seconds of nothing. Fix TTFT first, then stream to make the rest feel quick.

5. Keep the connection warm and close

Some of your TTFT is pure plumbing, and it hides where nobody looks.

Every fresh HTTPS connection pays for a TLS handshake before a single byte of your prompt moves. If your app opens a new connection per request, you are paying that tax every time. Reuse connections with keep-alive and connection pooling so the handshake happens once, not on every call.

Distance costs you too. A request from a server in Mumbai to a model endpoint in Virginia spends real milliseconds just crossing the planet and back, twice, before prefill even starts. Pick a provider region near your users where you can, and cut the number of hops between your app and the model. If you are on a serverless setup, watch for cold starts, since a function spinning up from zero can add more delay than the model does.

6. Route around slow and rate-limited providers

Here is the failure mode that ruins TTFT and has nothing to do with your prompt. The provider is having a bad minute.

When a provider is overloaded, your request sits in its queue, and queue time is part of TTFT. When you hit a rate limit, you get a 429 and your client backs off and retries, which stacks seconds onto the first token before it ever appears. Neither of these is fixed by shortening your prompt, because the model never even started.

The fix is to not be stuck with one provider. Send the request to the fastest healthy option, and when one starts failing or crawling, fall over to another automatically instead of retrying into the same wall. This is adaptive load balancing applied to model traffic, and it is the difference between one bad provider-minute taking down your p99 and your users never noticing.

Where a Gateway Does This For You

Let me be honest about what a gateway cannot do first.

A gateway cannot shorten your prompt. It cannot make a provider's GPU run faster. If your app only calls one model and your prompt is already lean, you do not need a gateway for TTFT. You will get much bigger gains just by trimming your system prompt.

Where an AI gateway helps is when you want to handle caching, retries, and monitoring across multiple services without rebuilding them in every backend repo. The gateway sits between your application and your model providers as a reverse proxy, so you wire those fixes once in your infrastructure instead of in your app code.

I have used Bifrost, an open-source AI gateway built in Go, and it handles these levers in one place:

First, it gives you TTFT metrics out of the box. Measuring first-token latency manually usually means writing custom streaming wrappers in Python or Node.js to timestamp the first chunk. Bifrost measures bifrost_stream_first_token_latency_seconds at the proxy layer for every request, giving you clean p50 and p99 charts across all your providers without extra client telemetry.

Second, it provides semantic caching. If a user asks a question that matches a previous one, Bifrost serves the answer straight from cache in about 10 to 15 milliseconds. That turns an 800-millisecond prefill wait into almost nothing. Plus, cache writes happen asynchronously in the background, so cache misses do not add any latency to your stream.

Third, it runs automatic fallbacks. When a provider is overloaded or returns a 429 rate limit, your request sits in a queue and your TTFT spikes. With Bifrost, you can set fallback rules: if your primary model hangs or errors, it instantly shifts the request to a healthy backup provider instead of leaving your user staring at an empty screen.

Finally, there is the question of proxy overhead. Adding a gateway puts an extra hop in your network path. If that hop adds 30 or 50 milliseconds, it defeats the entire purpose of optimizing latency. Bifrost is written in Go and adds around 20 microseconds of overhead under load, meaning the hop itself is practically invisible.

You still have to fix your prompts first. But once your prompts are clean, putting a gateway in front of your models lets you handle caching, fallbacks, and latency tracking through simple configuration instead of messy glue code.

Start Here

If you do nothing else from this whole post, do three things, in this order.

Measure TTFT as its own metric, so you stop flying blind. Shorten and cache your prompt, because that is where the biggest, cheapest wins live. Then stream, so the rest of the answer feels fast.

The truth most infrastructure posts bury is that for an app that calls an API, time to first token is mostly about prompt length and caching, not about exotic GPU tricks you will never touch. The GPU tricks are real, but they belong to the people running the models. Your levers are the prompt you send, the cache in front of it, and the route it takes. Those you own completely.

Start by logging your p50 and p99 first-token time this week. I would bet the number surprises you, and I would bet a fat system prompt is the reason.

If you have a TTFT horror story, or a fix that worked that I left out, tell me. I am @swapnoneel123 on X, and I write more of these on my site.

What Is an MCP Registry? Discovering & Governing MCP Servers

Swapnoneel Saha — Thu, 03 Sep 2026 17:01:44 +0000

Most people consider an MCP registry as "npm for MCP servers." It is a nice line to explain the concept, but it is partially true and extremely half-baked.

An MCP registry is a searchable catalog of MCP servers. It stores metadata, like a server's name, where to install it, and how a client should connect, so an AI agent or a developer can find servers to plug in. But here is the part the npm comparison hides: a registry is a discovery layer, not a trust layer. It tells you a server exists. It does not tell you the server is safe, or that your agents should be allowed to run it.

That gap between "I found it" and "I trust it" is where teams get burned. Discovering MCP servers and governing MCP servers are two different jobs, and a registry only does the first one. So let's get both straight.

If you want to know more about what an MCP server is and how it works, I wrote a beginner's guide to MCP servers first. This post picks up where that one left off.

What an MCP Registry Actually Is

An MCP registry is a metadata catalog. It does not hold the server's code. It holds a small record that describes the server and points to wherever the code actually lives.

This trips people up, so let me separate the two clearly. Your code lives on a package registry: npm, PyPI, Docker Hub. That is where the actual bits get downloaded from. The MCP registry sits one level up and just stores a pointer. For example, a weather-mcp package can live on npm, and the MCP registry entry maps "weather v1.2.0" to npm:weather-mcp. The registry is the index card. npm is the shelf.

Each entry follows a standard shape called server.json. It records the server's unique name, where to find it (an npm package name, or a remote server URL), how to run it (command-line args, environment variables), and some discovery data like a description and what the server can do. That is it. A registry is a stack of these server.json cards, plus an API to search them.

And that is genuinely useful. Before any of this, finding an MCP server meant scrolling GitHub READMEs and random Discord links. A registry turns that into one queryable list. But notice what a stack of index cards can and cannot do for you, because that is where the trouble starts.

The Official MCP Registry

The one most people mean by "the MCP registry" is the official MCP Registry, which launched in preview on 8 September 2025. It is open source, and it is backed by a serious group: Anthropic, GitHub, PulseMCP, Microsoft, and others.

Its job is deliberately narrow. In the maintainers' own framing, it is a centralized metadata repository, and the metadata it stores is intentionally unopinionated. It does not rank servers, and it does not review or rate them. It just holds the server.json records and serves them over a REST API.

Here is the design choice that matters most, and almost nobody mentions it. The official registry is not really built for your app to read directly. It expects to be consumed by "downstream aggregators", which pull its data on a schedule (say, once an hour) and then layer their own curation, ratings, or search on top. The registry is the wholesale source of truth. The nice front-ends you actually browse are meant to be built by other people.

And in practice, as of 2026, most MCP clients still do not read the registry directly as a built-in source. So for now you usually meet it through one of those aggregators, not through your agent going and querying it live.

Two more things worth knowing early. First, the official registry does not host private servers, so an internal mcp.acme-corp.internal server has no place there. Second, it is in preview, which means data can reset and the shape can still change. So treat it as a fast-moving foundation, not a finished product.

MCP Registry vs Marketplace vs Gateway

This is the question I see confused most, so let me draw the lines hard. These are three different things doing three different jobs.

A registry is discovery in its rawest form. It answers one question: what servers exist, and how do I connect to them? It is a passive list.

A marketplace is discovery made pleasant. Sites like Smithery, Glama, and mcp.so give you search, categories, install buttons, and community ratings. Increasingly they read from the official registry underneath and add their own polish on top. A marketplace is a registry with a nice storefront and an opinion.

A gateway is something else entirely. It is not about finding servers, it is about controlling them at runtime. A gateway sits in the traffic path between your agents and your MCP servers, and it enforces the rules every time a call is made: who is allowed to use this tool, how often, at what cost, with what data. A registry is a directory. A gateway is a checkpoint.

The one line to remember: a registry without a gateway gives you discovery but no runtime governance. Your agents can find tools, but nothing stops an unauthorized call once they connect. Hold onto that, because it is the seam the rest of this post walks through.

How Discovery Actually Works

So how does a client go from "I need a GitHub server" to a running connection? Through the registry API and the server.json record, and the interesting part is how it proves a server is who it claims to be.

Server names use a reverse-DNS format: io.github.username/server-name, or com.example/server. That name is not decorative. It ties the server to a namespace that the publisher had to prove they own, through GitHub or DNS verification. If you publish under io.github.swpn0neel/*, the registry made you show you actually control that GitHub account first. So a name is a small ownership claim, verified at publish time.

Here is the piece that makes the whole thing extensible. The registry ships an open OpenAPI spec, which means anyone can stand up their own registry that speaks the exact same interface. A public marketplace can implement it. A company can implement a private one for internal servers. Any MCP client that already knows how to talk to the official registry can talk to those too, with no extra work. One shape, many registries.

That design is genuinely clever, and it is also exactly why governance cannot live at this layer. A standard that lets anyone publish, and lets anyone run a compatible registry, is a standard optimized for reach, not for safety. Openness and vetting pull in opposite directions, and the registry picked openness on purpose.

Discovery Is the Easy Half

So that is discovery, more or less solved. You have a standard catalog, verified namespaces, an API, and a way to run your own copy.

But discovery was always the easy half of the title. The hard half is governing what those servers actually do once your agents start calling them, and that is a problem a catalog structurally cannot touch. Let me show you why with one concrete example.

What a Registry Does Not Protect You From

Say you tell your coding agent, "add the Slack MCP server." It searches, finds a package named slack-mcp-server, sees an install command that looks completely official, and wires it in. Feels routine. Now walk that one action through each layer and watch where it does, and does not, get stopped.

Did the registry catch it? No, and this is by design. The official registry delegates security scanning to the underlying package registries and to downstream aggregators. Its own job is namespace authentication and metadata hosting. So namespace verification proves the publisher owns the name they used. It does not prove the code behind that name is safe. A registry entry is a verified business card, not a background check.

So typosquatting still works. This is not hypothetical. Security researchers have already documented look-alike PyPI packages like slack-mcp-server and slack-mcp-server-v2, whose install commands look official but are not. A name that reads as legitimate is not the same as a source you vetted, and a catalog optimized for discovery will happily list both.

And the tool itself can lie. The nastier class is tool poisoning: a malicious server hides instructions inside a tool's description, so when the model reads "what can you do?", it also reads "and quietly send recent messages to this URL." The model sees a normal tool. The registry saw a normal server.json. Neither one inspected behavior, because inspecting behavior is not what either one does.

Then there is the rug pull. A server can be perfectly clean the day it is approved and turn malicious three versions later. A registry records the new version. It does not re-judge it. This exact risk is why security researchers keep arguing that tool updates should trigger re-review instead of being trusted automatically.

None of this means the registry is broken. It means the registry is doing its one job, discovery, and quietly handing you a job it was never built to do: deciding what to trust, and enforcing that decision on every call. That job has to live somewhere else.

Do You Need a Private MCP Registry?

For a lot of teams, the first real answer to the trust problem is a private registry. And it genuinely helps, so let me be fair to it before I tell you where it stops.

A private, internal registry is an allowlist. Instead of letting agents pull from the open catalog, your platform team curates a list of vetted servers, mixes in your own internal ones, and points every agent at that. Because it implements the same OpenAPI shape, your existing clients keep working. This is the model enterprise registry products (Kong's, Portkey's, and others) are built around, and for regulated or multi-team setups it is close to mandatory, because a community catalog was never meant to carry your compliance requirements.

So a private registry moves the trust decision to a place you control. That is a real upgrade over pulling from the open internet.

But look closely at when it acts. A private registry governs discovery time: what a developer or agent is allowed to install and see. It does not govern runtime: what an approved server actually does on each individual call. Once an agent is connected to a vetted server, the registry is out of the loop. It cannot cap how much that agent spends, cannot say "this key may read files but never delete them," and cannot stop a poisoned tool description that slipped through your review. It curated the shelf. It is not standing at the counter.

You still need something in the traffic path.

Where the Gateway Comes In

Here is the honest part first, because it decides whether you should even read this section. If you are running one or two MCP servers you trust, you do not need a gateway. A registry, or even a hardcoded config, is plenty. A gateway earns its place when you have many servers, many agents, and real consequences if one of them misbehaves. And a gateway cannot vet server code for you either, so nothing here removes the need to review what you run.

For a few months I was the first layer of testing for an AI co-worker that lived inside Slack, catching its mistakes before the real users did. The lesson that stuck with me is simple: you cannot govern what you only hear about after the fact. Governing has to happen at runtime, in the path, while the call is being made.

What a gateway adds is the one thing a registry structurally cannot: enforcement on every call, at runtime. That is where Bifrost, an open-source AI gateway built by Maxim AI, fits the picture. It can act as an MCP gateway: you connect all your MCP servers to Bifrost, and your agents connect to Bifrost instead of to each server directly. One endpoint in front of everything, sitting exactly where the checks need to happen.

Once you are in that position, three things change.

First, access stops being all-or-nothing. Bifrost filters tools per virtual key, so a given key only sees the tools you explicitly listed for it. You configure it plainly: a client named filesystem, with tools_to_execute set to ["read_file", "list_directory"], and that key can do exactly those two things and nothing else. It is deny-by-default. The read-only agent literally cannot see a delete tool, so a poisoned or over-eager call has nothing to grab.

Second, the token problem gets solved instead of tolerated. Remember how a registry lets you connect dozens of servers? Once you do, every request stuffs every tool's description into the model's context, and with 150-plus tools the model burns most of its budget just reading the menu. Bifrost's Code Mode fixes this by exposing four generic meta-tools and letting the model write code to discover and run what it needs on demand, instead of pre-loading every definition. In their own benchmark of 508 tools across 16 servers, that cut input tokens by up to 92.8%, from 75.1M tokens down to 5.4M, while keeping a perfect pass rate. That is a vendor benchmark on a vendor harness, so weigh it as one, but the mechanism is sound and the direction is not subtle.

Third, control lives in one place. All your keys and connections sit behind the gateway instead of scattered across config files, and you can attach budgets and rate limits to each virtual key, so an agent that goes haywire hits a spending cap instead of your invoice. This is the runtime governance the registry handed off. The registry told your agents what exists. The gateway decides what is allowed, every single time.

So the two are not competitors. A registry and a gateway are the two halves the title promised, and they sit at different points in the flow: the registry at discovery, the gateway at runtime.

What Do You Actually Need?

Let me put it as plainly as I can, because the whole post comes down to one distinction.

A registry answers "what exists?" A gateway answers "what is allowed?" You will eventually want both, and the mistake I see most is assuming that a shiny catalog, public or private, has solved the second question. It has not. It was never trying to.

So here is the path I would actually take. If you are just exploring, use the official MCP registry or a marketplace to find good servers, and keep the count small. If you are a team standing this up for real, add a private registry so you control what gets discovered in the first place. And the moment agents are making real calls with real credentials and real budgets, put a gateway in the traffic path, because that is the only layer that can enforce a decision at the moment the decision matters.

Discovery is a catalog problem. Governance is a runtime problem. Solve them in the right places and MCP stops feeling scary. Try to solve governance with a directory, and you will find out the hard way that an index card never stopped anybody from doing anything.

I run coding agents like Claude Code every day, and my own MCP config has quietly grown from one server to a small pile without me really deciding to. That drift is exactly how this bites you, one harmless-looking addition at a time. So if you have hit the same sprawl, I would genuinely like to hear how you are handling it. Drop a comment, or come find me on X and let's compare setups.

What Is an MCP Server? A Beginner's Guide

Swapnoneel Saha — Wed, 02 Sep 2026 18:48:39 +0000

An MCP server is just a small program that gives an AI model access to tools and data it cannot reach on its own.

In plain words, it is a bridge. Through one shared standard called the Model Context Protocol, it lets an AI read your files, query a database, or call an API without you having to write custom glue code every single time. It can be a tiny script running on your laptop, or a service running in the cloud.

None of this is as complicated as the acronym makes it sound. And it is worth getting right, because MCP is quickly becoming the default way AI agents interact with the real world. So, let's dive in!

What Is an MCP Server

An MCP server is a program that exposes three things to an AI model: tools it can run, data it can read, and prompt templates it can reuse. Simply put, the model connects to the server, asks "what have you got?", and then uses whatever comes back. Strip away the jargon and an MCP server is a standard adapter between one AI and one outside system.

The "server" word trips people up, so let me kill that confusion first. It does not mean a rack in a data center. A filesystem MCP server can be a tiny script on your own laptop. It is called a server only because it serves requests, the same way your local dev server does when you run npm run dev. Small program, answers questions, done.

And that is really the core of what people mean by MCP server meaning: it is the thing on the other end of the connection that knows how to do actual work, like reading a file or querying a database, and knows how to describe that work in a format the model understands.

Where MCP Came From and Why It Caught On

The Model Context Protocol is an open standard for connecting AI models to outside tools and data. Anthropic released it in November 2024, and within a year it went from a niche idea to something Claude, ChatGPT, Cursor, and VS Code all speak.

Here is the problem it fixed. Before MCP, if you wanted an AI to use some tool, you wrote the glue yourself, every single time. A couple of years back at Keploy I built a RAG chatbot over their docs, wiring the vector embeddings and the retrieval together by hand. That was one data source and one integration, fully custom. Now picture doing that again for GitHub, then Slack, then your database, then for a different model that expects a different shape. It does not scale, and everyone was rebuilding the same connectors.

The official docs describe MCP as a USB-C port for AI. One connector shape, and anything can plug into anything. You build an MCP server once, and every client that speaks the protocol can use it. That is the entire reason it caught on so fast: it turned a pile of one-off integrations into one standard everyone could share.

How Does an MCP Server Actually Work?

An MCP server works through three roles: a host, a client, and a server. The host is the app you are actually using, like Claude Desktop or Cursor. For each server it wants to talk to, the host spins up one client, and that client keeps a dedicated line open to one MCP server. Messages go back and forth as JSON-RPC 2.0, which is a plain, boring, well-understood format, and that is a good thing.

The server can offer three kinds of thing, and it is worth knowing them by name:

Tools: actions the model can run, like "read this file", "run this query", or "create this GitHub issue".
Resources: data the model can read for context, like a file's contents or a database schema. Read-only, no side effects.
Prompts: ready-made templates for common tasks, so the model does not start from scratch every time.

The flow itself is simple. The client connects, then asks the server to list what it has (a tools/list call), then runs one of them (a tools/call). In JSON it looks about like this:

// "What can you do?"
{ "jsonrpc": "2.0", "id": 1, "method": "tools/list" }

// "Okay, do this one."
{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "get_weather",
    "arguments": { "city": "Kolkata" }
  }
}

One more thing worth knowing early: a server runs in one of two places. A local server talks over stdio (standard input and output), which is just two programs on the same machine passing text back and forth, no network involved. A remote server talks over HTTP, so it can live anywhere and serve lots of clients at once. Same protocol either way, which is the point.

MCP Server vs API: What's the Difference

This is the question I get most, so let me be direct. An MCP server usually sits on top of a normal API. It is not a replacement for one. The difference is who the thing is built for.

A regular API is built for a developer who already read the docs and hardcoded the exact calls. It assumes someone knows the endpoints ahead of time. An MCP server is built for a model that shows up knowing nothing, asks what is available, and figures out which tool to use on its own. That runtime discovery is the whole trick, and a plain API does not do it.

So when people ask about MCP vs API, the honest answer is that most MCP servers are a thin wrapper around an API you could have called yourself. What MCP adds is a standard way to describe the tools, so any model can find them and use them without a human wiring each call. The API does the work. The MCP server makes that work legible to an AI.

What Can You Actually Do With One

Plenty, and you probably already have the pieces. The most common MCP servers connect a model to everyday systems: a filesystem server for reading and writing files, a GitHub server for issues and pull requests, a Postgres server for querying a database, a Slack server for messages. Anthropic keeps a set of reference servers on GitHub, and most companies now ship an official one for their own product.

I run Claude Code every day, and that is where most people meet their first MCP server. You bolt one onto a coding agent or a desktop chat app, and suddenly it can touch your files, search the web, or open a pull request without you copy-pasting anything. The MCP tools it offers just show up as things the model can now do.

The pattern is always the same. Find a server for the system you care about, point your client at it, and the model gains a new skill. One server is genuinely a five-minute setup, and it feels a little bit like magic the first time an agent edits a real file on your disk!

Do You Need an MCP Gateway

Here is the honest part first: if you are running one or two MCP servers, you do not need a gateway. It solves a problem you do not have yet, so skip this section and go play with a server instead.

But things change once the count goes up. Say you have got ten servers wired into your agents. Three problems show up fast. First, every request dumps the full tool list of every server into the model's context, so with 150-plus tools the model burns most of its budget just reading the menu. Second, each server has its own keys and auth, scattered across config files with no single place to rotate them. And third, there is no shared way to say who is allowed to call what, or to cap spending. That is tool sprawl, and it turns a neat setup into a liability.

An MCP gateway fixes this by sitting right in the middle. Think of it like a power strip. Instead of plugging ten different cables into ten wall sockets, your AI agent connects to the gateway once. The gateway handles the rest.

This is where Bifrost, an open-source AI gateway built by Maxim AI, comes in. It can act as an MCP gateway, tying all your servers together so your AI talks to one neat endpoint instead of juggling ten.

It also fixes that token problem in a really clever way. Remember how 100 tools eat up your context? Normally, the model has to re-read every single tool's description on every prompt just to know what's on the menu. That burns tokens fast.

Bifrost fixes this using Code Mode. Instead of dumping every tool definition into the prompt, it gives the model four generic tools and lets it write code to find and run what it needs on demand. In their benchmarks, that cut input tokens by up to 92.8%. That is not a small difference, that is basically your whole bill! Very similar to the cut-costs-at-the-gateway idea I talked about with semantic caching.

On top of that, you get one central place for safety and control. All your API keys live in one spot instead of being scattered across random config files. You can set per-key tool filtering and budgets, meaning you decide exactly which agent can use which tools and how much it can spend. And by default, tools do not run automatically unless you explicitly turn that on. That last part matters a lot, because an agent running tools silently without your permission is a bad day waiting to happen.

Where to Start

You do not need to read a massive spec or write complex code to get started with MCP. The whole idea is small once the acronym stops scaring you: it is just a standard plug that lets your AI reach outside tools.

If you want to try it out today, do not overthink it. Pick an app you already use, like Claude Desktop, Cursor, or Claude Code. Then grab just one pre-built server, like the filesystem or GitHub server. Wiring it into your config file takes about five minutes.

Once it is running, give the model a real prompt: ask it to inspect a local folder or check a repo. The moment you see an agent reach outside its chat box and touch a real file on your machine, the whole protocol clicks.

And my advice would be to keep it tiny. You do not need a gateway or fancy routing when you are only running one or two servers. Play with one first, see how it feels, and only put a gateway like Bifrost in front when you have a dozen servers and your token bill starts climbing.

If you set up your first MCP server after reading this, I would genuinely like to hear how it went. Drop it in the comments, or come say hi on X and we can chat about it.

Frequently Asked Questions (FAQs)

What is an MCP server in simple terms? It is a small program that gives an AI model a set of tools and data through a shared standard, so the model can read a file, query a database, or call an API without custom glue code. Think of it as a standard adapter between one AI and one outside system.

What is the difference between an MCP client and an MCP server? They are the two ends of one connection. The server offers tools, data, and prompts. The client, created by an app like Claude Desktop, connects to the server and uses what it offers. One app can run many clients, one per server it talks to.

Are MCP servers safe to use? They are as safe as how you run them. A single local server you trust is low risk. The danger shows up when you run many unvetted servers with their own credentials and let an agent run tools automatically, which is exactly why keeping auto-execution off and routing servers through a gateway with tool filtering is worth doing once you scale.

How do you build your own MCP server? You use one of the official SDKs, define a few tools with their inputs, and expose them over stdio or HTTP. Start tiny, with one or two tools, get it working with a client you already use, and grow from there instead of trying to wrap an entire API on day one.

Fixing Claude Rate Limit Errors with an AI Gateway

Swapnoneel Saha — Wed, 26 Aug 2026 18:04:49 +0000

Ever had a 429 from Claude that just would not clear, no matter how patiently you backed off?

Yeah. That one was never a rate limit.

A Claude rate limit error is an HTTP 429 carrying "type": "rate_limit_error", and it means your organization crossed a per-minute request ceiling, a per-minute token ceiling, or its monthly spend cap. Which one decides the fix, and the response body tells you which before you write any retry code.

And the difference matters far more than it sounds, because two of those three conditions get strictly worse the moment you retry them.

The five failures behind one Claude rate limit error

rate_limit_error is not one condition. Anthropic returns the same string for several different situations, and the correct response may be to wait a few seconds or to change a setting that retrying cannot fix.

The Claude API errors reference documents them separately, so you can tell them apart from the response alone.

What you see	What it actually is	Does retrying help?
`429` + `rate_limit_error` + a `retry-after` header	A real per-minute limit. You went too fast.	Yes, after `retry-after` seconds
`429` + `rate_limit_error` + `details.error_code: enforced_spend_limit_reached`, and no `retry-after`	Your usage tier's monthly spend cap	No. Access returns at 00:00 UTC on the 1st
`400` + `invalid_request_error`, message starting `You have reached your specified API usage limits`	A spend limit you set yourself in the Console	No. Raise or remove your own limit
`429` right after a sharp traffic ramp	An acceleration limit, not your steady-state limit	Partly. Ramp up gradually instead
`529` + `overloaded_error`	Anthropic is busy. Nothing to do with you.	Yes, but the queue is global

That last row is the one that trips people up most. The Claude overloaded error gets searched more than any specific 429 phrasing, and almost every guide lumps it in with rate limits. It isn't yours. A 529 means the API is temporarily overloaded across all users, so no amount of tier upgrading or key rotation on your side moves it.

The second row is easy to misdiagnose. It looks identical to a normal rate limit at the status-code level, the official SDKs will happily auto-retry it, and every retry fails. The tell is the missing retry-after header plus that error_code.

So before anything else, log the full error body. Not err.status. The body.

import anthropic

client = anthropic.Anthropic()

try:
    msg = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": "hi"}],
    )
except anthropic.APIStatusError as e:
    print(e.status_code)                          # 429
    print(e.response.headers.get("retry-after"))  # None means spend cap
    print(e.body)                                 # the error_code lives here
    print(e.request_id)                           # req_018Ee... for support

The retry-after check on that second line is doing almost all of the diagnostic work. Present means wait. Absent on a 429 means stop retrying and go look at billing.

What actually counts against your Claude rate limits

Now, the per-minute limits. There are three of them running at once, per model, and you can trip any one.

RPM, requests per minute.
ITPM, input tokens per minute.
OTPM, output tokens per minute.

They are enforced per model class, so Sonnet traffic and Haiku traffic draw from separate buckets and can run at full speed at the same time. Worth knowing: on the current lineup, Sonnet 5 has its own bucket while Sonnet 4.6 and 4.5 share a combined one, and the same split applies to Opus 5 versus the Opus 4.x family. Pinning half your traffic to an older Sonnet does not double your Sonnet 5 headroom.

The rate limits documentation also says something that should change how you think about bursts. The limiter is a token bucket, which means capacity refills continuously instead of resetting on a clean minute boundary. A 60 RPM limit behaves much more like 1 request per second than like 60 free requests at the top of every minute. So if you fan out 60 concurrent calls at 12:00:00 you will get rate limited, even though your average for that minute was exactly at the ceiling.

I hit this exact shape of problem building Scholarian, which fetched and ranked over 10,000 academic papers. The moment you write a loop that fans out one model call per document, your average throughput looks perfectly fine on a dashboard and your p99 latency (the slowest 1% of calls) is a wall of 429s, because nothing in that loop is pacing itself.

The one number most people get wrong

For most Claude models, only uncached input tokens count against ITPM.

Specifically: input_tokens and cache_creation_input_tokens count, and cache_read_input_tokens does not. So a cache hit is free as far as your rate limit is concerned, and it is billed at a reduced rate on top of that. (Claude Haiku 3.5 is the exception and does count cache reads.)

Do the arithmetic on that, because it is not a small effect. Say you are on the Start tier with 2,000,000 ITPM on Sonnet 5, and you are running a coding agent that ships roughly 180,000 tokens of context per turn.

Without caching, 2,000,000 divided by 180,000 is about 11 turns per minute for your entire organization.

With prompt caching where 170,000 of those tokens are cache reads, only 10,000 counts against ITPM. That is 200 turns per minute off the exact same limit! Anthropic's own worked example frames it the same way, and it is why they publish your cache hit rate on the Usage page right next to the rate limit charts.

One nuance people miss: cache_creation_input_tokens does count. Writing the cache costs you full ITPM once, and reading it is free. So a workload that keeps rebuilding the cache because the prefix keeps shifting gets the worst of both.

Output is simpler. OTPM is measured on tokens actually produced, in real time, and max_tokens never factors into it. There is genuinely no rate limit penalty for setting a generous max_tokens, so stop shaving it down for that reason.

The headers you should be reading instead of guessing

Every response, including the successful ones, carries your current standing. And most people never look at them.

curl -sS -D - -o /dev/null https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-5","max_tokens":16,
       "messages":[{"role":"user","content":"hi"}]}'

Run that and you get back a block of anthropic-ratelimit-* headers. The ones worth wiring into a metric:

anthropic-ratelimit-requests-remaining
anthropic-ratelimit-input-tokens-remaining
anthropic-ratelimit-output-tokens-remaining
the matching -reset fields, which are RFC 3339 timestamps rather than durations

Two gotchas here. The remaining values are rounded to the nearest thousand, so treat them as a gauge and not as an accountant. And the generic anthropic-ratelimit-tokens-* triplet reports whichever limit is currently most restrictive, which means the number can jump between input and output accounting between requests without anything being wrong.

Identify the failure and read the remaining-capacity headers before changing retry logic. Backoff cannot fix a spend cap, and tuning it first wastes time.

At one of my previous roles I built an internal tool that pulled the data logs off an AI product and turned them into latency and slowdown reports, and the useful part was never the clever part. It was just having the numbers somewhere you could look at them without reproducing the bug first. Same idea here. Export those three headers as gauges and most rate limit debugging stops being detective work.

Retrying correctly, and where retrying stops helping

The rule in plain English: if the response gave you a retry-after, wait exactly that long. If it did not, double your wait each attempt and add randomness so your workers do not all wake up at the same instant.

import random, time, anthropic

client = anthropic.Anthropic(max_retries=0)  # we handle it ourselves

def call_with_retry(**kwargs):
    for attempt in range(5):
        try:
            return client.messages.create(**kwargs)
        except anthropic.APIStatusError as e:
            if e.status_code not in (429, 500, 502, 503, 504, 529):
                raise
            hinted = e.response.headers.get("retry-after")
            if e.status_code == 429 and hinted is None:
                raise  # spend cap, so retrying is pointless
            wait = float(hinted) if hinted else min(2 ** attempt, 30)
            time.sleep(wait * random.uniform(0.8, 1.2))
    raise RuntimeError("exhausted retries")

The jitter multiplier on the last line is the bit people skip, and it is the bit that matters at concurrency. Without it, 50 workers that all got rate limited at the same moment will all retry at the same moment, and you get a thundering herd that reproduces the original 429 on a fixed schedule forever.

Also notice max_retries=0 at the top. The official SDKs retry twice by default with their own backoff, honoring retry-after. That default is fine for a script. It is not fine when you are also running your own retry loop, because the two layers multiply and you quietly end up with 3x the attempts you thought you configured.

But here is the honest limit of all of this, and it is why the post does not end here.

Retries are a queueing strategy. They smooth out bursts against a ceiling you are near. They do nothing at all when you are structurally above the ceiling, because every retry is just another request competing for the same bucket. If your steady-state demand is 3,000,000 input tokens per minute and your limit is 2,000,000, no backoff curve in the world fixes that. You need more ceiling.

Setup is done, now the part that actually raises the ceiling

Everything above happens inside one API key against one endpoint. Reading the error properly and then caching aggressively will genuinely resolve most Claude rate limit errors, and if you are a solo developer on one key, honestly stop reading here and go turn on prompt caching. You will get more out of that one afternoon than out of any infrastructure.

But if you are past that point, the interesting fact is that "Claude capacity" is not one pool. It is four, and they are metered by different organizations running different quota systems.

Four separate places Claude capacity lives

1. The direct Claude API, by usage tier. Start, Build, Scale, then Custom. Each tier carries both per-minute limits and a monthly spend cap, and as of mid-2026 those caps sit at 500USD on Start, 1,000USD on Build, and 200,000USD on Scale. Tiers move up automatically as you build usage history, and you can also request an increase from the Console.

2. Amazon Bedrock. Claude on Bedrock runs against AWS service quotas, which are per-account and per-region and adjustable through the Service Quotas console. These have no relationship to your Anthropic tier at all. Bedrock even splits its own quotas across two inference endpoints, tracked separately for the same underlying model.

3. Google's platform. Claude models run there through the model garden against Google Cloud quotas, again on a completely separate meter.

4. Priority Tier. Anthropic sells committed capacity with its own anthropic-priority-* headers and its own bucket, sitting alongside your standard limits rather than replacing them.

The consequence is the interesting part. An organization pinned at 100% of its direct-API ITPM might have an entirely idle Bedrock quota sitting in two regions. The capacity exists, and it is already paid for. The problem is that your application has one base URL and one key, so it cannot reach any of it.

One clarification before moving on, because this is the most common mix-up of all. If your error reads API Error: rate limit reached inside Claude Code on a Pro or Max plan, none of this section applies to you, because that is a subscription meter and not the API. Two different systems, two different clocks, and I wrote about Claude Pro usage limits separately.

Where a gateway changes the math

This is the job an AI gateway does, which you will also see called an LLM gateway or an LLM proxy. It is a proxy that speaks every provider's API shape, sits between your code and the model, and holds the routing decisions your application shouldn't have to.

Let me lead with what it cannot do, because that part gets oversold constantly. A gateway does not raise Anthropic's ceiling. Your Start tier is still your Start tier. If you run one Anthropic key through a gateway and change nothing else, you will hit exactly the same 429 at exactly the same token count, plus a few microseconds of hop.

What it changes is how many buckets a single request can reach, and how fast it gives up on a bad one. Bifrost is Maxim AI's high-performance, open-source AI gateway that unifies access to 20+ providers through a single OpenAI-compatible API. Its retry and fallback layer distinguishes upstream failures from credential failures: it retries 5xx and network errors against the same key, rotates keys for 429, 401, 402, and 403 responses, and moves to the next provider only after the current provider's retry budget is exhausted. That behavior maps directly to the different Claude failure modes in this post.

Pooling keys so one bucket is not the whole story

Rate limits are set at the organization level, so a second organization with its own key is a second bucket. A gateway can treat several keys as one logical pool.

{
  "providers": {
    "anthropic": {
      "keys": [
        { "name": "primary",  "value": "env.ANTHROPIC_API_KEY",   "models": ["*"], "weight": 0.7 },
        { "name": "overflow", "value": "env.ANTHROPIC_API_KEY_2", "models": ["*"], "weight": 0.3 }
      ]
    }
  }
}

Traffic splits 70/30 by weight. But the part that matters for 429s is what happens on failure. Bifrost rotates keys when the failure is bound to the credential rather than to the request, which it defines as 429, 401, 403 and 402. A rate limited key gets marked used for that cycle and the request carries on against another key. Auth and billing failures mark the key dead for the remainder of that request. Once every key has been tried, the set resets and a fresh weighted round begins.

Key selection is scored rather than round robin, using recent error rates, latency, and observed rate limit hits, with weights recomputed on a short interval and penalties decaying once a key recovers. That is adaptive load balancing applied to credentials instead of servers, and it matters here because a key that just returned a 429 is a bad destination for the next 30 seconds specifically, and not permanently.

Falling back across channels, not just keys

Key pooling only helps if you have more keys. Falling back across the four capacity pools helps even when you don't.

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{ "role": "user", "content": "..." }],
  "fallbacks": [
    "bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0",
    "vertex/claude-sonnet-4-5@20250929"
  ]
}

Same request, same Anthropic model family, three completely independent quota systems. If the direct API is capped, the call lands on Bedrock's per-region quota, which knows nothing about your Anthropic tier.

The execution detail worth knowing is that each provider in the chain gets its own full retry budget. A primary at max_retries: 3 with two fallbacks also at 3 means up to 12 attempts on one call, so set your client timeout with that in mind. Retries and fallbacks documents the backoff as min(initial × 2^attempt, max) × jitter(0.8-1.2), with defaults of 500ms initial and a 5000ms cap.

And a real gotcha, stated plainly because it will bite you: max_retries defaults to 0. Retries are off until you turn them on.

{ "network_config": { "max_retries": 3, "retry_backoff_initial": 500, "retry_backoff_max": 5000 } }

Two more things that buy headroom

Caching at the gateway instead of per service. If four services ask the same question, an application-level cache inside each one misses four times. A gateway cache hits three of them, and every hit is a request that never touches your ITPM at all. Bifrost's version only engages when a request carries an x-bf-cache-key header, defaults to a 5 minute time-to-live and a 0.8 cosine similarity threshold, and can run in exact-match mode with no embedding provider at all. That threshold is the whole product, and I went into why in a post on semantic caching.

Stopping one tenant from eating the org bucket. Your organization limit is shared, so one runaway batch job starves the interactive traffic sitting behind it. Governance limits let you cap consumption per virtual key before the request ever reaches Anthropic, in tokens or in requests, with calendar-aligned reset periods running from 1m up to 1Y. When a cap trips, the gateway reports the breach with its own code instead of a vague 429: token_limited or request_limited on a 429, budget_exceeded on a 402, and 403 for a blocked model or provider. The governance docs carry the full mapping.

That is a real difference in debuggability. "Your batch job hit its token cap" is a fix you can act on in a minute. "Anthropic said 429" is an afternoon.

Pointing Claude Code at it

Since a lot of people arriving at this error are inside a coding agent rather than a Python service, the integration is two environment variables. Bifrost exposes an Anthropic-compatible endpoint, so the client never knows anything changed.

export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic
export ANTHROPIC_API_KEY=dummy-key

The path is /anthropic, and not /v1/anthropic. And the key can be a placeholder because the real credentials live inside the gateway.

Gateway traffic is billed per token against API keys. It uses a separate meter from Pro or Max subscriptions, not a larger subscription allowance.

What a gateway will not fix

Four things, because a post that only lists the wins is a brochure.

It does not help with a 529. An overloaded error is Anthropic-wide capacity, so a fallback to Bedrock might dodge it, but no amount of key pooling on the direct API will.

It does not make a cross-provider fallback behave identically. Sonnet 5 on the direct API and Sonnet 4.5 on Bedrock are not the same model, and if your prompts are tuned tightly or you lean on a specific tool-calling shape, your fallback path needs its own evals. A fallback that silently produces worse answers is arguably worse than a clean 429.

It does not fix a workload that is simply too large. If you need 5x your current ceiling permanently, the answer is a tier increase or committed capacity, and routing is the bridge that gets you there without downtime in the meantime.

And the vendor benchmarks are vendor benchmarks. Bifrost's page claims 20 microseconds of added latency at 5,000 requests per second, and a 50x advantage on that same p99 number over LiteLLM. Those are Maxim's own numbers on Maxim's own harness, and you should treat them the way you treat every published benchmark, which is as a reason to go run your own.

What you should actually do, in order

Ordered by what each one costs you, cheapest first. Stop at the first one that fixes it.

One. Log the full error body and check for retry-after. If it is missing on a 429, you have a spend cap and no retry strategy will ever help you.

Two. Export the three anthropic-ratelimit-*-remaining headers as gauges. You cannot tune what you cannot see.

Three. Turn on prompt caching and check your hit rate on the Usage page. This is the single biggest win available and it costs you an afternoon, because cached reads do not count against ITPM on most models.

Four. Add retry-after-aware backoff with jitter, and make sure you are not stacking it on top of the SDK's own retries.

Five. Move anything non-interactive to the Message Batches API, which has its own separate rate limits.

Six. Request a tier increase. It is free and it is slow, so start it before you need it.

Seven. Only now, put a gateway in front and pool keys or channels. This is real infrastructure with real operational cost, and it is worth it once you have genuinely run out of ceiling, and mostly not before.

The reason I would put it dead last rather than first is that most 429s I have run into were not capacity problems at all. They were a cache that never got turned on, or a retry loop with no jitter, or a spend cap that everybody kept retrying into. Fix those and the ceiling stops being the constraint.

If you have hit a 429 shape that doesn't fit any of the five rows in that table, drop it in the comments, I'd genuinely like to see it. And if you want more of this sort of thing, I'm on X and I write everything up first on my site.