DEV Community: claire nguyen

Best Tools to Block Sensitive Data From Reaching Public LLMs

claire nguyen — Thu, 09 Jul 2026 10:31:24 +0000

Organizations deploying large language models (LLMs) in production face significant data privacy challenges. This guide explores the leading solutions for sensitive data protection with LLMs, focusing on how tools like Bifrost provide robust guardrails, data redaction, and endpoint governance for enterprise-grade security.

The proliferation of large language models (LLMs) across enterprises introduces unprecedented opportunities for innovation, but also new vectors for data leakage. Sensitive information, ranging from personally identifiable information (PII) and protected health information (PHI) to confidential business secrets, can inadvertently reach public LLMs through user prompts or agent interactions. Preventing this data exposure is a critical concern for compliance, security, and maintaining trust. This article examines the various tools and strategies available to safeguard sensitive data from reaching public LLMs, highlighting key features and providing an overview of leading solutions.

The Challenge of Sensitive Data in AI Applications

As AI applications become integrated into daily workflows, employees often use LLMs for tasks that involve sensitive data, such as summarizing documents, drafting emails, or analyzing customer interactions. Without proper controls, these prompts can expose proprietary information or regulated data to third-party model providers. This unintentional data sharing poses significant risks:

Compliance Violations: Breaching regulations like GDPR, HIPAA, SOC 2, and CCPA can result in severe penalties and reputational damage.
Security Risks: Exposure of intellectual property, trade secrets, or access credentials can lead to competitive disadvantages and system vulnerabilities.
Reputational Harm: Data breaches erode customer trust and can have long-lasting negative impacts on a company's public image.
Shadow AI: Employees often use public LLM tools directly on their machines, bypassing corporate network controls. This "shadow AI" creates a blind spot where sensitive data can flow unregulated.

Mitigating these risks requires specialized tools that can inspect, filter, and redact data before it leaves the organization's control.

Key Features of Sensitive Data Protection Tools for LLMs

Effective tools for blocking sensitive data from public LLMs typically offer a combination of the following features:

Data Redaction and Masking: Automatically identifies and replaces sensitive information (e.g., credit card numbers, social security numbers, email addresses) with placeholders or obfuscated values.
PII/PHI Detection: Utilizes predefined patterns and machine learning to detect categories of PII and PHI across various data formats.
Customizable Guardrails: Allows organizations to define their own rules and policies for what constitutes sensitive data, enabling the blocking or redaction of specific keywords, regex patterns, or proprietary information.
Access Control and Virtual Keys: Implements granular access controls to LLM resources, often through virtual keys, to manage who can access which models and under what data governance policies.
Audit Logging: Maintains immutable logs of all requests and responses, providing an audit trail for compliance and forensic analysis.
Real-time Enforcement: Processes prompts and responses in real-time to prevent sensitive data from ever reaching the external LLM provider.
Endpoint Governance: Extends data protection policies to individual employee machines, governing AI usage in desktop applications, browsers, and coding agents, addressing the shadow AI problem.
Integration with Existing Security Infrastructure: Compatibility with existing identity providers (SSO/OIDC) and data loss prevention (DLP) systems.

Top Tools for Protecting Sensitive Data with LLMs

Several solutions are available that offer varying degrees of sensitive data protection for LLM workloads.

Bifrost

Bifrost stands out as an open-source AI gateway that provides comprehensive security and governance features designed to prevent sensitive data from reaching public LLMs. It is particularly well-suited for enterprises requiring robust compliance and control over their AI infrastructure. Bifrost, an open-source AI gateway from Maxim AI, unifies access to over 1000 models via a single API, while integrating advanced data protection mechanisms.

Key sensitive data protection features:

Guardrails: Bifrost offers extensive guardrail capabilities, including native Secrets Detection (backed by Gitleaks) to catch API keys and credentials, and highly configurable Custom Regex for blocking or redacting organization-specific sensitive patterns (including a built-in PII Detection template). It also integrates with external guardrail providers like AWS Bedrock Guardrails, Azure Content Safety, and Patronus AI.
Virtual Keys and Access Control: The platform's virtual keys enable fine-grained access control, allowing administrators to enforce policies on what data can be sent, which models can be used, and at what cost.
Audit Logs: All prompts and responses are logged in immutable audit trails, essential for SOC 2, GDPR, and HIPAA compliance.
Bifrost Edge for Endpoint Governance: A unique differentiator is Bifrost Edge, which extends the gateway's governance and security controls directly to employee machines. This endpoint agent ensures that AI traffic from desktop apps, browser AI, and coding agents is routed through Bifrost, effectively preventing "shadow AI" data leakage with endpoint enforcement on each device. The same guardrails configured in the Bifrost AI gateway are applied automatically to endpoint AI usage.
Deployment Flexibility: Bifrost supports in-VPC deployments and air-gapped environments, ensuring sensitive data never leaves the organization's private cloud infrastructure.

Best for: Enterprises and regulated industries that require a high-performance, open-source, and fully controllable solution for sensitive data protection, comprehensive governance, and endpoint AI security.

Cloudflare AI Gateway

Cloudflare offers an AI Gateway designed to provide caching, rate limiting, and observability for LLM APIs. Its focus is primarily on performance and cost optimization at the network edge. For sensitive data, Cloudflare's platform can be configured with Workers AI and Data Loss Prevention (DLP) to identify and redact sensitive information using machine learning models and predefined patterns. Its integration with the broader Cloudflare security ecosystem offers additional layers of protection.

Best for: Organizations already leveraging Cloudflare for network security and content delivery, looking to extend those capabilities to their LLM API traffic with integrated data protection at the edge.

Kong AI Gateway

Kong is an API gateway that has extended its capabilities to support AI workloads. The Kong AI Gateway provides a flexible platform for managing, securing, and extending AI APIs. For data protection, Kong's plugin ecosystem allows for the integration of custom policies and third-party DLP solutions. It can be configured to perform data masking or redaction using specific plugins or custom logic applied to API requests and responses. Kong is particularly strong for organizations that already manage their APIs with Kong.

Best for: Enterprises with existing Kong API gateway deployments that need to add sensitive data protection and governance layers to their LLM API calls using a flexible plugin architecture.

Google Cloud DLP

While not an AI gateway itself, Google Cloud Data Loss Prevention (DLP) is a powerful tool that can be integrated with LLM workflows for sensitive data protection. It excels at discovering, classifying, and redacting sensitive data (including over 150 predefined detectors for PII, PHI, and financial data) across various data sources. Teams can use Cloud DLP in conjunction with an LLM gateway or directly within their application logic to inspect and transform prompts and responses before they interact with LLMs.

Best for: Organizations heavily invested in Google Cloud, seeking a robust, scalable, and highly accurate data loss prevention service to integrate into their custom LLM applications and infrastructure.

How to Choose the Right Solution for Your Organization

Selecting the best tool for blocking sensitive data from public LLMs depends on several factors:

Deployment Environment: Consider whether your organization requires on-premises, VPC, or cloud-native deployment options.
Existing Infrastructure: Evaluate how well the solution integrates with your current API management, security, and identity systems.
Compliance Requirements: Assess the tool's ability to meet specific regulatory standards (e.g., GDPR, HIPAA, SOC 2) through features like audit logging, data redaction, and access control.
Scalability and Performance: Ensure the solution can handle your anticipated LLM traffic volume without introducing significant latency.
Endpoint Governance Needs: Determine the importance of extending data protection policies to individual user machines to address shadow AI.
Customization and Extensibility: Look for platforms that allow custom guardrails, plugin development, or integration with specialized DLP services.

For organizations that prioritize open-source flexibility, high performance, comprehensive governance, and full control over their AI infrastructure, including robust endpoint protection against shadow AI, Bifrost presents a compelling solution. Its combination of advanced guardrails, virtual keys, audit logging, and the unique capabilities of Bifrost Edge provides an end-to-end framework for securing sensitive data in the age of LLMs.

Sources

Cloudflare. "Data privacy in Workers AI". Cloudflare Documentation. https://developers.cloudflare.com/workers-ai/data-privacy/
Google Cloud. "Data Loss Prevention". Google Cloud. https://cloud.google.com/dlp

Async LLM inference in CI: stop build workers blocking on slow jobs

claire nguyen — Thu, 25 Jun 2026 13:21:46 +0000

TL;DR: Async inference through an AI gateway lets CI build workers submit a long LLM job, get an id back, and poll later, so a 30-second model call stops holding a worker hostage. Here's how I wired it with Bifrost.

Our build workers at Buildkite were each blocked for up to 35 seconds waiting on a single LLM call that summarised failed test output. With a few hundred concurrent builds running through our compute cluster, that's a pile of expensive compute sitting idle on one synchronous request to a model provider. We moved those jobs behind Bifrost, the open-source AI gateway by Maxim AI, and switched them to async submit-and-poll so the worker could get back to running the actual build while the summary cooked in the background.

What async inference actually does

Async inference is a request pattern where the client submits a job, gets an identifier back straight away, and polls for the result later instead of holding the connection open. With Bifrost you set x-bf-async: true on the request and get an x-bf-async-id in return, then poll that id once the model has finished. The docs overview covers the submit and poll lifecycle.

The win is mechanical, not magic. A worker that no longer blocks on a slow upstream can pick up the next build step. On a fleet where each agent costs real money per minute, freeing 35 seconds per build adds up fast across a few hundred concurrent runs.

Why synchronous LLM calls stall a build fleet

A build agent is a finite resource. When it makes a blocking HTTP call to an LLM and the provider takes 30-plus seconds, that agent is doing nothing but waiting on a socket. Multiply that by every failing build wanting a summary, and you've quietly turned your model provider's latency into your queue depth.

We saw exactly this. P95 latency on the summariser sat around 28 seconds, and during a flaky-test storm the build queue backed up because agents were parked on those calls. The compute was healthy; the scheduling was wrong. The fix is to decouple "ask for a summary" from "wait for a summary."

Wiring async submit and poll

The change was small. We added two headers and split one blocking call into a submit step and a later poll step. The Bifrost endpoint stays OpenAI-compatible, so the request body didn't change at all, which is the point of a drop-in replacement.

# Submit: fire the job, return immediately with an id
curl -s -X POST http://bifrost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-bf-async: true" \
  -H "x-bf-dim-team: build-platform" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "Summarise this failed test log..."}]
  }'
# response carries x-bf-async-id: job-8f21

# Poll later, from a separate build step, once the agent has moved on
curl -s http://bifrost:8080/v1/chat/completions \
  -H "x-bf-async-id: job-8f21"

We submit at the start of the post-build hook, run the rest of the cleanup, then poll near the end. By the time we poll, the summary is usually ready, so the agent almost never waits. The x-bf-dim-team header tags the request with our team name, which Bifrost auto-forwards to logs, traces, and Prometheus so we can see which team's jobs are driving spend.

Keeping costs and failures visible

Async jobs are easy to lose track of, so observability matters more, not less. With Bifrost the custom x-bf-dim-* dimension headers flow straight into the observability layer, which writes asynchronously and adds under 0.1ms of overhead per the benchmarking docs. That let us build a Grafana panel keyed on team and job type without instrumenting our own code.

Failover still applies to async jobs. We kept automatic fallbacks configured so that if our primary provider returns 502s, the gateway retries against a secondary before the job id ever comes back failed. On the throughput side, a single instance sustains 5,000 RPS at 100% success with roughly 11µs of gateway overhead on a t3.xlarge, per the published benchmarks, so the gateway itself was never the bottleneck in our queue.

Trade-offs and limitations

Async is not free. You now own a polling loop and the job ids it depends on. If a build agent dies between submit and poll, you need those ids in durable storage or the result is orphaned. We push ids into the build's metadata so a retried step can recover them.

On the Bifrost side, self-hosting carries real operational weight. A production deployment needs Postgres backing it, which is one more stateful service for my team to run and patch. Clustering for high availability is an enterprise feature, not part of the open-source core, so a single-node deploy is a single point of failure you have to plan around. The ecosystem is also younger than LiteLLM, so there's less community Q and A when you hit an edge case. None of that was a dealbreaker for us, but plan the operational side before you commit.

Wrapping up

Switching CI summarisation to async inference through Bifrost took the blocking time off our build agents and stopped a slow model provider from setting our queue depth. The headers are simple, the endpoint stays OpenAI-compatible, and the spend stays visible per team. If you run LLM calls inside a build fleet and your agents are parked waiting on them, async submit-and-poll is worth a look: https://getmaxim.ai/bifrost/book-a-demo

How to See Every MCP Server Developers Have Configured

claire nguyen — Wed, 24 Jun 2026 18:26:39 +0000

Developers frequently configure Model Context Protocol (MCP) servers with AI agents, yet many organizations lack visibility into these connections. This article explores the challenges of shadow AI in the context of MCP servers and how Bifrost and Bifrost Edge provide a solution for comprehensive visibility and governance.

The rapid adoption of AI agents has introduced new complexities for enterprise security and governance. As developers integrate powerful AI tools into their workflows, they often connect these tools to external services and internal systems through Model Context Protocol (MCP) servers. Without a centralized way to track and control these connections, organizations face a significant "shadow AI" problem, where critical data pathways operate outside of IT and security oversight. Gaining visibility into every MCP server configured across an organization's endpoints is essential for maintaining security and compliance. Bifrost, an open-source AI gateway, with its endpoint component Bifrost Edge, offers a robust framework for addressing this challenge.

The Challenge of Ungoverned MCP Servers and Shadow AI

AI agents are no longer passive tools; they actively plan, call external tools, and execute actions, often with delegated authority. The Model Context Protocol (MCP), introduced in November 2024, enables this by providing a standardized way for AI systems to interact with external data sources, applications, and services. It acts like a universal adapter, allowing AI assistants to query databases, read files, call APIs, and even run shell commands. This capability, while powerful for developers, creates a new security risk: shadow MCP.

Shadow MCP refers to the MCP servers that employees configure for their AI applications without formal security review or central governance. These servers can be wired directly into developer tools like Claude Code, Cursor, or VS Code Copilot Chat, often with little to no visibility for IT or security teams. The risks are substantial:

Data exfiltration: Ungoverned MCP servers can inadvertently expose sensitive data to unauthorized external systems.
Privilege escalation: An agent using an MCP server might inherit the broad access of the developer who installed it, potentially accessing privileged systems without explicit authorization.
Compliance gaps: Without an audit trail of MCP server usage and data flows, organizations struggle to meet regulatory requirements like SOC 2, GDPR, HIPAA, and ISO 27001.
Misconfiguration and vulnerabilities: Misconfigured MCP servers can create new attack surfaces, acting as backdoors into internal systems.

A 2026 Verizon Data Breach Investigations Report noted a fourfold increase in shadow AI detections, highlighting it as a common insider action in enterprise environments. This growing challenge underscores the need for comprehensive visibility into endpoint AI.

What is an MCP Server?

The Model Context Protocol (MCP) is an open standard designed to enable AI applications to connect with external data and services seamlessly. It standardizes how AI agents discover and interact with external "tools" at runtime. These tools can include anything from internal databases and file systems to web search engines and custom APIs.

An MCP ecosystem involves three primary components:

MCP Host: The AI application or environment (e.g., an AI-powered IDE or conversational AI) that contains the LLM.
MCP Client: A component within the MCP host that facilitates communication between the LLM and the MCP server.
MCP Server: The external service that provides context, data, or capabilities to the LLM. It exposes specific "tools" that the AI agent can invoke to perform actions.

For example, an AI coding assistant might use an MCP server to access a company's codebase, an issue tracker, or internal documentation. This allows the AI to provide real-time, context-aware assistance beyond its initial training data. The protocol leverages JSON-RPC 2.0 messages for communication between client and server.

Bifrost Edge: Gaining Visibility into Endpoint MCP Usage

Traditional security tools often fail to close the visibility gap around shadow AI because they focus on network perimeters, which endpoint AI frequently bypasses through direct, encrypted connections. Effectively governing MCP servers requires controls at the point where AI actually runs: the endpoint itself.

Bifrost addresses this by combining the Bifrost AI gateway, as the control plane and policy engine, with Bifrost Edge, which extends that same governance to every machine. Bifrost Edge is a lightweight agent that runs natively on macOS, Windows, and Linux devices. Its primary function is to route all AI traffic from the endpoint through the organization's Bifrost gateway automatically. This brings desktop applications, browser AI, coding agents, and, crucially, the MCP servers those tools connect to, under a single, unified governance framework.

Bifrost Edge tackles shadow MCP by providing essential visibility:

Fleet-wide MCP Server Inventory: Bifrost Edge discovers and inventories the MCP servers configured within each AI application across the entire fleet of devices. This creates a live, deduplicated catalog of every MCP server in use, allowing security teams to answer "what MCP servers are running on our fleet?" with real data.
Real-time Discovery: New MCP servers are detected as they appear, rather than during periodic audits. This continuous monitoring ensures that administrators have an up-to-date view of their AI ecosystem.
Supported Applications: Bifrost Edge supports discovery for a growing list of popular AI applications, including Claude Code, Claude Desktop, Gemini CLI, OpenCode, Codex, and Cursor. This ensures comprehensive coverage for the tools developers commonly use.

This visibility provides the foundation for effective governance, transforming an invisible threat into an observable, manageable aspect of the enterprise AI landscape.

From Visibility to Control: Approving and Denying MCP Servers

Once Bifrost Edge provides visibility into configured MCP servers, the Bifrost AI gateway enables administrators to apply granular policy controls. The governance configured at the gateway level automatically applies to endpoint AI traffic routed by Edge.

Administrators can manage detected MCP servers through a centralized dashboard:

Per-Server Allow/Deny Decisions: The system allows admins to make explicit allow or deny decisions for each discovered MCP server. This policy is then enforced directly on the device. A denied server cannot be used, even if an application had it configured before the policy was established.
Approval Workflows: When Edge detects a new MCP server, it can automatically request approval in the admin console. Administrators can configure whether pending servers are allowed or blocked while awaiting review, ensuring control from the moment of discovery.
Unified Governance Policies: Existing Bifrost features like virtual keys, budgets, rate limits, and guardrails extend to MCP server traffic. This means the same content safety and data loss prevention policies that apply to API calls at the gateway also protect prompts and responses from endpoint AI, preventing sensitive content like secrets or PII from leaving the machine.
Audit Logs: Every MCP tool call is recorded in immutable audit logs, providing a critical trail for compliance with standards such as SOC 2, GDPR, and HIPAA.

Deployment and Integration for Comprehensive Governance

Bifrost Edge is designed for fleet-wide deployment, rather than relying on manual per-machine setup. Organizations can push Edge to every device using existing device management (MDM) platforms, delivering a managed configuration that points each machine to the organization's Bifrost gateway. This ensures that governance applies across the entire fleet without requiring individual users to reconfigure their AI tools.

Supported MDM platforms include Jamf, Microsoft Intune, Kandji, Omnissa Workspace ONE, and JumpCloud, covering macOS, Windows, and Linux endpoints. This integration with existing IT infrastructure streamlines the rollout process, making it feasible to achieve comprehensive endpoint AI governance at scale.

Bifrost Edge is currently in alpha and available to enterprise customers of the Bifrost gateway. Teams interested in gaining this level of visibility and control can request early access to evaluate its capabilities.

By pairing the powerful Bifrost AI gateway as a policy engine with Bifrost Edge for endpoint enforcement, organizations can transform their approach to AI agent governance. This combined strategy ensures that every MCP server, whether sanctioned or not, becomes visible and governable, helping to secure sensitive data and maintain compliance in an evolving AI landscape.

Sources

Governing MCP Servers on Employee Devices: A Critical Step for Enterprise AI

claire nguyen — Wed, 24 Jun 2026 18:26:36 +0000

As AI agents become integral to enterprise operations, their ability to interact with external tools and data sources via Model Context Protocol (MCP) servers introduces powerful new capabilities. However, this extensibility also creates a significant security and governance challenge: managing which MCP servers employees connect to from their company machines. Without proper controls, this can lead to "shadow AI" — ungoverned AI usage that bypasses security policies and exposes sensitive data.

This article examines why governing MCP server usage on endpoint devices is crucial and how a comprehensive AI gateway, paired with endpoint AI governance, provides the necessary controls. Bifrost, an open-source AI gateway from Maxim AI, offers a solution that extends centralized policy enforcement directly to employee machines, ensuring that all AI interactions align with enterprise security and compliance standards.

What are MCP Servers and Why Do They Matter?

Model Context Protocol (MCP) is an open standard designed to enable AI models, particularly large language models (LLMs), to interact with external tools and data sources in a consistent and secure manner. Think of MCP as a standardized communication layer allowing AI clients (like Claude, ChatGPT, or coding agents) to discover and invoke tools without requiring custom, one-off integrations for every service. An MCP server is the implementation of this protocol, acting as a bridge between an AI agent and external systems like APIs, databases, file systems, or SaaS applications.

These servers allow AI agents to move beyond mere reasoning to taking actions—reading files, calling APIs, querying databases, and sending messages. This capability is what makes AI agents so powerful in a business context, enabling automation and complex workflows.

The Rise of Ungoverned AI Tool Usage (Shadow AI)

While MCP servers offer immense utility, their ease of creation and deployment has inadvertently led to a proliferation of unmanaged tools within organizations. Developers can spin up an MCP server in minutes, making it simple to expose functionality and experiment. However, this agility often means that IT and security teams lose visibility into what tools are running, what data they access, and what actions they can perform. This phenomenon, often termed "shadow AI," presents several critical risks:

Data Exfiltration: MCP servers, when connected to filesystems or APIs, can be manipulated by malicious prompts to extract sensitive company data without detection.
Unauthorized Command Execution: A compromised or malicious MCP server could execute arbitrary commands or access systems with elevated privileges, bypassing intended access controls.
Absence of Audit and Policy: Without centralized governance, there is no unified audit trail, budget control, or guardrails applied to these interactions, making compliance (e.g., SOC 2, GDPR, HIPAA) extremely difficult.
Supply Chain Risks: MCP servers often rely on various software components, making them vulnerable to supply chain attacks, where malicious code could be introduced.
Tool Poisoning and Tool Shadowing: Attackers can craft malicious tools with names similar to legitimate ones, or embed hidden directives in tool metadata, leading AI agents to invoke unintended or harmful actions.

The problem is that traditional network controls and endpoint detection tools were not designed to manage this new class of AI-driven, tool-calling traffic. A gateway can only govern traffic configured to flow through it. When employees install desktop apps, browse AI in the browser, or paste MCP server configurations directly into their coding agents, that traffic often bypasses central controls entirely. This creates a significant governance gap where critical enterprise data and systems are exposed.

How Bifrost Edge Addresses MCP Server Governance

Bifrost, the AI gateway, functions as the central policy engine for AI traffic. To address the challenge of ungoverned MCP server usage, Bifrost Edge extends this same governance directly to the endpoint. Bifrost Edge runs on every machine in an organization, ensuring that all AI traffic, including connections to MCP servers, is routed through the central Bifrost AI gateway where policies are enforced. This approach provides critical visibility and control over what has historically been a significant blind spot.

Bifrost Edge brings several core capabilities to bear on MCP server governance:

Fleet-Wide Inventory and Discovery: Edge actively inventories the MCP servers configured within AI applications across the entire device fleet. Administrators gain a live, deduplicated catalog of every discovered MCP server, along with details on where it's configured and across how many devices. This provides the much-needed visibility to answer questions like "what MCP servers are running on our fleet?".
Granular Allow/Deny Decisions: With a clear inventory, administrators can make explicit allow or deny decisions for each discovered MCP server. A denied server is blocked at the device level, preventing any data from leaving the machine for that server, even if a user had it previously configured. This shifts from reactive responses to proactive policy enforcement.
Approval Workflows: When Edge detects a new MCP server, it can automatically request approval in the Bifrost admin console. Administrators configure whether pending servers are allowed or blocked while awaiting review, providing a controlled path for new tools [cite: Edge admin approvals docs].
Existing Governance Applies: The same virtual keys, budgets, rate limits, and guardrails configured in the Bifrost AI gateway automatically apply to endpoint AI traffic, including MCP server interactions [cite: Edge security docs]. This ensures consistency of policy across all AI workloads. For instance, MCP tool filtering controls which specific tools an approved MCP server can call based on the virtual key in use.

Granular Control with Centralized Policy

Administrators manage MCP server approvals and denials through the Bifrost admin console. This provides a centralized dashboard to review discovered AI apps and MCP servers, making allow/deny decisions that are then enforced on devices across the fleet [cite: Edge admin approvals docs]. This centralized approach streamlines management, ensuring that policy changes take effect rapidly across all connected machines without requiring manual intervention on individual devices.

The underlying policy mechanisms, such as guardrails and audit logs, protect the entire AI data flow. Guardrails, including native Secrets Detection and Custom Regex (e.g., for PII), inspect prompts and responses, catching sensitive content before it leaves the company boundary. Audit logs maintain an immutable record of every AI request, providing the necessary evidence for compliance reporting (SOC 2, GDPR, HIPAA, ISO 27001).

Streamlined Deployment for Endpoint AI Governance

Deploying endpoint AI governance across an entire organization requires seamless rollout. Bifrost Edge is built for fleet-wide deployment using existing device management (MDM) platforms [cite: Edge deployment MDM docs]. Organizations can push the Edge agent to every machine via tools like Jamf, Microsoft Intune, Kandji, Omnissa Workspace ONE, and JumpCloud [cite: Edge deployment MDM docs].

This managed deployment means that machines arrive pre-configured to point at the organization's Bifrost. After a single sign-in via the organization's SSO, the Edge agent seamlessly applies governance to all supported AI traffic [cite: Edge how it works docs]. This approach eliminates the burden of per-app setup for users and ensures that comprehensive governance is in place from the moment a device is provisioned.

The Broader Context: Comprehensive AI Governance

Governing MCP servers on employee devices is a critical component of a larger strategy for enterprise AI governance. By combining the powerful policy engine of the Bifrost AI gateway with the endpoint reach of Bifrost Edge, organizations gain full visibility and control over their AI footprint. This AI Gateway + Bifrost Edge narrative ensures that all AI interactions—whether directly integrated via the gateway or used on employee laptops—adhere to the same security, compliance, and cost management policies.

Bifrost Edge is currently in alpha, with teams registering for onboarding to implement this critical layer of enterprise AI security. The ability to automatically discover, approve, and deny specific MCP servers across a fleet of devices completes the governance story, transforming shadow AI into governed AI.

Sources

Securing MCP Servers Across a Company: A Practical Guide

claire nguyen — Wed, 24 Jun 2026 18:25:46 +0000

Securing Model Context Protocol (MCP) servers is crucial for enterprise AI governance. This guide explores the risks of ungoverned MCP server use and how Bifrost with Bifrost Edge provides a comprehensive solution for visibility and control.

AI agents and advanced LLMs increasingly rely on Model Context Protocol (MCP) servers to extend their capabilities, enabling them to read files, call APIs, and interact with external tools. While this tool-use functionality enhances AI utility, it also introduces significant security and compliance risks if not properly governed. For many organizations, the proliferation of MCP servers across employee machines, often outside IT visibility, constitutes a critical security blind spot.

Understanding the MCP Server Security Challenge

The Model Context Protocol allows AI agents to discover and execute external tools, fundamentally changing how AI applications interact with company data and systems. This capability is powerful, but it bypasses traditional security perimeters.

A primary challenge is the "shadow AI" problem. Employees may configure MCP servers within their coding agents (e.g., Claude Code, Cursor) or desktop AI applications without centralized oversight. This leads to:

Data Leakage Risks: Ungoverned MCP servers can facilitate the exfiltration of sensitive company data to unauthorized external services or models.
Compliance Violations: Without an audit trail or control over tool execution, organizations face difficulty adhering to regulatory requirements like GDPR, HIPAA, or SOC 2.
Malicious Tool Execution: An agent might be instructed to use a compromised or unapproved MCP server, potentially leading to unauthorized access or system manipulation.
Lack of Visibility: Security teams often have no clear inventory of which MCP servers are being used, by whom, or for what purpose, making risk assessment impossible.

This lack of visibility and control at the endpoint creates a significant gap in an organization's AI security posture, making it imperative to implement robust governance strategies.

How Bifrost Addresses MCP Server Security

Securing MCP servers requires a comprehensive approach that combines centralized policy enforcement with endpoint governance. Bifrost, an open-source AI gateway from Maxim AI, provides this unified solution, ensuring that AI traffic—including MCP server interactions—is visible, controlled, and compliant.

Bifrost functions as an MCP gateway, centralizing AI model and tool access. It can expose a curated set of tools and also manage connections to external MCP servers. The same virtual keys, budgets, rate limits, and guardrails configured in Bifrost apply to MCP interactions, providing a foundational layer of security.

The crucial component for extending this governance to every machine is Bifrost Edge. Bifrost Edge is an endpoint agent that runs on macOS, Windows, and Linux devices, routing all AI traffic—including requests to MCP servers—through the organization's Bifrost gateway. This ensures that every AI interaction, regardless of the application or its location, is subjected to the same security policies.

Comprehensive MCP Server Governance with Bifrost Edge

Bifrost Edge extends the governance capabilities of the Bifrost AI gateway directly to the endpoint, offering unparalleled control over MCP server usage. This includes:

Automated Discovery and Inventory: Bifrost Edge automatically inventories MCP servers configured within AI applications on employee machines. This capability builds a live, fleet-wide catalog of all discovered MCP servers, finally giving administrators visibility into what tools agents are using in production. This eliminates the blind spot of "shadow AI" where MCP servers operate unnoticed.
Centralized Approval Workflow: Once MCP servers are discovered, administrators can review them within the Bifrost console. They can then make explicit per-server allow or deny decisions. This means a denied server cannot be used by any AI application on any governed device, even if a user previously configured it.
Enforced Security and Guardrails: Because all MCP server traffic routes through Bifrost, every guardrail already configured at the gateway applies automatically. This includes native secrets detection, custom regex patterns (for PII, compliance, or internal data), and integrations with third-party content safety solutions like AWS Bedrock Guardrails, Azure Content Safety, and Patronus AI. These guardrails protect sensitive information from leaving the organization via agent tool calls.
Audit Logs for Compliance: Every interaction involving an MCP server, whether approved or blocked, is recorded in immutable audit logs. This provides a comprehensive trail for compliance, forensic analysis, and ensuring adherence to internal security policies and external regulations (SOC 2, GDPR, HIPAA, ISO 27001).
Per-Virtual Key MCP Tool Filtering: Bifrost's virtual keys provide granular control over which MCP tools (and by extension, which MCP servers) are accessible to specific users, teams, or projects. This allows organizations to segment access based on roles and requirements, ensuring that only authorized agents can interact with approved tools.

Deploying Bifrost Edge for Fleet-Wide Governance

For organizations with large fleets of employee machines, manual deployment and configuration of endpoint agents are impractical. Bifrost Edge is designed for mass deployment through existing Mobile Device Management (MDM) platforms.

Supported MDM solutions include Jamf, Microsoft Intune, Kandji, Omnissa Workspace ONE, and JumpCloud. This enables silent, fleet-wide rollout of the Edge agent, pre-configured with the necessary connection settings to point devices to the organization's Bifrost gateway.

The typical first-launch flow for an end-user involves:

Bifrost Edge installs silently via MDM.
A one-time setup approval on first run.
The user signs in through their browser using existing single sign-on (SSO), linking the device to their identity and syncing policies.
Governance automatically applies to all supported AI traffic, including MCP server interactions.

This streamlined deployment and user-onboarding process ensures that organizations can quickly bring all endpoint AI traffic under governance, closing shadow AI gaps without disrupting user workflows or requiring extensive manual configuration.

Conclusion

The rise of AI agents and MCP servers introduces new vectors for data exfiltration and compliance risks. Securing these interactions is not merely a technical challenge; it is a critical requirement for maintaining enterprise security and regulatory adherence. By combining the powerful policy engine of the Bifrost AI gateway with the endpoint enforcement capabilities of Bifrost Edge, organizations gain comprehensive visibility and control over all AI traffic, including the often-overlooked realm of MCP server usage. This integrated approach ensures that AI innovations can be adopted safely and compliantly across the entire company.

Teams evaluating AI gateway and endpoint governance solutions can request a Bifrost demo to see how it can secure their MCP server landscape or review the open-source repository.

Sources

"The Model Context Protocol: Bridging the Gap Between LLMs and Tools." Maxim AI Blog. https://www.getmaxim.ai/blog/mcp-protocol/
"What is MCP and how does it secure code models?" Hugging Face Blog. https://huggingface.co/blog/what-is-mcp-and-how-does-it-secure-code-models
"MCP Overview." Bifrost Documentation. https://docs.getbifrost.ai/mcp/overview
"Govern MCP servers." Bifrost Edge Documentation. https://docs.getbifrost.ai/edge/mcp-governance
"Admin: Approvals." Bifrost Edge Documentation. https://docs.getbifrost.ai/edge/admin-approvals
"Security & guardrails." Bifrost Edge Documentation. https://docs.getbifrost.ai/edge/security
"Audit logs." Bifrost Documentation. https://docs.getbifrost.ai/enterprise/audit-logs
"MCP tool filtering (per virtual key)." Bifrost Documentation. https://docs.getbifrost.ai/features/governance/mcp-tools
"Deploy with MDM." Bifrost Edge Documentation. https://docs.getbifrost.ai/edge/deployment-mdm

What Shadow MCP Is and How to Detect It

claire nguyen — Wed, 24 Jun 2026 18:24:19 +0000

Shadow MCP refers to the unauthorized use of Model Context Protocol servers on organization devices. Bifrost provides the visibility and governance tools needed to identify and secure these connections.

The Model Context Protocol (MCP) is an open standard that allows large language models (LLMs) to access local data, execute code, and query external databases. While this protocol enables highly capable AI agents, it also introduces a new category of security risk known as Shadow MCP. Bifrost, an open-source AI gateway developed by Maxim AI, provides the infrastructure to bridge this visibility gap by centralizing how AI tools interact with local and remote resources.

What is Shadow MCP

Shadow MCP is the usage of MCP servers and tool configurations by employees without the knowledge or authorization of the organization's security and IT teams. It is a subset of "Shadow AI" where users connect their AI chat applications, such as Claude Desktop or ChatGPT, to local tool servers that can read files, access internal APIs, or interact with system processes.

Because MCP servers are often lightweight and easy to run locally, users frequently install them to automate repetitive tasks. However, these servers operate outside the typical enterprise security perimeter. Without a centralized governance layer, security teams cannot see which tools the AI is using, what data is being accessed, or whether those tools are being used to exfiltrate sensitive information.

The Security Risks of Ungoverned MCP Servers

When an employee runs an ungoverned MCP server, they create a direct conduit between a third-party LLM and their company machine. Several technical risks emerge from this architecture:

Data Exfiltration: An MCP server designed to read files could be used by an AI model to summarize and then upload sensitive company documents to an external provider.
Unvetted Tool Execution: Many open-source MCP servers found in public repositories lack rigorous security audits. These servers might contain vulnerabilities or malicious code that can execute commands on the host machine.
Lack of Audit Trails: Most AI desktop applications do not provide detailed logs of which tools were called, what arguments were passed, or what data was returned. This makes incident response nearly impossible.
Credential Exposure: MCP servers often require API keys or environment variables to connect to other services. In a shadow environment, these credentials are stored locally and are often unencrypted.

To mitigate these risks, organizations must move from individual, localized tool configurations to a centralized MCP gateway that provides a single point of enforcement for all AI tool traffic.

How to Detect Shadow MCP on the Endpoint

Detecting Shadow MCP requires visibility into the AI applications running on employee machines. Most organizations have no visibility into the internal configuration files where MCP servers are defined. Detection typically involves three primary methods:

Scanning Local Configuration Files

Many AI applications store their MCP configurations in standard locations. For example, Claude Desktop stores its configuration in a claude_desktop_config.json file. Security teams can use scripts to scan for these files, but this method is often reactive and can be bypassed by users moving files or using different applications.

Network Traffic Analysis

Organizations can monitor for traffic patterns associated with known MCP providers or unexpected outbound API calls. However, because MCP traffic is often encrypted and looks like standard HTTPS traffic to an LLM provider, this method frequently results in false negatives.

Endpoint AI Governance

The most effective way to detect Shadow MCP is through dedicated endpoint agents. Bifrost Edge runs as an always-on agent on macOS, Windows, and Linux devices. It automatically inventories the MCP servers configured inside applications like Claude Desktop, Cursor, and Gemini CLI. This provides a live, fleet-wide inventory of every MCP server running on every machine in the organization.

Governing the Last Mile with Bifrost Edge

Bifrost applies governance and security controls such as virtual keys, budgets, and guardrails centrally, and Bifrost Edge extends that same governance and security to AI traffic on employee machines with endpoint enforcement on each device.

By using the combination of the AI gateway as a control plane and Bifrost Edge as the endpoint layer, administrators can transition from detection to active prevention. The approvals dashboard in Bifrost allows teams to review discovered MCP servers and mark them as approved or denied. Once a server is denied, Bifrost Edge blocks its execution on the device, ensuring that only vetted applications can interact with company data.

Implementation: Fleet-Wide MCP Visibility

For enterprise teams, manual detection is not scalable. Organizations can deploy Bifrost Edge using Mobile Device Management (MDM) platforms such as Jamf, Microsoft Intune, or Kandji. A managed MDM deployment ensures that every computer in the organization is automatically brought under the centralized governance policy.

Once deployed, Bifrost populates the devices dashboard, which lists every host, its owner, and the specific AI applications and MCP servers detected on the machine. This allows security teams to answer critical questions:

Which employees are using experimental MCP servers?
Are there unauthorized coding agents accessing internal source code?
Is there a specific version of a server running that contains a known vulnerability?

Beyond simple blocking, the gateway allows for tool filtering. This means an organization can allow an MCP server but restrict it to specific tools or functions, providing a more granular level of access control.

Building a Secure AI Workflow

Securing the Model Context Protocol requires a shift in how infrastructure is managed. Relying on users to self-report their tool usage is insufficient for modern security standards. By implementing a centralized governance hub and extending it to the endpoint, organizations can enable their teams to use AI tools productively without compromising the security of the corporate environment.

Teams evaluating their AI security posture can request a Bifrost demo to see these governance tools in action or explore the open-source repository to understand the underlying architecture.

Sources

Zero-downtime deploys for our self-hosted LLM gateway

claire nguyen — Wed, 24 Jun 2026 13:22:33 +0000

TL;DR: Getting zero-downtime deploys on a self-hosted LLM gateway comes down to readiness probes that gate traffic, connection draining on shutdown, and running more than one instance behind a load balancer. We did it with Bifrost on EKS.

A rolling upgrade of our self-hosted LLM gateway dropped roughly 0.4% of in-flight requests one afternoon because old pods stopped accepting connections before the replacements passed their readiness checks. We run Bifrost, the open-source AI gateway from Maxim AI, in front of every internal service at Buildkite that calls an LLM, so a botched upgrade hits more than one team at once. This post covers how we got to actual zero-downtime deploys on a self-hosted LLM gateway, the health-check wiring that made it work, and what still bites us.

What zero-downtime means for a self-hosted LLM gateway

Zero-downtime deploys for a self-hosted LLM gateway mean no in-flight request fails while you swap versions. You need readiness probes that gate traffic before a pod joins the pool, connection draining so terminating pods finish their work, and at least two instances behind a load balancer so the pool keeps serving while one node restarts. Miss any of those and you drop requests.

We had the load balancer part sorted from day one. The draining and probe wiring is where we got it wrong, and the failure mode was quiet: a handful of 502s during each deploy that nobody noticed until a downstream team's retry budget started complaining.

Why our first rolling upgrade dropped requests

The bug was the classic Kubernetes shutdown race. When a pod is deleted, kubelet sends SIGTERM and removes the pod from Service endpoints at the same time, but those two events are not ordered against in-flight load balancer routing. The Kubernetes pod termination docs spell out that endpoint removal and SIGTERM happen concurrently, so a request already in flight can still land on a pod that has begun shutting down.

Because Bifrost is a Go binary, it exits fast on SIGTERM, which made the race worse. The process was gone before the load balancer stopped sending it traffic. The fix was a preStop hook to hold the pod open long enough for endpoint propagation, plus a readiness probe so new pods only take traffic once they can actually serve. Running it as a drop-in replacement for the OpenAI base URL meant none of our services needed code changes while we sorted the deploy mechanics.

Rolling upgrades with clustering and load balancing

Here is the deployment shape that stopped the drops. The preStop sleep covers endpoint propagation, and terminationGracePeriodSeconds gives in-flight LLM calls time to finish, since a streaming completion can run for tens of seconds.

spec:
  replicas: 3
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: bifrost
          image: maximhq/bifrost:latest
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            periodSeconds: 5
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "15"]

With maxUnavailable: 0 and maxSurge: 1, the new pod has to pass its readiness probe before an old one leaves. Across three replicas, the request pool never drops below full capacity. Bifrost's automatic fallbacks and load balancing handle provider-side weighting, but the deploy-side availability is plain Kubernetes mechanics that you have to get right yourself.

For multi-node coordination we use Bifrost clustering so config and rate-limit state stay consistent across instances during a rolling swap. On the throughput side, a single instance sustains 5,000 RPS with about 11µs of added latency per request per the published benchmarks, so three replicas gave us plenty of headroom to lose one mid-deploy without backing up the queue.

Observability so we trust the deploy

A deploy you cannot watch is a deploy you cannot trust. We wired Bifrost observability into our existing Prometheus stack and tag every request with a deploy-version label using the x-bf-lh-* metadata headers. The metrics path uses async writes with under 0.1ms overhead, so the logging itself does not skew the latency we are trying to protect during a rollout.

Now each rolling upgrade shows up as a clean version cutover on the dashboard, with the 5xx rate flat through the transition. When something does regress, the per-version labels tell us within one scrape interval whether the new image is the cause. The governance and metrics resources cover the label model in more detail than I will here.

Trade-offs and limitations

Clustering is an enterprise feature, not part of the open-source core, so a single-instance hobby setup will not give you the cross-node state coordination described above. Be honest about that before you plan a multi-node topology.

Self-hosting also means you own the deploy. Bifrost gives you the gateway, but the readiness probes, preStop timing, and grace periods are yours to tune, and the right terminationGracePeriodSeconds depends on your longest streaming completion. The ecosystem is younger than LiteLLM, so you will find fewer random blog posts when you hit an edge case, though the docs covered what we needed. If you want a fully managed control plane instead of running pods, this self-hosted path is more work than it is worth.

Wrapping up

Zero-downtime deploys for a self-hosted LLM gateway are mostly about ordering: readiness before traffic, drain before exit, surplus capacity before you remove a node. Bifrost made the gateway layer fast and boring enough that the only thing left to get right was our Kubernetes wiring. If you want to see the clustering and governance pieces in action, you can book a demo.

Why graceful pod shutdown on EKS kept throwing 502s on deploy

claire nguyen — Wed, 24 Jun 2026 04:21:15 +0000

TL;DR: Graceful pod shutdown on EKS needs a preStop delay plus tuned ALB deregistration, otherwise the load balancer keeps routing to terminating pods and returns 502s mid-rollout. Here's the fix we shipped and how we tested it.

During a routine deploy of our build orchestration API on EKS last month, the ALB logged roughly 1,400 HTTP 502s across the 90 seconds it took to roll three replicas. Nobody got paged, but our availability SLO dashboard showed a clear dip on every deploy, and we deploy a fair few times a day. We run EKS 1.29 with the AWS Load Balancer Controller in IP target mode, and the root cause was a race in graceful pod shutdown that nobody had tuned for. No worries if you've hit this; it's common and the fix is small once you see it.

What graceful pod shutdown actually means on EKS

Graceful pod shutdown is the sequence Kubernetes runs when terminating a pod: it removes the pod from Service endpoints, executes any preStop hook, sends SIGTERM to the container, then SIGKILL once the grace period expires. The pod termination lifecycle is well documented, but the catch on EKS is that endpoint removal inside the cluster and target deregistration at the ALB happen on different clocks.

When the controller runs in IP mode, the ALB sends traffic straight to pod IPs, not through kube-proxy. So the in-cluster endpoint can vanish while the ALB still has that pod IP registered as a healthy target. For a few seconds, the load balancer routes requests at a pod that's already mid-shutdown, and you get 502s.

Why the load balancer keeps sending traffic to terminating pods

The timing gap is the whole problem. The moment a pod enters Terminating, kubelet starts the shutdown clock and the container can begin refusing connections. Meanwhile the ALB only stops sending new requests after it processes the deregistration, and connection draining respects the deregistration delay, which defaults to 300 seconds.

So you have two failure shapes. Either the app stops accepting connections before the ALB has stopped sending them, which is the 502 we saw, or you set the grace period too short and SIGKILL lands while requests are still in flight. The trick is keeping the pod alive and serving until the ALB has actually deregistered it. That's what the preStop hook buys you.

The preStop hook and deregistration delay fix

We added a preStop sleep so the container stays up and keeps serving in-flight requests while deregistration propagates, and we set a grace period longer than that sleep. The values came from watching how long deregistration took during a rollout, which for us was about 10 to 12 seconds.

spec:
  terminationGracePeriodSeconds: 40
  containers:
    - name: orchestrator
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 20"]

The sleep does nothing clever. It just delays SIGTERM so the pod keeps answering during the window where the ALB might still route to it. We also enabled pod readiness gates via the AWS Load Balancer Controller, so a new pod isn't marked Ready until the ALB reports its target as healthy. That closes the matching race on the scale-up side, where a fresh pod takes traffic before it's actually registered.

A cleaner long-term option is handling SIGTERM in the application itself: stop accepting new connections, finish in-flight ones, then exit. We're moving that way, but the preStop sleep got the 502s to zero on the same afternoon, so it shipped first.

Testing it with a game day

I don't trust a fix I haven't watched fail under load, so we ran a small game day. We pointed a constant load generator at the service and triggered a rolling deploy while watching the status codes. A simple loop with hey does the job:

hey -z 120s -c 50 https://orchestrator.internal/healthz
kubectl rollout restart deployment/orchestrator

Before the change, every rollout produced a burst of 502s in the hey summary that lined up exactly with the Terminating events in kubectl get pods -w. After adding the preStop hook and readiness gates, three consecutive rollouts under the same 50-concurrent load showed zero 502s. That's the only evidence I'll accept that graceful pod shutdown is genuinely working, because "never had a bad deploy" usually just means nobody measured one.

Trade-offs and limitations

A preStop sleep is a blunt instrument. It adds fixed latency to every pod termination, so a 20 second sleep across a large deployment slows your rollouts and your scale-down. If you're cost-sensitive on spot capacity, that delay is real money.

It also masks the better fix. Sleeping the container is a workaround for an app that doesn't drain connections on SIGTERM, and if your app holds long-lived connections, a flat sleep won't cover them. Readiness gates help, but they add a dependency on the AWS Load Balancer Controller being healthy, and they make pod startup slower because Ready now waits on ALB registration. Pick the grace period from observed deregistration time, not a number you reckon looks safe.

Where to go next

If your EKS deploys throw 502s, instrument the rollout first: run constant load, watch status codes against pod Terminating events, and measure your actual deregistration time before picking a preStop value. Then decide whether a sleep is enough or whether you want proper SIGTERM draining in the app. The next thing I'd try is replacing the static sleep with application-level connection draining, so the pod exits as soon as in-flight requests finish instead of always waiting the full window.

We made our LLM gateway a single point of failure. Then we tested it.

claire nguyen — Tue, 23 Jun 2026 13:21:51 +0000

TL;DR: We put an LLM gateway in front of about 40 internal services to get failover and one billing view. Then a game day showed the gateway itself was now the thing that took everything down. Here's how we ran two Bifrost replicas, what broke, and where LiteLLM and Portkey were honestly better for us.

Right, so the irony. We added a gateway to stop one flaky provider from taking down our internal tooling at Buildkite. Anthropic 529s, OpenAI timeouts, the usual. The gateway gave us automatic fallbacks and a single place to see spend. Lovely.

What it also gave us was a brand new single point of failure that nothing on our side had been tested against.

How we got here

We run a fair bit of LLM-backed tooling internally. PR summarisers, a flaky-test classifier, log triage. Around 40 services, all of them eventually calling OpenAI or Bedrock.

Every team had its own keys, its own retry logic, its own idea of a timeout. No worries when one service breaks. Real problem when a provider has a bad hour and 40 services all melt down differently.

So we put Bifrost in the middle. One OpenAI-compatible endpoint, automatic fallbacks between providers, and Prometheus metrics we didn't have to build. Spend went to one dashboard. Good outcome.

The catch is obvious in hindsight. Forty services now had a hard dependency on one box.

The game day

We do game days. "Never had an outage" usually means you never tested your failure handling, and I'd rather find the gap on a Tuesday than at 3am.

First run, single replica. We killed the gateway pod. 38 of 40 services started failing within 4 seconds. The two that survived had their own local fallback to a cached response. Everyone else just ate connection-refused.

Lesson one: a gateway you run as one replica is worse than no gateway. You've concentrated the blast radius and added a hop.

So we moved to two replicas behind a Kubernetes Service, with proper probes. Bifrost is stateless for routing, which makes horizontal scaling boring in the good way. Config lives in config.json and env-referenced secrets, so both pods read the same provider setup.

# bifrost-deployment.yaml (trimmed)
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: bifrost
          image: maximhq/bifrost:latest
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /metrics
              port: 8080
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /metrics
              port: 8080
            periodSeconds: 10

Second run, killed one of two pods. Readiness flipped that pod out in about 8 seconds. p99 on the surviving pod jumped from 180ms to roughly 340ms while it carried full load, then settled once Kubernetes scheduled a replacement. No service-level errors. That's the result we wanted.

What still bit us

Killing a pod is the easy test. The nasty one is a slow gateway, not a dead one.

We injected 700ms of latency on one replica using a sidecar. Bifrost stayed "ready" because /metrics answered fine, but real requests crawled. Health checks that only prove the process is alive don't prove it's useful. We ended up adding our own synthetic probe that does a tiny /v1/chat/completions call against a cheap model every 15 seconds and alerts on latency, not just liveness.

The other gotcha was client timeouts. Several services had no timeout at all, so a degraded gateway meant threads piling up. We standardised on a 30s client timeout and let Bifrost's retries and fallbacks handle the provider-side mess.

How it compares

We tested three before committing. All three do the core job. They differ on what happens when you push them.

Concern	Bifrost	LiteLLM	Portkey
Self-hosted HA	Stateless, easy 2+ replicas	Stateless, works fine	Self-host is heavier, more moving parts
Failover config	Per-request fallbacks, native	Solid router fallbacks	Strong, config-driven
Native Prometheus	Yes, built in	Yes	Via their stack
Latency overhead	Low, Go-based	Higher under load in our test	Low, but managed-first
Managed dashboard	Behind enterprise	Lighter	This is their strength

Honest read: if you want a polished managed dashboard with the least ops work, Portkey is genuinely ahead. If you're already deep in Python and want the widest community model coverage, LiteLLM's router is hard to beat and it's been battle-tested by a lot of people. We picked Bifrost because the Go binary held p99 better under our load test and the self-hosted clustering story was the least fiddly for our EKS setup.

Trade-offs and limitations

The gateway is still a dependency. Two replicas reduce risk, they don't delete it. If your config is wrong, both pods are wrong together.

Semantic caching saved us real money on repeated PR-summary calls, but it can serve a stale answer when prompts are near-identical but context differs. Worth tuning the similarity threshold rather than trusting defaults.

Adding the hop costs you something. Our measured median overhead was single-digit milliseconds, which is nothing next to provider latency, but it's not zero and you should measure it on your own traffic, not trust mine.

And clustering with adaptive load balancing sits in enterprise. The open-source path scales horizontally fine for our size, but know where the line is before you plan.

EBS gp2 burst credits ran dry and our builds slowed to a crawl

claire nguyen — Tue, 23 Jun 2026 04:21:28 +0000

TL;DR: A chunk of our EC2 build agents got slow at the same time every afternoon. No CPU pressure, no memory pressure, no network weirdness. It was EBS gp2 burst credits draining to zero, and the fix was a one-line volume type change to gp3 plus a CloudWatch alarm we should've had years ago.

Right, so this one annoyed me for about a week before the penny dropped.

I work on the core compute platform at Buildkite. We run a fleet of EC2 build agents that pick up jobs off a queue and run them. Sydney afternoon, roughly 2pm local, a handful of agents would start dragging. Builds that normally finished in 4 minutes were taking 11. Not all agents. Maybe 15% of the fleet at any given time.

The symptom that lied to us

First instinct was the obvious stuff. CPU? Flat at 30%. Memory? Plenty free. The agent process itself looked bored.

One of our test suites shells out to an LLM step for flaky-test classification, routed through a gateway, so my first paranoid thought was an upstream provider slowdown. We run that traffic through Bifrost so failover and latency are visible per-provider, and the dashboards there were clean. Not the model. Not the gateway. The slow part was local.

So I SSH'd onto a sick agent mid-build and ran iostat -x 1. There it was. %util pinned at 100, await sitting around 80ms, and the volume was doing maybe 100 IOPS when the workload clearly wanted more.

A gp2 volume. 100GB. Baseline 300 IOPS.

Burst credits, the thing nobody remembers

Here's the bit that bites people. A gp2 EBS volume gives you 3 IOPS per GB as a baseline. Our 100GB volumes get 300 baseline IOPS. Below 1TB, volumes earn burst credits and can spike to 3000 IOPS, but only while they've got credits in the bucket.

Those credits refill at the baseline rate. Burn faster than you refill and the bucket empties. When it hits zero you're hard-capped at 300 IOPS until it recovers.

Our build agents do a lot of small random writes. Cloning repos, unpacking caches, npm doing what npm does with its 40,000 tiny files. Early in an agent's life it's got a full credit bucket and flies. After a few hours of back-to-back builds, the bucket's empty. The 2pm pattern wasn't a time-of-day thing at all. It was just agents that had been alive long enough to drain their credits.

You can watch it happen. The metric is BurstBalance, a percentage, and we had zero alarms on it.

aws cloudwatch get-metric-statistics \
  --namespace AWS/EBS \
  --metric-name BurstBalance \
  --dimensions Name=VolumeId,Value=vol-0abc123def456 \
  --start-time 2026-06-20T00:00:00Z \
  --end-time 2026-06-20T06:00:00Z \
  --period 300 \
  --statistics Minimum

Run that against a sick volume and you'll see Minimum walking down toward 0 over the build session. Clean as.

Why gp3 fixes it

gp3 doesn't do the burst-credit dance. You get a flat 3000 IOPS baseline regardless of size, and you can provision up to 16,000 if you pay for it. No bucket, no draining, no surprise cliff at hour three.

It's also cheaper for our shape of workload. gp3 storage is about 20% less per GB than gp2, and the first 3000 IOPS are included.

Volume type	Baseline IOPS (100GB)	Burst behaviour	Predictable under sustained load
gp2	300	Credits to 3000, then cliff	No
gp3	3000 (flat)	None, provision up to 16k	Yes
io2	Provisioned, up to 64k	None	Yes, but pricey

For a build fleet, gp3 is the boring correct answer. io2 is overkill unless you genuinely need tens of thousands of sustained IOPS, and we don't.

The migration is a volume modification, no snapshot dance, no detach:

aws ec2 modify-volume \
  --volume-id vol-0abc123def456 \
  --volume-type gp3 \
  --iops 3000 \
  --throughput 125

We baked it into the launch template so every new agent comes up gp3. Existing volumes got modified in a rolling batch over a maintenance window. p95 build duration on the affected cohort dropped from 9.2 minutes back to 4.1.

Trade-offs and limitations

gp3 isn't free of footguns. The default throughput is 125 MB/s, and if your workload is throughput-heavy rather than IOPS-heavy you'll need to bump that separately, because gp3 decouples the two. We left ours at 125 and it's fine, but I've seen teams forget and wonder why their big sequential reads didn't speed up.

modify-volume also has a cooldown. You can't modify the same volume again for 6 hours, so if you fat-finger the IOPS number you're waiting it out. Plan the values before you run it.

And this didn't make our agents infinitely fast. It removed an artificial cliff. If a build is genuinely I/O bound at 3000 IOPS, gp3 buys you headroom, not magic. The real long-term fix for us is shrinking what we write to disk per build, which is a slower piece of work.

Last thing. Burst credits exist on other AWS resources too, with different names and different cliffs. T-series instance CPU credits. NAT gateway. If you've got one burst-credit surprise in your stack, you've probably got more hiding.

Semantic caching our flaky-test summariser: 58% fewer LLM calls

claire nguyen — Mon, 22 Jun 2026 13:22:33 +0000

TL;DR: Our internal flaky-test summariser at Buildkite was firing ~40k LLM calls a day, and most were near-duplicates of failures we'd already explained. Switching on semantic caching in Bifrost cut live provider calls by 58% and dropped p50 latency on cache hits from ~900ms to about 40ms. It also kept the feature alive when our primary provider browned out for 11 minutes.

The feature that wouldn't shut up

On our platform team (eight of us) we shipped a small thing last quarter: when a test goes flaky in a Buildkite pipeline, we pass the failure output to an LLM and stick a plain-English summary on the build page. Devs liked it. The provider bill less so.

By March it was making roughly 40,000 calls a day against anthropic/claude-haiku, with openai/gpt-4o-mini as the fallback. p50 latency sat around 900ms. The monthly bill crept past $310. Not catastrophic. But the calls were doing the same work over and over.

Why the calls were so repetitive

Here's the bit that bugged me. Flaky tests are flaky for the same reasons across builds. A timeout in payments_spec.rb looks almost identical on Tuesday as it did on Monday, minus a timestamp and a container ID.

So we were paying full freight to summarise text we'd already summarised. Different bytes, same meaning. A normal key-based cache misses all of these because the strings never match exactly. That's the whole problem semantic caching solves: it matches on meaning, not on an md5 of the prompt.

We already ran everything through Bifrost as our gateway, mostly for the automatic failover. Turns out the semantic caching was sitting right there.

Turning it on

Bifrost runs as a single Go binary in front of our summariser. We added the cache plugin to the gateway config and pointed it at a small embedding model so we weren't paying much per lookup.

{
  "plugins": [
    {
      "name": "semantic_cache",
      "config": {
        "embedding_model": "openai/text-embedding-3-small",
        "threshold": 0.92,
        "ttl_seconds": 86400
      }
    }
  ]
}

The threshold is the knob that matters. At 0.92 cosine similarity two failures have to be genuinely close before we serve a cached summary. We started at 0.97, which was too strict (hit rate sat around 20%), and walked it down while spot-checking summaries against the real failures.

Settled on 0.92. Cache hit rate landed at 58% over the first three weeks. On a hit, the summariser returns in ~40ms instead of waiting on a provider round trip. No code change in our app, since Bifrost speaks the same OpenAI-compatible API we already called.

What the brownout taught us

Two weeks in, our primary provider had a rough afternoon. Elevated errors and timeouts for 11 minutes. Normally Bifrost's fallback kicks the traffic to the secondary, which it did.

But the cache did something I hadn't planned for. More than half the requests during that window never reached either provider, because they matched recent failures already in the cache. The blast radius shrank on its own. The fallback handled the genuinely new failures, the cache absorbed the repeats, and nobody filed a ticket. She'll be right, basically.

That's the reliability angle people miss with caching. It's not only a cost lever. It's load shedding you get for free when an upstream goes wobbly.

Bifrost vs LiteLLM vs Portkey

We looked at the obvious alternatives before committing. All three can do semantic caching. They're not the same.

Capability	Bifrost	LiteLLM	Portkey
Semantic cache	Built in, config-driven	Yes, via Redis + embeddings	Yes, mature
Failover + cache together	Single binary	Proxy + Redis to wire up	SaaS, polished
Self-host	Go binary, Docker	Python proxy	Self-host or cloud
Dashboard	Built-in web UI	Community UI	Strongest of the three
Provider breadth	23+	Very broad	Broad

Honest read: LiteLLM has the bigger community and the widest provider list, and if you already run Redis their cache is well-trodden. Portkey's dashboard and analytics are the slickest of the lot, and for a team that wants a managed SaaS it's hard to argue against.

We picked Bifrost because we self-host on ECS and wanted the failover and the cache in one Go process, not a Python proxy plus a Redis we'd have to babysit. Fewer moving parts to break on a game day.

Trade-offs and Limitations

Semantic caching isn't free of sharp edges, and pretending otherwise would be daft.

The threshold is a real risk. Set it too loose and you'll serve a summary from a different failure that happens to read similarly. We caught two of these at 0.88 during tuning. A bad summary on a build page erodes trust fast, so we erred conservative at 0.92 and accept a lower hit rate for it.

Embeddings add a little latency and cost on every lookup, including misses. With text-embedding-3-small it's small, but it's not zero. For workloads where every input is genuinely unique, you'll pay the embedding tax and get almost nothing back.

Cache invalidation is on you. When we changed the summariser's prompt, every cached entry was suddenly stale against the new format. We dropped the TTL to 24 hours so the cache rolls over daily rather than holding stale shapes for a week.

And it doesn't replace failover. The cache helped during the brownout, but only because we had recent traffic. Cold cache plus dead provider equals a bad time. Keep your fallback chain regardless.

What is shadow AI and how to govern it

claire nguyen — Mon, 22 Jun 2026 06:14:58 +0000

Shadow AI is the use of AI tools inside an organization without IT's knowledge or approval. This guide explains what it is, why it creates real security and compliance risk, and how the Bifrost AI gateway together with Bifrost Edge brings that usage under governance on every machine.

Most of the AI tools employees rely on at work run on their own machines and reach a model provider directly, without passing through any corporate network checkpoint. A developer can install a desktop assistant, paste in proprietary source code, and send it to a third-party model before anyone in security knows the tool exists. Industry analysts call this pattern shadow AI: the use of AI tools or applications by employees without the approval or oversight of the IT department. The scale is no longer marginal, as a 2026 Gartner survey of cybersecurity leaders found that 69 percent have evidence or suspect that employees are using public generative AI at work.

What shadow AI is

Shadow AI is the use of AI tools, models, and services by employees without the knowledge, approval, or governance of an organization's IT or security teams. It is a subset of shadow IT, the broader category of hardware and software that IT has not approved, but it carries risks that older shadow IT controls were never designed to handle. The distinction matters for how an organization should respond.

Where shadow IT generally involves an unapproved application or storage service, shadow AI centers on systems that process, generate, and retain data in ways that are difficult to reverse. Common forms include:

Public chatbots and assistants used on work data, such as the ChatGPT app or Claude Desktop signed in with a personal account.
AI features inside browser tabs and SaaS products that an employee turns on without review.
Coding agents in the terminal and IDE, including Claude Code, Codex, and Cursor, that read source code and call external services.
MCP servers wired into those tools, which can read files, call APIs, and act on a user's behalf.

Salesforce's 2026 Workforce AI Survey found that 67 percent of employees use AI at work, while only 18 percent of organizations have a formal AI security policy. Adoption at that pace, with no governing layer underneath it, is what turns ordinary productivity into a security exposure.

Why shadow AI is a security risk, not just a policy gap

Shadow AI raises measurable security and compliance risk because sensitive data reaches systems that the organization cannot see, control, or audit. Gartner predicts that by 2030, more than 40 percent of organizations will experience security or compliance incidents tied to the use of unauthorized AI, and the reasons follow directly from how these tools are used.

The exposure goes well beyond a single pasted prompt. Several distinct failure modes make shadow AI harder to contain than earlier forms of unapproved software:

Data leaves the organization in a form that cannot be recalled. Once proprietary text enters a third-party model, it may be retained or used in ways the organization has no ability to reverse, unlike a file that can be deleted from a server.
Compliance and audit trails break down. Legal and security teams cannot demonstrate where regulated data went, whether retention rules were followed, or whether residency obligations were met when the traffic never passed through a governed path.
AI agents inherit standing access. An assistant connected through an MCP server can read email, repositories, and internal systems on a continuing basis, so the question shifts from what a user pasted once to what a connected tool can reach at any time.
Governance trails adoption. Most organizations still have no reliable way to see which AI tools and connections are in use, which leaves the bulk of this activity outside any policy or review.

These concerns are not hypothetical; they describe what happens when fast, employee-driven adoption runs ahead of any mechanism for seeing or controlling it.

Why existing controls miss shadow AI

Traditional network controls miss most shadow AI because the activity does not behave like the traffic those controls were built to inspect. Network proxies and data loss prevention systems observe what crosses the corporate network, yet a large share of AI usage runs on the endpoint and connects straight to a provider over an encrypted channel that resembles ordinary web traffic.

Three gaps recur across the older approaches:

Network filtering and data loss prevention operate at the perimeter, so AI requests that originate and resolve on the device fall outside their view.
Blocklists depend on a known list of destinations, and new AI tools, browser features, and MCP servers appear faster than any list can track.
Written policies state what employees should do, but a document does not enforce anything at the moment a prompt is sent.

The common thread points toward the fix: the AI runs on the endpoint, where the person and the tool actually meet, so the endpoint is the one place where every request can be seen and governed before data leaves the machine.

How the Bifrost AI gateway with Bifrost Edge govern shadow AI

Governing shadow AI well takes two things that fit together: one place to define policy, and a way to apply that policy to the AI running on every machine. Bifrost, the open-source AI gateway built by Maxim AI, is that one place. The gateway already holds the virtual keys, budgets, and rate limits that tie AI usage to a person or project, the guardrail profiles that inspect prompts and responses, and the audit logs that record every exchange. The limitation, until now, has been reach: those controls governed only the traffic that someone had configured to point at the gateway.

Bifrost Edge closes that gap by running on each machine and routing all supported AI traffic through Bifrost, so the same virtual keys, budgets, guardrails, and audit logs that already protect gateway traffic now apply to the desktop apps, browser AI, and coding agents people use day to day. The gateway stays the single control plane, and Edge becomes its reach to the endpoint, so there is no second policy model to build or maintain.

A request from any supported AI tool follows the same governed path on every machine:

A user works in a desktop app, a browser AI surface, or a coding agent exactly as before, with no base URL change and no SDK swap.
Bifrost Edge routes that request through the organization's Bifrost rather than letting it go straight to the provider.
Bifrost ties the request to the user's virtual key and its budget, runs the configured guardrails, and writes the exchange to the audit log.
The governed response returns to the original app, with sensitive content already caught or redacted.

Guardrails apply before data leaves the machine

The guardrail profiles configured in Bifrost apply to endpoint traffic with no extra setup on the device. A guardrail runs before a prompt reaches a model and again before a response returns, so secrets and personal data are caught or redacted before they leave the machine. Built-in coverage includes Gitleaks-backed secrets detection for leaked API keys, tokens, and credentials, a PII detection template built on custom regex, and content safety, alongside integrations with AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, CrowdStrike AIDR, GraySwan Cygnal, and Patronus AI.

Visibility into MCP servers across the fleet

Most organizations cannot say which MCP servers their employees have connected to AI tools. Bifrost Edge inventories the MCP servers configured inside each supported app and builds a live picture across the fleet of which servers are in use, on which apps, and on how many devices. Administrators then allow or deny each server individually, and Edge enforces that decision on the device, even for an app that had the server configured before the policy existed. MCP discovery covers the major AI apps that support it, including Claude Code, Claude Desktop, Gemini CLI, OpenCode, Codex, and Cursor.

App policy enforced on every device

App governance lets administrators decide which AI applications are permitted across the organization. Approved apps run normally, with their traffic governed through Bifrost, while disallowed apps are blocked before any data leaves the machine. When Edge encounters an app or MCP server it has not seen, it requests approval from the admin console, and administrators choose whether pending items are allowed or blocked while a decision is pending. Policy changes reach the whole organization at once, without anyone revisiting individual machines.

Rollout through your existing device management

Bifrost Edge deploys through the device management platforms an organization already runs, including Jamf, Microsoft Intune, Kandji, Omnissa Workspace ONE, and JumpCloud, across macOS, Windows, and Linux. The managed configuration carries only the connection settings that point each machine at the organization's Bifrost, and identity and keys come from the user's single sign-on, so no secrets sit on the device. After the first sign-in, governance stays in sync with the gateway, and central changes to app policy, MCP allow and deny lists, and routing reach the fleet on their own.

Common questions about shadow AI

Is shadow AI the same as shadow IT?

Shadow AI is a subset of shadow IT, but the risk profile differs. Shadow IT covers any hardware or software that IT has not approved, whereas shadow AI specifically involves tools that process and retain data in a model, which makes the exposure harder to reverse and more likely to spread across teams.

Can shadow AI be detected?

Shadow AI can be detected when AI requests are observed at the point where they originate. Because much of the usage runs on the endpoint, a layer that operates on the device, such as Bifrost Edge, can inventory the apps and MCP servers in use and route their traffic through a gateway where it becomes visible and auditable.

How do you govern AI coding agents?

Coding agents such as Claude Code, Codex, and Cursor run locally and often connect directly to model providers and MCP servers. Routing their traffic through Bifrost applies the same guardrails, budgets, and audit logging used for the rest of an organization's AI, while app and MCP policies determine which agents and tools are allowed on each machine.

Where this leaves enterprise teams

Shadow AI persists because the activity happens on the endpoint and moves faster than perimeter controls can follow, so better intentions and longer policy documents do not resolve it on their own. The organizations that handle it well treat it first as a visibility problem and then as an enforcement problem, governing AI where people actually use it rather than where the network happens to see it.

Pairing the Bifrost AI gateway with Bifrost Edge gives security and platform teams one control plane for that work, with the gateway defining the virtual keys, budgets, guardrails, and audit logs, and Edge, currently in alpha, extending them to every machine in the organization. Teams sizing up shadow AI can review how the combined approach works on the Bifrost Edge overview and register there for alpha access.

DEV Community: claire nguyen

Best Tools to Block Sensitive Data From Reaching Public LLMs

The Challenge of Sensitive Data in AI Applications

Key Features of Sensitive Data Protection Tools for LLMs

Top Tools for Protecting Sensitive Data with LLMs

Bifrost

Cloudflare AI Gateway

Kong AI Gateway

Google Cloud DLP

How to Choose the Right Solution for Your Organization

Sources

Async LLM inference in CI: stop build workers blocking on slow jobs

What async inference actually does

Why synchronous LLM calls stall a build fleet

Wiring async submit and poll

Keeping costs and failures visible

Trade-offs and limitations

Wrapping up

Further reading

How to See Every MCP Server Developers Have Configured

The Challenge of Ungoverned MCP Servers and Shadow AI

What is an MCP Server?

Bifrost Edge: Gaining Visibility into Endpoint MCP Usage

From Visibility to Control: Approving and Denying MCP Servers

Deployment and Integration for Comprehensive Governance

Sources

Governing MCP Servers on Employee Devices: A Critical Step for Enterprise AI

What are MCP Servers and Why Do They Matter?

The Rise of Ungoverned AI Tool Usage (Shadow AI)

How Bifrost Edge Addresses MCP Server Governance

Granular Control with Centralized Policy

Streamlined Deployment for Endpoint AI Governance

The Broader Context: Comprehensive AI Governance

Sources

Securing MCP Servers Across a Company: A Practical Guide

Understanding the MCP Server Security Challenge

How Bifrost Addresses MCP Server Security

Comprehensive MCP Server Governance with Bifrost Edge

Deploying Bifrost Edge for Fleet-Wide Governance

Conclusion

Sources

What Shadow MCP Is and How to Detect It

What is Shadow MCP

The Security Risks of Ungoverned MCP Servers

How to Detect Shadow MCP on the Endpoint

Scanning Local Configuration Files

Network Traffic Analysis

Endpoint AI Governance

Governing the Last Mile with Bifrost Edge

Implementation: Fleet-Wide MCP Visibility

Building a Secure AI Workflow

Sources

Zero-downtime deploys for our self-hosted LLM gateway

What zero-downtime means for a self-hosted LLM gateway

Why our first rolling upgrade dropped requests

Rolling upgrades with clustering and load balancing

Observability so we trust the deploy

Trade-offs and limitations

Wrapping up

Further reading

Why graceful pod shutdown on EKS kept throwing 502s on deploy

What graceful pod shutdown actually means on EKS

Why the load balancer keeps sending traffic to terminating pods

The preStop hook and deregistration delay fix

Testing it with a game day

Trade-offs and limitations

Where to go next

Further reading

We made our LLM gateway a single point of failure. Then we tested it.

How we got here

The game day

What still bit us

How it compares

Trade-offs and limitations

Further Reading

EBS gp2 burst credits ran dry and our builds slowed to a crawl

The symptom that lied to us

Burst credits, the thing nobody remembers

Why gp3 fixes it

Trade-offs and limitations