AI Enterprise Governance: Implementing Runtime Controls at the AI Gateway

Why static security policies fail for generative AI, and how Bifrost enforces real-time budgets, guardrails, and role-based access across your entire model footprint.

If you have been following my journey through local AI engines, terminal coding assistants, and the world of Model Context Protocol (MCP), you know how fast this ecosystem moves.

What has been happening in the past few weeks, though, is changing once and for all the public alert sentiment on the risks of running AI agents. We saw data leaks, encryptions broken, deleted production database and privacy concerns.

The attention and alert level has never been so high: in fact, once an organization moves past individual experimentation and starts deploying agents across dozens of departments, a new bottleneck emerges.

It is no longer a matter of making agents work. It becomes a game of keeping them inside safe operational boundaries.

We need new rules for the AI agency storm

A few weeks ago, I spoke with an IT director at a financial technology firm. Their engineering team had deployed internal coding assistants, customer support agents, and document search pipelines across three separate cloud environments.

On paper, they had configured traditional cloud firewalls, issued shared corporate API keys, and required single sign-on for their cloud console.
In practice, their operational reality was chaotic.

Two developers had embedded a master Anthropic API key into a staging service to debug a pipeline over the weekend. A background agent loop encountered an unhandled exception, retried a failed prompt thousands of times, and consumed thousands of USD in credits in under three hours.
Meanwhile, a compliance audit revealed that a customer-facing agent had received prompt payloads containing unredacted client financial records, which were subsequently sent to an unapproved external provider endpoint.

This story is becoming common across software teams.

Traditional perimeter security relies on static rules: IP allowlists, single API keys, and periodic access reviews.

Generative AI traffic is dynamic, non-deterministic, and frequently automated through autonomous agent loops. Static rules cannot inspect fluid prompt contexts, enforce real-time financial caps per consumer, or prevent an autonomous agent from invoking an unauthorized database action mid-stream.

To secure generative AI workflows at scale, governance must move directly into the execution path. 

It must operate as a zero-trust runtime control plane that inspects, routes, redacts, and governs every single token before it leaves your network.

This is where the enterprise security layer of the Bifrost LLM + MCP Gateway comes in.

💡 Note: this article is about Enterprise LLM governance, but these concepts and tools appl also to every level of AI governance, even our home-labs.

Virtual Keys: the core identity unit for every AI request

For years, the standard approach to managing model access was simple: create an account with a provider, generate an API key, copy it into an environment file, and share it across your application cluster.

In an enterprise environment, this pattern is a major security vulnerability.

When ten microservices share a single upstream API key, you lose all operational visibility. You cannot determine which service exceeded a rate limit, which user triggered a compliance alert, or how much budget a specific customer project consumed.

💡 For curiosity: browse GitHub and look how many persona project you can find about AI governance and API keys monitoring. There have been hundreds of attempts to avoid this kind of non-traceability.

Bifrost replaces shared provider keys with a core governance abstraction: Virtual Keys (bf-virtual-key).

[ Application / Developer Agent ]
               │
               ▼  (bf-virtual-key-engineering-01)
     ┌───────────────────┐
     │  Bifrost Gateway  │ ──► Evaluates: Identity, Budgets, Rate Limits, Guardrails
     └───────────────────┘
               │
               ▼  (Provider Key safely pulled from Vault)
    [ Upstream LLM Provider ]

A Virtual Key is a scoped, per-consumer identity issued to a specific application, department, developer, or external customer integration.

Upstream provider credentials (such as your actual OpenAI, Anthropic, or Google Vertex keys) remain encrypted inside the gateway or your enterprise secret manager.

Applications never touch real provider keys. They authenticate to Bifrost using their assigned Virtual Key.

Hierarchical Budget Controls

One of the most immediate challenges in managing AI infrastructure is preventing unexpected billing spikes.

Autonomous agent loops can rapidly consume tokens if they enter infinite execution cycles.

Bifrost addresses this by implementing hierarchical budget controls directly within the Virtual Key framework. You can define financial ceilings at three distinct layers:

1. Virtual Key Level: Assign hard or soft spend caps to individual developers, staging microservices, or terminal agents.
2. Team Level: Aggregate spending across multiple Virtual Keys belonging to a specific department, such as Engineering, Marketing, or Customer Support.
3. Customer Level: Enforce tenant-level spend limits for multi-tenant software platforms that offer AI features to end users.

If a Virtual Key exhausts its hourly or monthly budget, Bifrost handles the event based on your policy.

You can configure the gateway to block further requests for that key, drop a notification into your monitoring pipeline, or automatically shift traffic to a lower-cost fallback model (such as a local llama.cpp instance or a self-hosted model running on vLLM).

Dual-Axis rate limits

Financial budgets prevent long-term overspending, but they do not protect against sudden, high-burst API requests that trigger provider rate limit errors.

To address this, Bifrost enforces rate limits across two distinct axes for every Virtual Key:

• Requests Per Minute (RPM): Controls the overall volume of calls to prevent API spamming.
• Tokens Per Minute (TPM): Regulates token volume to prevent large context windows from saturating upstream provider throughput quotas.

When you enforce these limits at the gateway layer, you can prevent a single runaway process from degrading performance for the rest of your organization.

Identity Sync, Fine-Grained RBAC, and Vault Secrets

Managing hundreds of Virtual Keys manually as an organization scales quickly becomes impractical. Enterprise security teams require central identity management and automated provisioning.

Bifrost connects directly with existing enterprise identity systems, linking corporate user directories to AI model access.
Here below a simplified diagram for the process:

┌───────────────────────────┐
│  Okta / Microsoft Entra   │ (OIDC / SSO Sync)
└─────────────┬─────────────┘
              │
              ▼
┌───────────────────────────┐
│      Bifrost Gateway      │
│  - Identity Mapping       │
│  - Custom RBAC Roles      │
│  - Vault Secrets Fetch    │
└─────────────┬─────────────┘
              │
              ├──────► [ HashiCorp Vault ]
              ├──────► [ AWS Secrets Manager ]
              ├──────► [ Azure Key Vault ]
              └──────► [ Google Secret Manager ]

SSO and OIDC Identity Providers

Bifrost supports OpenID Connect (OIDC) and Single Sign-On (SSO) integration with major enterprise identity providers, including Okta and Microsoft Entra ID (formerly Azure AD).

When an engineer launches a terminal coding agent or logs into a desktop client through Bifrost Edge, their corporate identity is verified against your primary directory. User group memberships are automatically synchronized, assigning appropriate access policies without manual administrative intervention.

Fine-Grained Role-Based Access Control (RBAC) and Data Access Control

Authentication identifies who the consumer is; authorization determines what they are permitted to do.
Bifrost provides fine-grained Role-Based Access Control (RBAC) coupled with Data Access Control (DAC) through configurable Access Profiles.

Instead of granting global access to all connected models and data pipelines, administrators define granular permission policies:

• Model Access Restrictions: A junior developer profile can be restricted to fast, lightweight models for daily tasks, while senior staff or automated release pipelines are authorized to invoke frontier models.

• Feature Restrictions: Certain roles can be permitted to run standard text completions while being blocked from executing raw code interpreters or file system tools.
• Data Access Control (DAC): Access Profiles restrict which internal knowledge bases, database indexes, or document stores a specific user group can query through the gateway.

Zero-Plaintext Secret Management via External Vaults

Storing raw provider keys in plain text inside environment variables or static configuration files is a major compliance risk for regulated industries.
Bifrost integrates natively with leading enterprise secret managers:

• HashiCorp Vault
• AWS Secrets Manager
• Google Secret Manager
• Azure Key Vault

When Bifrost routes a request to an upstream provider, it fetches the necessary credentials dynamically from your secure vault at runtime. API keys are decrypted in memory, used to execute the request, and never written to disk or exposed in user-facing configuration dashboards.

In-Flight Guardrails and Policy-Driven Routing

Securing the perimeter and managing identities is only half the battle.
Because generative AI outputs are variable, security policies must also inspect payload contents as traffic flows through the gateway.

Bifrost implements In-Flight Guardrails that act as real-time filters on both incoming prompts and outgoing model responses.

Real-Time Validation and Redaction

Guardrails in Bifrost operate directly in the request-response path, introducing negligible latency while executing multi-stage safety checks:

• PII and Sensitive Data Redaction: Integrating with detection engines like Microsoft Presidio and Azure AI, Bifrost identifies sensitive data - such as Social Security numbers, credit card details, medical record identifiers, and custom internal project code names - and redacts or hashes them before the prompt leaves the internal network.

• Secrets Detection: The gateway scans outgoing payloads for leaked credentials, private RSA keys, AWS access tokens, or internal database connection strings, blocking requests that contain exposed secrets.
• Schema Enforcement: For structured agent workflows, Bifrost validates JSON responses against predefined schemas. If a model generates malformed outputs or unexpected fields, the gateway can request a correction or drop the payload before it reaches downstream production systems.
• Custom Policy Rules: Organizations can define custom Common Expression Language (CEL) assertions to enforce business-specific safety compliance, such as blocking prompts containing competitive benchmarking requests or unvetted external URLs.

Policy-Driven Routing and Approved Model Enforcement

In regulated environments, allowing applications to route requests to arbitrary external model endpoints creates significant legal exposure.

Bifrost enables administrators to write strict routing policies that dictate exactly which models can process specific classes of data:

{
  "policy_name": "enforce-compliant-routing",
  "rules": [
    {
      "condition": "context.data_classification == 'PII_SENSITIVE'",
      "allowed_providers": ["in-vpc-vllm-cluster", "azure-hipaa-instance"],
      "action": "PERMIT"
    },
    {
      "condition": "context.data_classification == 'PUBLIC'",
      "allowed_providers": ["openai-gpt4o", "anthropic-claude"],
      "action": "PERMIT"
    }
  ]
}

If an application attempts to send sensitive internal data to an unapproved public endpoint, Bifrost intercepts the request, blocks the call, and logs a compliance violation event.

If an approved primary endpoint becomes unavailable or hits a rate limit, the policy router automatically redirects the request to a designated compliant fallback provider.

MCP Tool Scoping and Endpoint Extension

Model Context Protocol (MCP) allows language models to interface with external tools, databases, and APIs. Many Agent Harness use them: they have an unified standard protocol, so you can use them in Claude Code, Codex OpenCode and all others without changing a thing.

However, giving an agent access to an MCP server without granular boundaries creates serious operational risk.
If an agent is connected to a database server, you must ensure it can execute read queries without having the authority to execute drop or truncate operations.

Bifrost enforces tool scoping at the Virtual Key level using MCP Tool Filtering.

                           ┌───────────────────────────┐
                           │      Bifrost Gateway      │
                           └─────────────┬─────────────┘
                                         │
                 ┌───────────────────────┴───────────────────────┐
                 ▼                                               ▼
  [ Virtual Key: Customer Support ]             [ Virtual Key: Systems Admin ]
  - Filtered Tools:                              - Unrestricted Tools:
    ✓ crm_lookup_customer                          ✓ crm_lookup_customer
    ✓ crm_get_order_history                        ✓ crm_update_record
    ✗ crm_delete_customer                          ✓ crm_delete_customer

When an agent requests the list of available tools, Bifrost intersects the upstream MCP server's tool catalog with the Virtual Key's allowed tool profile. Unauthorized tools are stripped from the response context entirely.

The model remains unaware that restricted tools exist, removing the possibility of accidental or malicious invocation.

Bifrost Edge: Extending Governance to the Endpoint

Centralized gateway governance is highly effective for cloud-hosted applications, but developer laptops represent a major unmanaged perimeter.

When an engineer runs an AI-assisted IDE or terminal tool locally, they might configure the tool to bypass central gateway rules.
Bifrost Edge extends these runtime governance controls directly down to the physical endpoint.

Running as a lightweight agent in the background on macOS, Windows, or Linux, Bifrost Edge automatically detects local AI traffic generated by desktop applications, browser tabs, or terminal harnesses.

It routes that traffic through your central Bifrost Gateway without requiring developers to manually edit configuration files or manage local environment keys.

Virtual Key restrictions, budget caps, PII redaction rules, and MCP tool filters are applied to local desktop activities just as strictly as they are to production cloud pipelines.

Immutable Audit Logs, Data Lakes, and Telemetry

In regulated industries governed by standards like SOC 2, GDPR, HIPAA, and ISO 27001, security policies must be verifiable through audit records.

Saying you have security policies is not enough; you must be able to prove every action taken by every model and agent across your infrastructure.

Bifrost provides a comprehensive telemetry and logging framework designed to meet enterprise compliance standards.

Cryptographically Traceable Audit Trails

Every request passing through Bifrost generates an immutable audit log entry. Each log record captures:

• The originating Virtual Key, authenticated user identity, and team assignment
• The target provider, exact model version, and routing path selected
• Prompt token counts, completion token counts, total latency, and precise financial cost
• Intercepted guardrail events, including redacted PII elements or blocked policy violations
• Exact parameters and execution results for all invoked MCP tools

Content logging can be selectively masked or encrypted per environment to ensure that sensitive user inputs are never stored in plain text within log storage.

Data Lake and Storage System Exports

For long-term compliance archiving and security analysis, Bifrost streams audit records directly to enterprise data lakes and storage systems.

Built-in streaming adapters support direct exports to Amazon S3, Google Cloud Storage, BigQuery, Apache Kafka, and Google Cloud Pub/Sub. Security teams can query historical AI operations using their existing SIEM tools to detect anomalous usage patterns or build custom compliance reporting dashboards.

Native Observability Integrations

Operational monitoring is critical for maintaining infrastructure health. Bifrost exposes deep system telemetry out of the box:

• OpenTelemetry (OTel): Full support for distributed tracing, allowing you to trace a user request as it moves through frontend applications, the Bifrost Gateway, internal vector databases, and external model endpoints.
• Prometheus Metrics: Standard metrics endpoints exposing request rates, error codes, token usage velocity, and microsecond-level latency histograms.
• Datadog Connector: Pre-built integration dashboards providing instant visibility into model consumption, cost distribution across Virtual Keys, and active guardrail blocks.
• Maxim Observability Suite: Deep integration with Maxim's enterprise platform for advanced evaluation, regression testing, and prompt monitoring.

In-VPC Deployment and High-Availability Clustering

Enterprise security policies often prohibit sensitive data from traversing public proxy layers or multi-tenant management systems.

Bifrost is designed to run entirely within your secure corporate perimeter.

┌────────────────────────────────────────────────────────────────────────┐
│                        Virtual Private Cloud (VPC)                     │
│                                                                        │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │                    Bifrost Cluster (In-VPC)                    │   │
│   │                                                                │   │
│   │   ┌─────────────┐       ┌─────────────┐       ┌─────────────┐  │   │
│   │   │ Gateway N1  │ ◄───► │ Gateway N2  │ ◄───► │ Gateway N3  │  │   │
│   │   └─────────────┘       └─────────────┘       └─────────────┘  │   │
│   │          ▲                     ▲                     ▲         │   │
│   └──────────┼─────────────────────┼─────────────────────┼─────────┘   │
│              │                     │                     │             │
│   ┌──────────┴────────┐   ┌────────┴─────────┐   ┌───────┴─────────┐   │
│   │ Self-Hosted vLLM  │   │ Encrypted Vault  │   │ Local Audit Log │   │
│   └───────────────────┘   └──────────────────┘   └─────────────────┘   │
└────────────────────────────────────────────────────────────────────────┘

Air-Gapped and Regulated Architecture

You can deploy Bifrost directly into your private cloud (AWS VPC, Google Cloud VPC, Azure VNet) or air-gapped on-premises environments.

In this architecture, zero telemetry, configuration data, or prompt payloads leave your controlled network space. 

Internal data encryption uses AES-256-GCM with keys derived via Argon2id, protecting cached parameters and key configurations at rest.

High-Throughput performance and active-active Clustering

Security controls must not create performance bottlenecks. Written in pure Go, Bifrost introduces less than 15 microseconds of execution overhead per request under typical workloads.
To support large enterprise operations, Bifrost scales horizontally in active-active cluster configurations:

• State Synchronization: Distributed nodes synchronize state, active rate limits, and budget tracking in memory using lightweight synchronization protocols.
• High Availability: If a cluster node fails, traffic is redistributed seamlessly without dropping active client connections or resetting rate limit counters.
• Scalability: Benchmarked at over 5,000 requests per second (RPS) per node, a Bifrost cluster can handle heavy enterprise traffic loads while maintaining sub-millisecond routing decisions.

To see how runtime gateway governance differs from traditional API management, consider this side-by-side comparison:

Action Plan: implementing runtime Governance in 10 Minutes

If you want to move your organization from static API keys to runtime governance, you can test these controls locally today before rolling them out to your cloud infrastructure.

Step 1: Fire up the Local Gateway

Open your terminal and launch the gateway:

npx -y @maximhq/bifrost

The administration dashboard will open at http://localhost:8080.

Step 2: Configure a Provider via Secret Manager

In the dashboard, navigate to Model Providers. Instead of entering a raw API key in plain text, select your enterprise secret manager backend (such as HashiCorp Vault or AWS Secrets Manager) and provide the secret path string.

Step 3: Create Your First Virtual Key and Budget Policy

Navigate to Virtual Keys and select Create Virtual Key:

• Name the key engineering-coding-agents.
• Set a maximum monthly spend limit of $200.00.
• Set a rate limit of 100 Requests Per Minute and 100,000 Tokens Per Minute.
• Assign a fallback route pointing to a local or secondary model if the primary budget is exceeded.

Step 4: Enable In-Flight PII Redaction

Under the Guardrails tab, toggle on the PII Masking Profile. Select the entities you wish to redact (e.g., credit card numbers, email addresses, explicit secret keys).

Step 5: Test a Request Using the Virtual Key

Send a test request using cURL, passing your newly created Virtual Key in the header:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer bf-virtual-key-engineering-coding-agents" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "user", "content": "Hello, my secret token is AWS_TOKEN_123456789. Please optimize this query."}
    ]
  }'

Check your Bifrost audit dashboard. You will see that the request was logged under the engineering-coding-agents key, the sensitive token was redacted in flight, the financial cost was deducted from the key's budget, and the metrics were streamed to your local telemetry pipeline.

---

Conclusion: The Shift to Zero-Trust AI Infrastructure

The rapid adoption of generative AI has gone beyond traditional perimeter security models. Treating model access as a simple matter of handing out API keys is no longer viable for enterprise engineering teams.

Bifrost transforms how organizations manage model infrastructure: with the simple rule of placing runtime controls directly at the gateway layer.

You gain total visibility into model operations, protect internal systems from data leakage, prevent run-away financial costs, and enforce fine-grained role-based access across every model, tool, and endpoint in your enterprise network.
Security does not have to come at the expense of developer speed. 

With sub-15 microsecond latency, zero-config endpoint routing via Bifrost Edge, and native integrations with the enterprise identity and secret management tools you already rely on, Bifrost provides a clear path for enterprise AI deployment.

---

Have questions about setting up OIDC mapping, configuring custom guardrail policies, or deploying Bifrost clusters inside your VPC? Leave a comment below or join the discussion!