AI Guardrails in the LLM Request Path with Bifrost

#security #ai #devops #go

Learn how Bifrost enforces AI guardrails in the LLM request path to catch leaked secrets, redact personal data, and block prompt injections in real time.

Organizations running LLMs in production risk exposing sensitive customer data or system credentials through prompt injections and unvetted model completions, which often bypass conventional network filters. To counter these emerging threats, deploying AI guardrails in the LLM request path has become a necessary architectural standard. Bifrost, a high-performance, open-source AI gateway written in Go by Maxim AI, provides a centralized, inline proxy to define, execute, and scale these safety policies across multiple LLM providers.

This article explores how inline AI guardrails operate inside a proxy architecture, how to balance local and remote evaluation, and how to configure programmatic rules to preserve data privacy.

What are AI Guardrails in the LLM Request Path?

AI guardrails in the LLM request path are automated, policy-enforced security filters that evaluate prompts and model completions as they travel between an application and an LLM provider. Unlike standard application firewalls, these guardrails parse semantic text to identify prompt injections, block sensitive data exposure, and scrub responses for toxic content before payloads can cross security trust boundaries.

Securing these interactions is critical for mitigating the primary risks identified by the OWASP GenAI Security Project. These risks include Prompt Injection (LLM01), Sensitive Information Disclosure (LLM02), and Improper Output Handling (LLM05). As generative AI adoption accelerates, these vulnerabilities create significant operational and compliance liabilities.

According to a security forecast by Gartner, by 2028, 25% of all enterprise generative AI applications will experience at least five minor security incidents per year, up from just 9% in 2025. Deploying runtime guardrails directly within the request pipeline is the primary strategy for preventing these incidents without modifying core application logic.

The Architecture of Inline Guardrail Execution

Enforcing guardrails at the gateway layer introduces a proxy between the application code and the LLM API. This separation of concerns ensures that safety policies remain uniform regardless of which model or provider (such as OpenAI, Anthropic, or Azure OpenAI) is target_url.

The execution flow of an inline guardrail pipeline operates across distinct request and response phases:

The Request Pre-Inference Phase: The application client sends an LLM request to the proxy endpoint. Bifrost, the AI gateway, intercepts the raw input payload. Before the request is sent to the LLM provider, the gateway runs configured pre-inference guardrail checks. If a policy violation (such as a hardcoded API key or a prompt injection pattern) is detected, the gateway triggers an immediate intervention. It rejects the request, returns a standardized GUARDRAIL_INTERVENED error, and logs the event.
Upstream Forwarding: If all pre-inference checks pass (or if redaction rules sanitize the payload), the gateway forwards the request to the target provider. This routing can also employ failover and load balancing rules to maintain uptime during provider outages.
The Response Post-Inference Phase: The LLM provider processes the prompt and returns the output (either as a full text block or as an aggregated stream). The gateway intercepts this output before it reaches the application. Post-inference checks scan the text for hallucinations, sensitive data leaks, or unaligned topics.
Final Delivery: Safe responses are returned to the application client. Violations trigger redaction or blocking, preventing unsafe completions from reaching the user.

Running complex filters inline can introduce latency if the proxy layer is poorly optimized. However, Go-based platforms like Bifrost add only 11 microseconds of mean overhead per request at 5,000 requests per second in sustained benchmarks. This latency profile allows engineering teams to enforce safety policies without degrading application performance.

Local vs. Remote Guardrail Providers

Security teams must decide where content validation actually occurs. The Bifrost gateway supports a hybrid approach, allowing teams to combine low-latency local providers with feature-rich remote API providers.

Local Guardrail Providers

Local providers run entirely in-process within the gateway memory space. Because they require no external network hops, they execute in sub-millisecond timeframes.

Secrets Detection: Bifrost includes an embedded, Gitleaks-backed Secrets Detection engine. It scans LLM request and response payloads for leaked credentials, private keys, database connection strings, and API tokens. Running locally ensures that sensitive API keys are caught before they ever leave the private cloud network.
Custom Regex Engine: For organization-specific patterns (such as proprietary project names or internal database IDs), the Custom Regex provider runs in-process using Go's RE2-compatible engine. This engine also includes pre-configured PII templates for social security numbers, phone numbers, and email addresses.

Remote Guardrail Providers

Remote providers delegate semantic checking to dedicated external safety APIs. These are useful for complex tasks like contextual grounding, topic blocking, and natural language classification.

Supported remote integrations include:

AWS Bedrock Guardrails: Provides managed safety filters, custom topic boundaries, and contextual grounding checks to verify model outputs against source documents.
Azure Content Safety: Evaluates text for hate speech, self-harm, sexual content, and violence.
CrowdStrike Falcon AIDR: Syncs AI runtime security events and threat detections with a centralized Falcon security console.
Third-Party Specialties: Integrates with specialized AI safety platforms like Patronus AI, GraySwan Cygnal, and Lakera Guard for jailbreak detection.

The following table compares the characteristics of local and remote guardrail providers:

Guardrail Provider	Location	Latency Profile	Primary Target Use Case
Custom Regex	Local (RE2 Engine)	Near Zero	Blocking known internal IDs, private codenames, and standard PII
Secrets Detection	Local (In-Process)	Sub-millisecond	Intercepting AWS keys, database strings, and API tokens
AWS Bedrock Guardrails	Remote API	Network Dependent	Enterprise content safety, contextual grounding, and custom topic blocking
CrowdStrike Falcon AIDR	Remote API	Network Dependent	Consolidating AI security telemetry into corporate SOC dashboards
Patronus AI / GraySwan	Remote API	Network Dependent	Real-time jailbreak detection and model response calibration

Configuring Guardrail Rules and CEL Expressions

Policies in Bifrost are managed by separating the provider (the tool that performs the evaluation) from the rule (the logic that determines when and where the check occurs).

The gateway uses Common Expression Language (CEL) expressions to build granular, conditional execution logic. For example, security teams can configure a rule to run expensive remote guardrails only on external-facing traffic, while keeping internal developer keys on low-latency local filters.

A typical configuration file (config.json) defines both parts under the guardrails_config block:

{
  "guardrails_config": {
    "guardrail_providers": [
      {
        "id": 1,
        "provider_name": "secrets",
        "policy_name": "intercept-credentials",
        "enabled": true,
        "config": {
          "action": "redact",
          "redaction_mode": "runtime_reversible"
        }
      },
      {
        "id": 2,
        "provider_name": "regex",
        "policy_name": "redact-personal-data",
        "enabled": true,
        "config": {
          "patterns": [
            {
              "pattern": "[A-Z0-9._%+-]+@[A-Z0-9.-]+\\.[A-Z]{2,}",
              "description": "Email address",
              "entity_type": "EMAIL",
              "flags": "i",
              "action": "redact"
            }
          ]
        }
      }
    ]
  }
}

By linking these providers to specific virtual keys, admins can customize safety parameters for different departments or client applications.

Resolving Redaction: Runtime vs. Logs-Only Policies

When a guardrail provider identifies sensitive data, blocking the request entirely is not always the best user experience. Instead, the gateway supports robust redaction strategies to modify the payload in flight.

Bifrost provides three distinct redaction modes to balance application utility with data privacy compliance:

runtime: The gateway replaces detected sensitive text (such as an email address) with a placeholder (like [REDACTED_EMAIL]) before forwarding the payload to the LLM. The raw value is also redacted in the gateway's internal logs.
logs_only: The LLM receives the raw, unredacted text to preserve full contextual understanding. However, the raw sensitive information is completely scrubbed from audit logs and downstream tracing integrations such as Datadog. This mode ensures compliance with data protection laws like GDPR and SOC 2 without breaking the model's reasoning capabilities.
runtime_reversible: The gateway redacts the text in transit and logs but maps the redacted placeholder back to the original value when the model's response returns. This is ideal for customer support agents that need to process inquiries containing account IDs without exposing those IDs to third-party LLM providers.

This inline redaction flow remains fully compatible with other gateway performance features, such as semantic caching, ensuring that cached query responses do not accidentally store unredacted secrets.

Extending Gateway Safety to the Endpoint with Bifrost Edge

Centralized gateways only govern the traffic that developer teams manually point toward them. To address shadow AI, where employees run local desktop chat applications or terminal coding agents, organizations must extend security policies to the machine level.

The combined AI Gateway + Bifrost Edge architecture accomplishes this by enforcing consistent safety controls directly on individual employee machines. Bifrost Edge (currently in alpha) runs as a lightweight endpoint agent that transparently intercepts local LLM requests and routes them through the central gateway policy engine.

This means the same endpoint security profiles, Gitleaks rules, and custom regex policies applied in-VPC are automatically enforced for terminal environments, browser AI interfaces, and desktop applications, with no manual per-app reconfiguration required. From the Edge overview dashboard, admins can monitor and authorize local AI tools and Model Context Protocol (MCP) servers across the entire corporate fleet, establishing consistent data governance from the developer's laptop to the production cloud.

Next Steps

Securing generative AI applications requires proactive intervention. Relying solely on a model's default alignment leaves systems vulnerable to prompt injections and accidental credential leakage.

Engineering teams evaluating options for centralized AI security can review the open-source repository on GitHub to run the gateway locally. For custom deployments or large-scale team governance, scheduling a Bifrost demo provides a direct walkthrough of enterprise-grade clustering, role-based access controls, and multi-provider guardrail pipelines.