DEV Community

Cover image for 7 Best Self-Hosted LLM Gateways for In-VPC and Air-Gapped Deployments
Yuki Haramoto
Yuki Haramoto

Posted on

7 Best Self-Hosted LLM Gateways for In-VPC and Air-Gapped Deployments

#aigateway #llm #selfhosted #devops

7 Best Self-Hosted LLM Gateways for In-VPC and Air-Gapped Deployments

A self-hosted AI gateway is the non-negotiable control plane for enterprises that need to keep sensitive data within their network perimeter. This guide compares the top 7 gateways for in-VPC and fully air-gapped deployments.

Moving AI workloads into production exposes a critical infrastructure gap for any organization handling sensitive data. While managed AI gateways offer convenience, routing prompts, completions, and API keys through a third-party service is not an option for teams in healthcare, finance, or government. For these use cases, data must never leave the organizational boundary. A self-hosted LLM gateway solves this by running entirely inside your own infrastructure, providing a unified control plane for AI traffic that you own and operate.

Self-hosting gives you complete data sovereignty, allowing you to enforce security policies, manage costs, and ensure reliability without external dependencies. In a Virtual Private Cloud (VPC) or a fully air-gapped environment, the gateway becomes the single point of entry and exit for all LLM traffic, ensuring every request is authenticated, audited, and compliant with internal governance rules.

This article evaluates the seven best self-hosted LLM gateways purpose-built for these secure environments. We'll compare them on performance, provider support, governance features, and ease of deployment in isolated networks.

What to Look for in a Self-Hosted AI Gateway

When deploying a gateway within a private network, the evaluation criteria shift. The most important factors include:

  • Deployment Flexibility: The gateway must run as a lightweight container (Docker) or within a Kubernetes cluster, with no dependencies on external cloud services for core functionality.
  • Performance: The gateway sits on the critical path of every AI request. Low latency overhead is essential. Gateways written in compiled languages like Go often outperform those based on interpreted languages.
  • Provider & Model Support: It should offer a unified, OpenAI-compatible API for a wide range of commercial models (OpenAI, Anthropic, etc.) and open-weight models served from internal endpoints (like vLLM).
  • Governance & Security: Features like virtual keys, role-based access control (RBAC), and immutable audit logs are critical for managing access and ensuring compliance in regulated environments.
  • Air-Gapped Operation: For the highest level of security, the gateway must function without any internet connectivity, managing all configurations and logging internally.

A blueprint schematic of a secure fortress, where the central keep is labeled 'AI Gateway' and all paths from the outsid

The Top 7 Self-Hosted LLM Gateways

Based on these criteria, here are the best self-hosted gateways for in-VPC and air-gapped deployments.

1. Bifrost

Best for: Performance-critical and enterprise-grade deployments.

Bifrost is a high-performance, open-source AI gateway from Maxim AI, built in Go. It's designed as a core infrastructure service, adding as little as 11 microseconds of overhead per request at 5,000 requests per second. This makes it exceptionally well-suited for high-throughput, low-latency workloads.

Bifrost is built for self-hosting, with deployment options via Docker, Kubernetes, or a simple NPX command. Its enterprise tier is designed explicitly for in-VPC and air-gapped environments, offering features like clustering for high availability, immutable audit logs for compliance (SOC 2, HIPAA), and native integration with vault systems for secrets management.

Key Features:

  • Unified API: A single OpenAI-compatible API for over 20 providers, including self-hosted model endpoints.
  • Reliability: Automatic failover, weighted load balancing, and intelligent routing to maintain uptime.
  • Enterprise Governance: Includes RBAC, virtual keys with per-key budgets and rate limits, and deep observability through Prometheus and OpenTelemetry.
  • MCP Gateway: Native support for the Model Context Protocol (MCP), allowing it to govern not just LLM calls but also agent and tool interactions.

2. LiteLLM

Best for: Maximum flexibility and community support.

LiteLLM is a widely adopted open-source library and proxy that normalizes calls across more than 100 LLM providers into a standard OpenAI API format. It's lightweight and can be deployed as a standalone proxy server within your infrastructure.

Its flexibility is its greatest strength. As a Python library, it can be deeply integrated into existing applications or run as a separate gateway service. The self-hosted proxy is configured via a single YAML file, making it straightforward to manage routing, fallbacks, and virtual keys in a GitOps workflow. While the open-source version is highly capable, an enterprise license is available to unlock features like SSO, audit logs, and advanced guardrails.

Key Features:

  • Extensive Provider Support: Unifies access to over 100 LLM providers.
  • Config-Driven: Manage all settings, from model routing to user budgets, in a single configuration file.
  • Observability: Integrates with tools like Langfuse for detailed tracing and logging.
  • Self-Hosted Control: Runs entirely within your network, ensuring data never leaves your perimeter.

3. Kong AI Gateway

Best for: Teams already invested in the Kong API ecosystem.

Kong Gateway is a popular open-source API gateway that has extended its powerful plugin architecture to manage AI traffic. For organizations that already use Kong to manage their microservices, adding the AI Gateway plugins is a natural extension.

It runs on-premise and can be deployed in a fully air-gapped environment. The AI Gateway provides a provider-agnostic API, allowing you to route requests to various LLMs while centralizing credential management, traffic control, and observability. It can also enforce governance policies, such as PII sanitization and allow/deny lists for prompts.

Key Features:

  • Plugin Architecture: Leverages Kong's mature ecosystem to add authentication, logging, and other capabilities.
  • Unified AI and API Management: Manage both LLM traffic and traditional API traffic from a single control plane.
  • AI-Specific Policies: Includes plugins for PII redaction, prompt engineering, and routing to multiple LLM providers.
  • Multi-Cloud and On-Premise: Designed to run anywhere, providing flexibility for hybrid deployments.

4. Traefik Enterprise AI Gateway

Best for: Kubernetes-native and GitOps-driven environments.

Traefik Enterprise has introduced an AI Gateway built on its popular cloud-native networking stack. It is designed to be Kubernetes-native and supports GitOps workflows, making it a strong fit for platform teams that manage infrastructure as code.

A key focus for Traefik is preventing vendor lock-in and supporting full infrastructure sovereignty. The gateway can be deployed in the cloud, on-premises, or in fully air-gapped environments. It provides a unified API for multi-LLM connectivity and integrates NVIDIA-powered safety guardrails.

Key Features:

  • Kubernetes-Native: Deep integration with Kubernetes for automated configuration and deployment.
  • GitOps Ready: Manage the entire gateway configuration through declarative files versioned in Git.
  • Built-in Guardrails: Includes NVIDIA-powered features for content safety and responsible AI.
  • Infrastructure Sovereignty: Designed from the ground up for self-hosted and air-gapped deployments.

A network switchboard with an operator plugging and unplugging cables. Each cable is a different color, representing dif

5. Envoy AI Gateway

Best for: Teams building custom AI infrastructure on Envoy Proxy.

Envoy AI Gateway is an open-source project built on top of the widely-used Envoy Proxy. It extends Envoy's capabilities to handle LLM-specific traffic, making it a powerful option for teams with deep expertise in Envoy who want to build a custom AI control plane.

The gateway introduces critical features like model-aware routing (routing based on the model field in the request body), credential injection, and token-aware rate limiting. Because it’s built on Envoy, it can be configured and managed using the Kubernetes Gateway API, fitting seamlessly into cloud-native environments.

Key Features:

  • Model-Aware Routing: Route traffic based on the requested model name in the JSON payload.
  • Credential Injection: Securely attach API keys to outbound requests so clients never handle secrets.
  • Token-Based Rate Limiting: Enforce budgets based on actual token usage from model responses.
  • Envoy Ecosystem: Leverage the vast feature set and community around Envoy Proxy.

6. FreeRouter

Best for: Cost-optimization with intelligent, self-hosted routing.

FreeRouter is an open-source, self-hosted AI model router designed as an alternative to managed services like OpenRouter. Its primary focus is on cost savings. It uses a 14-dimension classifier to automatically route simple prompts to cheaper, faster models while reserving more powerful models for complex requests.

It runs locally with your own API keys, ensuring there is no middleman or markup on requests. Configuration is managed via an external JSON file that can be reloaded without restarting the server. It’s a pragmatic choice for teams whose main goal is to control costs while maintaining data privacy.

Key Features:

  • Intelligent Routing: Automatically sends requests to the most cost-effective model based on prompt complexity.
  • Use Your Own Keys: No markup or platform fees; you pay providers directly.
  • Automatic Fallbacks: Retries failed requests with a fallback model to improve reliability.
  • Self-Hosted: Runs entirely within your own infrastructure.

7. LLM Gateway

Best for: A complete platform-in-a-box with a UI.

LLM Gateway offers a self-hostable platform that bundles the gateway with a dashboard, database, and caching layer into a single Docker Compose setup. This provides a more out-of-the-box experience compared to libraries or proxies that require you to build the surrounding infrastructure.

It provides a unified API for over 200 models, smart routing, response caching, and a UI for monitoring usage and costs. While there is a managed cloud version, the entire platform can be self-hosted, giving teams control over their data and deployment environment.

Key Features:

  • All-in-One Platform: Includes a UI, database, and caching, deployable with a single command.
  • Smart Routing: Route requests based on uptime, throughput, price, and latency.
  • Built-in Caching: Reduce costs and improve latency with built-in response caching.
  • Self-Hosted and Open-Source: The platform is AGPLv3-licensed and can run on your own hardware.

Conclusion

For enterprises operating in regulated or security-sensitive domains, a self-hosted AI gateway is a foundational piece of infrastructure. It ensures that AI innovation doesn't come at the cost of data security or compliance. While all the gateways on this list provide the core benefit of keeping traffic within your network, the best choice depends on your specific needs.

For teams needing raw performance and enterprise-grade governance, Bifrost stands out. For those who value maximum flexibility and a large community, LiteLLM is the default choice. Teams already using Kong or Traefik will find their AI gateway offerings to be a seamless extension of their existing infrastructure. Finally, projects like Envoy AI Gateway and FreeRouter provide powerful, specialized tools for teams building a more custom AI stack.

Sources

Top comments (0)