Why Agentic AI Needs a Gateway: Agentgateway Explained from First Principles

AI applications are rapidly moving beyond simple calls to a single language model.

A production agent may need to:

• Send requests to multiple LLM providers
• Discover and call MCP tools
• Communicate with other agents
• Access internal REST APIs
• Enforce authentication and authorization
• Apply rate limits, retries, and timeouts
• Track tokens, latency, cost, and failures
• Prevent unsafe prompts or tool calls

When every application implements these capabilities independently, the architecture quickly becomes difficult to operate.

This is where an agent gateway becomes useful.

An agent gateway creates a managed connectivity and policy layer between AI applications and the models, tools, services, and agents they use.

In this article, we will explore the basics of Agentgateway, an open-source, AI-focused gateway designed to support LLM traffic, Model Context Protocol connections, agent-to-agent communication, inference workloads, and traditional application traffic.

The problem: agents need more than an API endpoint

A simple AI application might call a model directly:

That is manageable during experimentation.

A production agentic system looks more like this:

Every connection introduces operational questions:

• Where are credentials stored?
• Which models can each application access?
• Which users may invoke sensitive tools?
• What happens when a provider becomes unavailable?
• How are token usage and latency measured?
• How do we prevent one application from consuming the entire budget?
• How do we apply consistent security policies across MCP, HTTP, and LLM traffic?

Embedding all this logic inside every agent creates duplicated code and inconsistent governance.

A gateway moves many of these cross-cutting responsibilities into a shared infrastructure layer.

What is Agentgateway?

Agentgateway is an open-source, AI-first data plane for connecting applications to agents, MCP tools, LLM providers, inference services, and conventional backends.

In Kubernetes, Agentgateway also includes a control plane. The control plane watches Kubernetes Gateway API resources and Agentgateway-specific custom resources, converts them into runtime configuration, and distributes that configuration to Agentgateway proxies.

The project supports several major connectivity scenarios:

1. Routing requests to hosted or local LLMs
2. Connecting clients to MCP servers
3. Agent-to-agent, or A2A, communication
4. Load balancing across inference services
5. Routing ordinary HTTP, gRPC, TCP, and TLS traffic

The broader idea is important: Agentgateway is not only an “LLM proxy.” It attempts to provide one connectivity layer for the different protocols that appear inside an agentic system.

Agent gateway versus traditional API gateway

Traditional API gateways remain valuable. They manage HTTP APIs, authentication, routing, TLS termination, rate limiting, and traffic policies.

Agentic applications introduce additional requirements.

Traditional gateway concern	Agent gateway concern
HTTP and gRPC routing	LLM, MCP, A2A, HTTP, and inference routing
Requests per second	Requests, tokens, model usage, and tool calls
API authentication	Model credentials, MCP OAuth, JWTs, and tool authorization
Service load balancing	Model routing and inference-aware load balancing
URL-level policies	Model-, prompt-, agent-, and tool-level policies
API observability	Token usage, time to first token, model latency, and agent traffic
Backend failover	Model-provider and endpoint failover

The two concepts are not necessarily competitors. An organization may continue using its existing API gateway while introducing an agent gateway for AI-specific connectivity.

The basic architecture

Agentgateway uses a control-plane and data-plane architecture in Kubernetes.

Control plane

The control plane is a Kubernetes controller.

It watches resources such as:

• Gateway
• HTTPRoute
• GRPCRoute
• AgentgatewayBackend
• AgentgatewayPolicy
• Kubernetes Service
• Kubernetes Secret

It translates these declarative resources into Agentgateway configuration and sends configuration updates to the proxies through xDS.

The control plane also updates Kubernetes resource status, helping operators determine whether a configuration was accepted and successfully programmed.

Data plane

The data plane is the Agentgateway proxy that processes live traffic.

It receives requests from clients, evaluates listeners, routes, backends, and policies, and forwards each request to the appropriate destination.

That destination could be:

• An LLM provider
• A local inference endpoint
• An MCP server
• Another agent
• A Kubernetes service
• A conventional HTTP or gRPC backend

The separation is useful because the control plane manages desired configuration, while the data plane handles the runtime request path.

Understanding the main Kubernetes resources

You do not need to understand every custom resource before getting started. Five concepts cover the basic request flow.

1. GatewayClass

A GatewayClass identifies the controller responsible for managing a gateway.

When the gatewayClassName is set to agentgateway, the Agentgateway controller manages that gateway.

2. Gateway

A Gateway defines where traffic enters the system.

It describes:

• Listening ports
• Protocols
• Hostnames
• TLS configuration
• Which namespaces may attach routes

Conceptually, it creates the front door.

3. HTTPRoute

An HTTPRoute decides where requests should go.

It can match traffic using information such as:

• Path
• Host
• HTTP method
• Headers
• Query parameters

It then forwards matching traffic to a backend.

4. AgentgatewayBackend

An AgentgatewayBackend describes an AI-aware destination.

For example, a backend may represent:

• An OpenAI model
• An Anthropic model
• Amazon Bedrock
• Azure OpenAI
• Gemini
• A local vLLM deployment
• An MCP server
• A group of MCP targets

This resource gives Agentgateway more information than it would receive from an ordinary hostname and port.

5. AgentgatewayPolicy

An AgentgatewayPolicy defines runtime behavior.

Depending on the use case, policies can address areas such as:

• Authentication
• Authorization
• Rate limiting
• Guardrails
• Request transformations
• Timeouts
• Retries
• Header modification
• Access control

This separation lets platform teams manage infrastructure policies without placing all the logic inside application code.

Installing Agentgateway on Kubernetes

The official quickstart assumes that you already have:

• A Kubernetes cluster
• kubectl
• Helm

For local experimentation, the documentation suggests using Kind. The commands below follow the documented Agentgateway 1.3 installation path available at the time of writing. Always check the current documentation before using fixed versions in a production environment.

Step 1: Create a local cluster

kind create cluster

Skip this step when you already have access to a Kubernetes cluster.

Step 2: Install the Kubernetes Gateway API CRDs

kubectl apply \
  --server-side \
  --force-conflicts \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.0/standard-install.yaml

Gateway API provides resources such as GatewayClass, Gateway, and HTTPRoute.

Agentgateway builds on this Kubernetes-native model instead of introducing a completely separate routing API.

Step 3: Install the Agentgateway CRDs

helm upgrade -i agentgateway-crds \
  oci://cr.agentgateway.dev/charts/agentgateway-crds \
  --create-namespace \
  --namespace agentgateway-system \
  --version v1.3.1 \
  --set controller.image.pullPolicy=Always

This installs Agentgateway-specific resource definitions such as AgentgatewayBackend and AgentgatewayPolicy.

Step 4: Install the control plane

helm upgrade -i agentgateway \
  oci://cr.agentgateway.dev/charts/agentgateway \
  --namespace agentgateway-system \
  --version v1.3.1 \
  --set controller.image.pullPolicy=Always \
  --wait

Verify that the controller is running:

kubectl get pods -n agentgateway-system

You should see the Agentgateway controller in the Running state.

Step 5: Create an Agentgateway proxy

Create a Gateway that uses the agentgateway GatewayClass:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway-proxy
  namespace: agentgateway-system
spec:
  gatewayClassName: agentgateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All

Apply it:

kubectl apply -f gateway.yaml

The control plane notices the new Gateway and provisions an Agentgateway proxy deployment for it.

Verify the resources:

kubectl get gateway agentgateway-proxy \
  -n agentgateway-system

kubectl get deployment agentgateway-proxy \
  -n agentgateway-system

Step 6: Access the proxy locally

A local Kind cluster normally does not provide an external load balancer address. Port-forward the proxy instead:

kubectl port-forward \
  deployment/agentgateway-proxy \
  -n agentgateway-system \
  8080:80

The gateway is now available through:

http://localhost:8080

At this point, the gateway exists, but it does not yet have a route or AI backend. The next step is to connect an LLM, MCP server, or ordinary HTTP service.

Example request flow for an LLM

Suppose an application needs to access an LLM provider.

The logical configuration becomes:

The provider credential can be stored in a Kubernetes Secret. An AgentgatewayBackend identifies the provider and references that secret. An HTTPRoute then sends incoming traffic to the backend.

The official OpenAI quickstart follows this exact pattern:

1. Store the provider API key in a Kubernetes secret
2. Create an AgentgatewayBackend
3. Create an HTTPRoute
4. Send the model request through the gateway

Agentgateway can rewrite the routed request to the provider’s model endpoint, allowing the client to call the gateway rather than integrating directly with the external provider.

This creates a useful separation:

Example request flow for MCP

MCP allows models and agents to discover and invoke external tools.

An MCP backend can point to a Kubernetes service or a static address. For Kubernetes services, Agentgateway uses the service configuration to identify MCP traffic and route it to the proper endpoint.

The official quickstart demonstrates deploying an MCP website-fetching tool, defining it as an AgentgatewayBackend, and exposing it through a route.

A gateway in front of MCP servers can become particularly valuable for:

• Central authentication
• Tool-level authorization
• Rate limiting
• Consistent endpoint discovery
• Auditing tool calls
• Hiding internal MCP server topology
• Combining multiple tool servers behind one endpoint

The last point is especially interesting. Agentgateway supports MCP multiplexing, where tools from multiple MCP backends can be presented through a shared gateway endpoint and routed to their original servers when invoked.

What an agent gateway does not replace

An agent gateway is infrastructure, not the complete agent runtime.

It does not replace:

• Agent planning
• Prompt design
• Conversation memory
• Business workflows
• Human approval logic
• Retrieval pipelines
• Evaluation frameworks
• Domain-specific reasoning
• Application-level error handling

A useful distinction is:

Agent framework:
Decides what the agent should do.

Agent gateway:
Controls how the agent connects to models, tools, services, and other agents.

For example, an agent framework may decide that it needs to call a customer-account tool.

The gateway can then determine:

• Whether the user is authorized to invoke that tool
• Which MCP server hosts the tool
• Whether the request exceeds a rate limit
• Which credentials should be attached
• How the call should be logged
• What should happen if the backend fails

The gateway governs connectivity. The application still owns intent and business behavior.

Why the Gateway API approach matters

Agentgateway is based on the Kubernetes Gateway API, an official Kubernetes project for managing Layer 4 and Layer 7 routing.

This gives Kubernetes teams a familiar declarative model:

Gateway → Route → Backend

Agentgateway extends that model for AI-specific scenarios rather than asking operators to abandon existing Kubernetes networking concepts. Its extensions add capabilities for protocols and destinations such as MCP, A2A, and LLM providers.

This can also support clearer organizational ownership:

• Platform teams manage gateways and shared infrastructure
• Security teams define policies
• AI teams define model and tool backends
• Application teams attach routes
• Operations teams monitor the runtime traffic

That is generally easier to govern than embedding provider keys and routing logic across dozens of agent repositories.

Production considerations

Agentgateway is promising, but installing a gateway does not automatically make an AI system production-ready.

High availability

The gateway becomes part of the critical request path. Run multiple replicas, configure disruption budgets, and test failure behavior.

Secret management

Kubernetes secrets are a starting point, not a complete enterprise secret-management strategy. Consider integration with your organization’s managed secret store and rotation process.

Latency

Every policy, transformation, guardrail, and external authorization call can add latency. Measure the complete path, including time to first token for streaming responses.

Streaming behavior

LLM streaming differs from ordinary HTTP responses. Test client disconnects, idle timeouts, retries, token accounting, and partially generated responses.

MCP sessions

MCP can introduce session and connection-management questions that do not appear in ordinary stateless REST traffic. Review the behavior of your selected MCP transport and test it with multiple proxy replicas.

Authorization granularity

Authentication tells the gateway who is calling. Authorization determines what that caller may do.

For agents, authorization may need to consider:

User identity
+ Agent identity
+ Application
+ Requested tool
+ Tool arguments
+ Environment
+ Data sensitivity

A valid token should not automatically grant access to every MCP tool.

Observability depth

Basic request metrics are not enough for complex agents.

Useful dimensions include:

• Model provider
• Model name
• Prompt and completion tokens
• Time to first token
• Total generation latency
• MCP server
• Tool name
• Tool latency
• Authorization decision
• Retry count
• Guardrail decision
• Estimated cost
• Final request outcome

A December 2025 independent review praised Agentgateway’s broad AI-focused feature set but also identified gaps in areas such as documentation, protocol completeness, and MCP-specific metrics. Because the project is evolving quickly, teams should verify important capabilities against the exact version they plan to deploy.

When should you consider an agent gateway?

An agent gateway becomes increasingly useful when:

• Several applications use the same model providers
• Agents connect to multiple MCP servers
• Different teams require different tool permissions
• Provider credentials are duplicated across services
• Model traffic requires centralized cost controls
• Security policies must be applied consistently
• You need model or endpoint failover
• Platform teams want standardized AI connectivity
• Agent-to-agent traffic must be governed
• AI workloads already run on Kubernetes

For a single prototype calling one model, a gateway might add unnecessary complexity.

For an enterprise running many agents, tools, models, and providers, that complexity already exists. A gateway provides a place to manage it deliberately.

Final thoughts

The transition from chatbots to agentic systems changes the networking layer.

The runtime is no longer only:

Client → API

It is becoming:

User
  → Agent
    → Model
    → Memory
    → MCP Tools
    → Enterprise APIs
    → Other Agents
    → Human Approval

That environment needs more than basic request forwarding.

It needs a connectivity layer that understands AI protocols, centralizes policies, protects credentials, manages traffic, and exposes what happens between an agent and its dependencies.

Agentgateway approaches this problem through a Kubernetes-native control plane, an AI-aware data plane, Gateway API resources, and specialized backends for models, MCP servers, inference endpoints, and agents.

It will not replace your agent framework or application architecture.

But it can provide the governed runtime boundary through which those systems communicate—and that boundary is becoming an important part of production agentic AI architecture.

References

• Agentgateway Kubernetes installation and proxy quickstart

• Agentgateway control-plane and data-plane architecture.