Cilium Gateway API: A Migration Guide from Ingress

#api #kubernetes

The fundamental capabilities of the Kubernetes Ingress API can be insufficient, especially in environments requiring complex traffic management and advanced security. In these scenarios, Cilium Gateway API offers a more flexible and scalable alternative. For teams looking to control application traffic more precisely within Kubernetes and integrate API gateway functionality directly into the cluster, migrating to the Gateway API provides significant advantages. This guide will cover the basic steps, motivations, and technical details of migrating from Ingress to Cilium Gateway API.

What are the Limitations of the Ingress API?

The Ingress API provides a basic mechanism for directing external traffic into Kubernetes, but this simplicity often becomes restrictive in advanced scenarios. While sufficient for tasks like basic HTTP/HTTPS routing and TLS termination, it struggles to meet the dynamic needs of modern microservice architectures. For example, it's a common problem for different teams to create conflicting configurations on the same Ingress Controller, which increases management complexity.

Ingress resource definitions often require adding Ingress Controller-specific extensions via annotations; this negatively impacts portability and standardization. Since each Ingress Controller (Nginx, HAProxy, Traefik, etc.) uses its own set of annotations, if you want to switch from one Ingress Controller to another or use different features, existing configurations might need extensive rewriting. This can pose a serious problem for long-term maintenance and operational overhead, especially in large and multi-tenant environments.

⚠️ Annotation Dependency

Annotations, heavily used in Ingress resources, present a significant challenge for portability and standardization within the Kubernetes ecosystem. Each Ingress Controller using its unique set of annotations makes it difficult to switch between different controllers or establish common best practices. This dependency can limit architectural flexibility in the long run.

Furthermore, the Ingress API is generally limited to HTTP/HTTPS traffic and does not directly support other protocols like TCP/UDP. This means a separate solution is required for non-HTTP applications such as database connections, real-time game servers, or custom network protocols. Features like traffic weighting, advanced header manipulation, client certificate validation (mTLS), or more sophisticated security policies are outside the basic scope of the Ingress API and often require manual configurations in the proxy layer beneath the Ingress Controller.

What is the Gateway API and Why is it Important?

The Gateway API is a next-generation API that offers a more flexible, extensible, and role-based approach to managing network traffic in Kubernetes. Designed to address the limitations of Ingress, it more clearly separates the responsibilities of different network personas (infrastructure providers, cluster operators, application developers). This separation reduces management complexity and increases operational efficiency in large enterprise environments and multi-tenant clusters.

The Gateway API operates through resources such as GatewayClass, Gateway, HTTPRoute, TCPRoute, TLSRoute, and UDPRoute. The Gateway, GatewayClass, and HTTPRoute resources are in the Generally Available (GA) v1 API version since Gateway API v1.0.0. The TLSRoute resource has been promoted to the v1 API version since Gateway API v1.5. GatewayClass defines the capabilities of a specific Gateway Controller, while Gateway resources configure the listener ports and protocols of the gateway (load balancer, proxy). Route resources like HTTPRoute define rules for directing incoming traffic to specific Kubernetes services, header manipulations, and advanced features like traffic weighting. This modular structure enables richer traffic management scenarios and facilitates future extensions.

This architecture reduces annotations dependency, as seen in Ingress, and directly offers more capabilities through the API's core spec. This means fewer configuration changes are needed when switching between different Gateway Controller implementations. Especially the advanced traffic splitting, header-based routing, and path rewriting features offered in the HTTPRoute resource provide flexible and powerful traffic control for microservice-based applications. Additionally, the provision of separate route resources for non-HTTP protocols like TCP, TLS, and UDP makes the Gateway API suitable for a wider range of applications.

Cilium's Gateway API Implementation

Cilium is an eBPF-based Kubernetes networking and security solution that offers a unique implementation of the Gateway API by combining it with its advanced data plane capabilities. Cilium's Gateway API integration provides not only traffic routing but also deep security policies from L3/L4 to L7 and high-performance observability. Thanks to eBPF's kernel-level efficiency, traffic processing occurs with minimal latency and consumes fewer resources compared to traditional proxy-based solutions.

Cilium Gateway API offers the flexibility to configure the Envoy proxy using CiliumEnvoyConfig (CEC) resources, specifically designed for L7 traffic management. This means complex HTTP request manipulations, JWT validation, or advanced rate limiting scenarios can be directly integrated with Gateway API configurations. For example, automatically rejecting requests to an API gateway if they exceed a certain rate, or directing traffic to different services based on the presence of specific headers, can be easily achieved.

ℹ️ The Power of eBPF

Cilium's eBPF-based data plane significantly enhances the performance and capabilities of the Gateway API. Kernel-level traffic processing provides lower latency and higher throughput, while also offering advanced security and observability features. This allows network policies to be applied directly in the data path and detailed telemetry data to be collected.

Furthermore, Cilium's network policies (CiliumNetworkPolicy) work in an integrated manner with the Gateway API, controlling not only traffic routing but also who can access and which protocols can be used. This is a critical capability for implementing a "zero-trust" architecture in a Kubernetes environment. Through this integration, Cilium goes beyond being just a Gateway Controller, combining a full-fledged networking and security solution with Gateway API concepts.

Migration Scenarios from Ingress to Gateway API

Migrating from Ingress to Cilium Gateway API involves moving the traffic of your existing applications to a new structure. This process should generally be gradual and requires understanding how existing Ingress configurations map to Gateway API resources. In the simplest scenario, you start by converting a single Ingress resource to a single HTTPRoute.

For example, a simple Ingress resource might look like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
spec:
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app-service
            port:
              number: 80
  tls:
  - hosts:
    - myapp.example.com
    secretName: my-app-tls

When migrating this Ingress resource to Cilium Gateway API, you first need to define a GatewayClass and a Gateway resource. GatewayClass points to the Cilium Gateway Controller, while the Gateway resource configures the listener ports and TLS certificates that will be exposed to the outside world.

Then, you can convert the existing Ingress rule into an HTTPRoute resource:

apiVersion: gateway.networking.k8s.io/v1 # Gateway API v1 version is used.
kind: HTTPRoute
metadata:
  name: my-app-route
spec:
  parentRefs:
  - name: my-gateway # Refers to the previously defined Gateway
  hostnames:
  - "myapp.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: my-app-service
      port: 80

This migration requires more attention, especially when there are complex path-based routings or host-based multiple Ingress rules. HTTPRoute resources offer flexibility beyond Ingress by defining header-based, query parameter-based, or method-based rules in the matches field. For example, splitting traffic based on specific header values for A/B testing or canary deployments, while possible with Ingress using annotations, is defined as a natural part of the API with HTTPRoute. This makes configurations more readable and manageable.

💡 Phased Migration Strategy

In large and critical systems, it's important to follow a phased strategy instead of a direct migration. First, deploy the new Gateway API configurations, direct a small portion of traffic to the new path (canary deployment), observe, and once you're sure it's working smoothly, switch all traffic over. This minimizes disruption.

If you have an application with multiple Ingress rules, you can group each rule as a separate HTTPRoute or as multiple rules within a single HTTPRoute. This choice depends on your application architecture and management preferences.

Advanced Traffic Management and Security Features

Cilium Gateway API offers a range of advanced traffic management and security features beyond traditional Ingress. These features are critically important, especially for large-scale and high-security enterprise environments. Cilium's eBPF-based data plane delivers these advanced capabilities with low overhead and high performance.

Advanced Load Balancing and Traffic Splitting: HTTPRoute resources support splitting traffic between multiple backendRefs using weight-based load balancing. This allows you to easily implement A/B testing, canary deployments, or blue/green deployment strategies. For example, you can expose a new application version to only 5% of users while directing the remaining 95% to the old version. This capability can be managed directly through the API spec without the complexity of annotations.

Protocol-Level Security and Policy Enforcement: Cilium, working with the Gateway API, can enforce detailed network policies at L3/L4 and L7 levels. This allows specific services to only accept traffic from certain sources or to restrict the use of specific HTTP methods. For example, you can allow only POST and GET requests to reach a backend service while rejecting all other methods. Additionally, thanks to Cilium's mTLS (mutual TLS) integration, you can ensure end-to-end encrypted and authenticated communication from the Gateway to backend services. This is vital, especially in systems handling sensitive data or as part of a "zero-trust" network segmentation.

apiVersion: gateway.networking.k8s.io/v1 # Gateway API v1 version is used.
kind: HTTPRoute
metadata:
  name: secured-app-route
spec:
  parentRefs:
  - name: my-gateway
  hostnames:
  - "secure.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /admin
      headers:
      - name: X-Auth-Token
        type: Present
    backendRefs:
    - name: admin-service
      port: 80
    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: X-Forwarded-User
          value: "authenticated-user"

In the HTTPRoute example above, it's defined that requests to the /admin path must contain a specific X-Auth-Token header. Additionally, an X-Forwarded-User header is added before the request is directed to the admin-service. Such manipulations and conditional routings are scenarios that are often not possible with Ingress or require very complex annotations. This flexibility offers the ability to integrate API gateway functionality directly into Kubernetes.

DDoS Mitigation and Rate Limiting: Thanks to Cilium's eBPF capabilities and Envoy integration, advanced rate limiting policies can be easily applied via the Gateway API. By restricting the number of requests from a specific IP address or a specific user ID within a certain time frame, you can prevent services from being overloaded or mitigate DDoS attacks. Since Cilium applies these policies at the kernel level, it provides a more efficient and resilient layer of protection compared to traditional proxy-based solutions.

Potential Challenges and Solutions

When migrating from Ingress to the Gateway API, it's possible to encounter some potential challenges. These challenges typically stem from conceptual differences, existing infrastructure dependencies, and debugging processes. However, with proper planning and tools, these obstacles can be overcome.

Conceptual Transition and Learning Curve: Migrating from the simple structure of Ingress to the more modular and role-based model of the Gateway API can create a learning curve, especially for teams new to Kubernetes. Understanding what new resources like GatewayClass, Gateway, and HTTPRoute mean and how they interact can take time. At this point, good documentation, example configurations, and interactive training are critically important. Internal knowledge sharing and practicing in small-scale test environments will facilitate this transition.

Existing Ingress Controller Dependencies: If your current Ingress configurations use Ingress Controller-specific annotations or custom templates, converting these dependencies to the new Gateway API model can be complex. You might specifically need to create custom Envoy configurations using Cilium-specific resources like CiliumEnvoyConfig. In such cases, it's important to carefully analyze your old Ingress configurations and determine which annotations correspond to which Gateway API or CiliumEnvoyConfig features. In some situations, running old and new systems in parallel for a period (a dual-stack approach) can be a migration strategy.

Debugging and Observability: When migrating to a new network layer, understanding traffic flow and troubleshooting problems can become difficult. Cilium's Gateway API implementation offers rich observability capabilities with tools like Hubble. Hubble visualizes eBPF-based flow data, allowing you to monitor traffic flow, network policy enforcement, and performance metrics in real-time.

# Start Hubble UI using Cilium CLI
cilium hubble enable # Enables Hubble components.
cilium hubble port-forward & # Starts local port forwarding for Hubble UI.

# Monitor traffic flow from Hubble UI
# Go to http://localhost:12000 in your web browser

Additionally, using the kubectl describe command on Gateway, HTTPRoute, and GatewayClass resources to check their status, events, and potential errors are common troubleshooting steps. Fundamental network layer issues like routing flaps, BGP routing decisions, or MTU/MSS mismatches can arise even when Gateway API configurations are correct, and Cilium's integrated observability tools greatly simplify identifying the root cause.

Conclusion

Migrating to Cilium Gateway API is a critical step towards making traffic management in Kubernetes environments more flexible, secure, and scalable. By overcoming the fundamental limitations of the Ingress API, its role-based design and extensible structure much better meet the requirements of modern microservice architectures. Cilium's eBPF-based implementation combines these capabilities with high performance, deep security policies, and comprehensive observability, offering a unique solution.

This migration process, while seemingly complex at first glance, can be successfully managed with a phased approach and the use of the right tools. It allows application teams more autonomy while also giving infrastructure teams better control over the gateway infrastructure. Especially advanced load balancing, strict security policies, and DDoS mitigation capabilities are indispensable for enterprise-level applications. In the future, the Gateway API is expected to become the standard for external traffic management in Kubernetes. The next step will be to explore the potential of Cilium Gateway API by experimenting with a simple HTTPRoute and Gateway resource in your own test environment.