DEV Community

Florient DOGBE
Florient DOGBE

Posted on

Routes, Ingress or Gateway API on OpenShift 4: Which to Choose and How to Migrate

#openshift #kubernetes #gatewayapi #networking

Every OpenShift cluster ships with at least two ways to expose an HTTP service, and since 4.14 there is a third. Route is the native OpenShift object, Ingress is the portable Kubernetes one, and the Gateway API is the upstream replacement for Ingress — implemented on OpenShift by an Istio-based gateway controller managed by the Cluster Ingress Operator.

This is not a cosmetic choice. The abstraction you pick determines where TLS is terminated, which timeouts you can tune, whether you can do header manipulation or traffic splitting without a sidecar hack, and which metrics you will have during an incident at 3 a.m. This article compares the three on concrete limits, then walks through migrating a production Route loaded with HAProxy annotations to a Gateway API HTTPRoute with re-encrypt TLS — without downtime.

Context: three abstractions, two data planes

The important thing to understand first: Routes and Ingress share the same data plane, the HAProxy-based IngressController (router-default in openshift-ingress). The Gateway API does not — it runs Envoy pods managed by OpenShift Service Mesh / Istio.

Route

route.openshift.io/v1. Native, stable, well understood by the Ingress Operator. It supports three TLS modes out of the box (edge, passthrough, reencrypt), weighted backends for canaries, and a long tail of behaviour exposed through haproxy.router.openshift.io/* annotations.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: payments
  namespace: payments
  annotations:
    haproxy.router.openshift.io/timeout: 120s
    haproxy.router.openshift.io/balance: leastconn
    haproxy.router.openshift.io/hsts_header: "max-age=31536000;includeSubDomains"
    haproxy.router.openshift.io/ip_whitelist: "203.0.113.0/24 198.51.100.7"
    router.openshift.io/cookie_name: payments-sticky
spec:
  host: payments.apps.example.com
  to:
    kind: Service
    name: payments-svc
    weight: 100
  port:
    targetPort: https
  tls:
    termination: reencrypt
    insecureEdgeTerminationPolicy: Redirect
    # destinationCACertificate omitted: the router uses the cluster
    # service CA when the backend uses a service serving certificate
Enter fullscreen mode Exit fullscreen mode

The limits are real: the annotation surface is a flat, untyped, string-based API. There is no schema validation, a typo silently does nothing, and anyone with edit in the namespace can set a 1-hour timeout or disable the rate limit. Rewrites are limited to rewrite-target, there is no request header manipulation, no path-based routing beyond a single prefix per Route, and no retry policy.

Ingress

networking.k8s.io/v1. On OpenShift, an Ingress is not served directly: the ingress-to-route controller converts it into one or more Routes. You therefore inherit every Route limitation plus a translation layer. Re-encrypt requires an OpenShift-specific annotation, which defeats the portability argument:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: payments
  namespace: payments
  annotations:
    route.openshift.io/termination: reencrypt
    route.openshift.io/destination-ca-certificate-secret: payments-ca
spec:
  ingressClassName: openshift-default
  tls:
    - hosts: ["payments.apps.example.com"]
      secretName: payments-edge-tls
  rules:
    - host: payments.apps.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: payments-svc
                port:
                  number: 8443
Enter fullscreen mode Exit fullscreen mode

Use Ingress when a Helm chart or an operator generates it for you and you do not want to fork the chart. Do not use it as a strategic choice on OpenShift: you get Route semantics with an extra controller in the path and worse debuggability (oc get route -l to find the generated child object).

Gateway API

gateway.networking.k8s.io/v1. Three roles, three objects: GatewayClass (cluster/infra), Gateway (platform team — listeners, ports, TLS, which namespaces may attach), HTTPRoute (application team — hostnames, matches, filters, backends). It is Tech Preview from OpenShift 4.14 and GA from 4.19 with the Red Hat OpenShift Service Mesh 3 operator; check the docs for your exact minor version.

What you actually gain: typed fields for timeouts and retries, header and URL rewrite filters, weighted traffic splitting as a first-class field, cross-namespace backends gated by ReferenceGrant, and clean RBAC separation. What you pay: a second data plane (Envoy pods, its own LoadBalancer service, its own DNS records) and a dependency on the Service Mesh operator lifecycle.

Where each one breaks

Need Route Ingress (OCP) Gateway API
Edge / re-encrypt / passthrough TLS Yes Yes, via annotations Terminate / Passthrough; re-encrypt via BackendTLSPolicy
Per-route request timeout Annotation (untyped) Inherited spec.rules[].timeouts.request
Header add/remove No No RequestHeaderModifier / ResponseHeaderModifier
Path rewrite rewrite-target only Same URLRewrite filter
Traffic split alternateBackends weights No backendRefs[].weight
Cross-namespace backend No No Yes, with ReferenceGrant
Retries / mirroring No No Yes (retry, RequestMirror)
Metrics HAProxy (haproxy_backend_*) Same Envoy (istio_requests_total, per-route labels)
RBAC separation infra/app Weak Weak Strong (Gateway vs HTTPRoute)

Rule of thumb: stay on Routes for internal apps, operator-managed endpoints and anything using passthrough for mTLS. Move to the Gateway API when you need typed traffic policy, canaries, header manipulation, or when several teams share one entry point and you want to stop handing out annotation privileges.

Migrating a Route to an HTTPRoute

Target: the payments Route above, re-encrypt to a backend serving TLS with an OpenShift service serving certificate.

Step 0 — inventory your annotations

You cannot plan a migration you have not measured. Dump every annotated Route in the cluster:

oc get routes -A -o json | jq -r '
  .items[]
  | select(.metadata.annotations != null)
  | [ .metadata.namespace, .metadata.name, (.spec.tls.termination // "none"),
      ( .metadata.annotations | to_entries
        | map(select(.key | test("haproxy\\.router\\.openshift\\.io|^router\\.openshift\\.io")))
        | map("\(.key)=\(.value)") | join(";") ) ]
  | @tsv' | column -t -s $'\t'
Enter fullscreen mode Exit fullscreen mode

Anything returning an empty annotation column is a trivial migration. The rest needs a translation decision.

Step 1 — enable the Gateway API

On GA versions, install the OpenShift Service Mesh 3 operator, then create the GatewayClass; the Ingress Operator reconciles the Istio control plane for you.

oc get csv -n openshift-operators | grep -i servicemesh
oc get crd gateways.gateway.networking.k8s.io \
  -o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}{"\n"}'
Enter fullscreen mode Exit fullscreen mode 
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: openshift-default
spec:
  controllerName: openshift.io/gateway-controller/v1
Enter fullscreen mode Exit fullscreen mode 
oc wait --for=condition=Accepted gatewayclass/openshift-default --timeout=300s
Enter fullscreen mode Exit fullscreen mode

Step 2 — the Gateway (platform team)

Deploy it in openshift-ingress, with an explicit namespace selector so that application teams cannot attach routes by accident.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-gateway
  namespace: openshift-ingress
spec:
  gatewayClassName: openshift-default
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.gw.apps.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: wildcard-gw-tls   # created by cert-manager, type kubernetes.io/tls
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: "true"
        kinds:
          - kind: HTTPRoute
Enter fullscreen mode Exit fullscreen mode 
oc label namespace payments gateway-access=true
oc get gateway prod-gateway -n openshift-ingress \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {end}{"\n"}'
oc get svc -n openshift-ingress -l gateway.networking.k8s.io/gateway-name=prod-gateway
Enter fullscreen mode Exit fullscreen mode

Note the dedicated wildcard *.gw.apps.example.com: using a different hostname than the live Route is what makes the migration reversible. The Ingress Operator publishes DNS records for listener hostnames inside the cluster base domain; for any other domain, point your own CNAME at the gateway LoadBalancer.

Step 3 — the HTTPRoute (application team) 

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments
  namespace: payments
spec:
  parentRefs:
    - name: prod-gateway
      namespace: openshift-ingress
      sectionName: https
  hostnames:
    - payments.gw.apps.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      timeouts:
        request: 120s          # replaces haproxy.router.openshift.io/timeout
        backendRequest: 30s
      filters:
        - type: ResponseHeaderModifier
          responseHeaderModifier:
            set:
              - name: Strict-Transport-Security
                value: "max-age=31536000; includeSubDomains"
      backendRefs:
        - name: payments-svc
          port: 8443
          weight: 100
Enter fullscreen mode Exit fullscreen mode 
oc get httproute payments -n payments -o jsonpath='{.status.parents[0].conditions}' | jq
Enter fullscreen mode Exit fullscreen mode

Accepted=True and ResolvedRefs=True are what you want. ResolvedRefs=False almost always means the Service port does not exist or the namespace label is missing.

Step 4 — re-encrypt with BackendTLSPolicy

reencrypt has no direct equivalent field; it is expressed as a policy attached to the Service.

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: payments-reencrypt
  namespace: payments
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: payments-svc
      sectionName: "8443"
  validation:
    caCertificateRefs:
      - group: ""
        kind: ConfigMap
        name: service-ca-bundle
    hostname: payments-svc.payments.svc
Enter fullscreen mode Exit fullscreen mode

Gotcha worth remembering: the OpenShift service CA operator injects the bundle under the key service-ca.crt, while BackendTLSPolicy expects ca.crt. Copy it:

oc create configmap service-ca-injected -n payments \
  --dry-run=client -o yaml | \
  oc annotate -f - --local -o yaml \
    service.beta.openshift.io/inject-cabundle=true | oc apply -f -

oc get cm service-ca-injected -n payments -o jsonpath='{.data.service-ca\.crt}' > /tmp/ca.crt
oc create configmap service-ca-bundle -n payments --from-file=ca.crt=/tmp/ca.crt
Enter fullscreen mode Exit fullscreen mode

The hostname field must match a SAN in the backend certificate — service serving certs contain <service>.<ns>.svc and <service>.<ns>.svc.cluster.local.

Step 5 — translate the remaining annotations

HAProxy annotation Gateway API equivalent
timeout HTTPRoute.spec.rules[].timeouts.request
hsts_header ResponseHeaderModifier filter
rewrite-target URLRewrite filter
balance Istio DestinationRule.trafficPolicy.loadBalancer
cookie_name / disable_cookies DestinationRule consistent-hash on cookie
ip_whitelist Istio AuthorizationPolicy with remoteIpBlocks
rate-limit-connections.* Envoy local/global rate limit via Istio
set-forwarded-headers Gateway/Istio XFF config (numTrustedProxies)

Anything in the bottom half of that table is Istio configuration, not Gateway API. Budget for it: that is the real cost of the migration, not the HTTPRoute itself.

Step 6 — test, cut over, roll back

Validate end-to-end before touching DNS, bypassing resolution:

GW_IP=$(oc get svc -n openshift-ingress \
  -l gateway.networking.k8s.io/gateway-name=prod-gateway \
  -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')

curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' \
  --resolve payments.gw.apps.example.com:443:$GW_IP \
  https://payments.gw.apps.example.com/api/healthz
Enter fullscreen mode Exit fullscreen mode

Then cut over at DNS: change the production record payments.apps.example.com from the router LoadBalancer to the gateway LoadBalancer, after adding the hostname to the Gateway listener and the HTTPRoute. Drop the TTL to 60s a day in advance. Keep the Route object in place — rolling back is a single DNS change, not a redeploy.

For internal consumers, an intermediate step is to flip a percentage of pods via backendRefs weights rather than DNS:

      backendRefs:
        - name: payments-svc
          port: 8443
          weight: 90
        - name: payments-svc-canary
          port: 8443
          weight: 10
Enter fullscreen mode Exit fullscreen mode

Step 7 — rewire observability

Your HAProxy dashboards stop filling up the moment DNS flips. Enable gateway access logs and point your alerts at Envoy metrics before the cutover, not after:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: gateway-access-logs
  namespace: openshift-ingress
spec:
  selector:
    matchLabels:
      gateway.networking.k8s.io/gateway-name: prod-gateway
  accessLogging:
    - providers:
        - name: envoy
Enter fullscreen mode Exit fullscreen mode 
oc logs -n openshift-ingress -l gateway.networking.k8s.io/gateway-name=prod-gateway -f
Enter fullscreen mode Exit fullscreen mode

haproxy_backend_http_responses_total becomes istio_requests_total{response_code=...}, and haproxy_backend_http_average_response_latency_milliseconds becomes the istio_request_duration_milliseconds_bucket histogram. Scrape them with a ServiceMonitor in user workload monitoring.

Security best practices

  • Keep TLS termination explicit. Prefer reencrypt / BackendTLSPolicy over edge for anything handling credentials or payment data; cleartext between the gateway and the pod is still cleartext on the SDN.
  • Pin a TLS floor. For Routes, set spec.tlsSecurityProfile (Intermediate or Modern) on the IngressController. For Gateway API, set it at the Istio control plane level (meshConfig.tlsDefaults.minProtocolVersion: TLSV1_3) — the Gateway object alone does not give you a cipher policy.
  • Close namespace ownership. oc patch ingresscontroller/default -n openshift-ingress-operator --type=merge -p '{"spec":{"routeAdmission":{"namespaceOwnership":"Strict","wildcardPolicy":"WildcardsDisallowed"}}}'. The Gateway API equivalent is allowedRoutes.namespaces.from: Selector plus ReferenceGrant for cross-namespace backends — never from: All on an internet-facing gateway.
  • Stop granting annotation power. A Route annotation is an RBAC-free policy knob. With the Gateway API, timeouts and TLS live on objects the platform team owns; restrict gateways and backendtlspolicies to platform RBAC and leave httproutes to app teams.
  • Never put certificate material in a Route. destinationCACertificate inlined in Git is a recurring finding. Use service serving certificates (service.beta.openshift.io/serving-cert-secret-name) or cert-manager with secrets referenced by name.
  • Trust your proxy chain explicitly. IP allowlists and X-Forwarded-For logging are only meaningful once numTrustedProxies matches your actual LB chain; otherwise an attacker spoofs the header.
  • Scan for forgotten exposure. oc get routes -A -o custom-columns='NS:.metadata.namespace,HOST:.spec.host,TLS:.spec.tls.termination' and flag every <none>.

Key takeaways

  • Routes and Ingress share the HAProxy router; Gateway API is a separate Envoy data plane with its own LoadBalancer, DNS and metrics. Plan capacity and observability accordingly.
  • Ingress on OpenShift is a compatibility shim, not a target architecture. Use it only for third-party charts.
  • The Gateway API's real value is typed configuration and RBAC separation — annotations become fields that are validated and owned by the right team.
  • Migrate on a parallel hostname, validate with curl --resolve, cut over with DNS, and keep the Route object until you are confident. Rollback should be one record change.
  • The hard part is never the HTTPRoute: it is the long tail of HAProxy annotations that map to Istio DestinationRule, AuthorizationPolicy or rate limiting. Inventory them first, and migrate the simple routes before the complicated ones.

Top comments (0)