DEV Community

NTCTech
NTCTech

Posted on • Originally published at rack2cloud.com

Kubernetes Ingress to Gateway API Migration: How to Move Without Breaking Production

#kubernetes #platformengineering #devops #cloudnative

Most Gateway API migrations don't fail during the cutover.

They fail in the translation layer — quietly, before traffic ever moves. The annotation audit skipped. The ingress2gateway output treated as deployment-ready. The staging environment that shared none of the complexity of production. By the time the failure surfaces, it looks like a Gateway API problem. It isn't. It's a migration preparation problem.

Ingress-NGINX hit EOL on March 24 — the repository is read-only, no patches, no CVE fixes. Kubernetes 1.36 drops April 22 with Gateway API as the centerpiece. The window where this was a future consideration closed.

migrate ingress to gateway api architecture diagram showing translation layer between flat ingress annotation model and three-tier gateway api resource hierarchy

Before You Migrate — The Annotation Audit

The annotation count per Ingress resource is the number that determines which migration path is actually viable. Run this before anything else:

Kubernetes ingress annotation complexity audit chart showing three migration risk tiers from simple to high-risk annotation surfaces

# Count annotations per ingress resource across all namespaces
kubectl get ingress -A -o json | \
  jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name): \(.metadata.annotations | length) annotations"' | \
  sort -t: -k2 -rn
Enter fullscreen mode Exit fullscreen mode

Three tiers, three different migration realities:

0–5 annotations — ingress2gateway 1.0 handles 80–90%. Most of what lands in your HTTPRoute manifests will be correct. Manual review still required.

6–20 annotations — partial translation. Common annotations (CORS, backend TLS, path rewrite, regex) are covered. Less common ones — configuration-snippet, auth-url, server-snippet — require architectural decisions.

20+ annotations — the tool cannot help you. What those annotations are collectively doing needs to be understood and redesigned before a single manifest is written.

Also find shared Ingress resources — single Ingress objects routing 40+ hostnames for multiple teams. These are coordination problems, not migration targets:

kubectl get ingress -A -o json | \
  jq -r '.items[] | select(.spec.rules | length > 5) |
  "\(.metadata.namespace)/\(.metadata.name): \(.spec.rules | length) host rules"'
Enter fullscreen mode Exit fullscreen mode

ingress2gateway 1.0 — Syntax Translator, Not Architecture Translator

ingress2gateway 1.0 is a genuine improvement — supports 30+ common Ingress-NGINX annotations with behavioral equivalence tests that verify runtime behavior in live clusters, not just YAML structure.

ingress2gateway print \
  --providers=ingress-nginx \
  --namespace=production
Enter fullscreen mode Exit fullscreen mode

Translates cleanly: host/path routing, TLS referencing existing Secrets, CORS headers, backend TLS, path rewrites, regex matching.

Does not translate:

  • nginx.ingress.kubernetes.io/configuration-snippet — custom Lua, no Gateway API equivalent
  • nginx.ingress.kubernetes.io/server-snippet — server-level config, no direct equivalent
  • nginx.ingress.kubernetes.io/auth-url / auth-signin — external auth, requires HTTPRoute filter or extension
  • ConfigMap global defaults — proxy buffer sizes, upstream keepalive, timeout values don't transfer automatically

Implicit defaults that disappear: Ingress-NGINX's ConfigMap applies defaults globally that aren't in your Ingress manifests. They don't transfer. Document your ConfigMap before migration.

What to Migrate First

Migration sequence matters more than migration speed.

Migrate first: New services with no Ingress config. Internal services with 2–3 host rules and no custom annotations. These establish the operational pattern.

Migrate second: Services with standard CORS, TLS, and path rewrite annotations ingress2gateway handles cleanly. Validate behavioral equivalence before decommissioning each Ingress resource.

Migrate last: configuration-snippet services, external auth integrations, shared Ingress resources, anything with a P1 incident in the last 90 days.

The Side-by-Side Pattern — The Only Safe Model

Side-by-side Kubernetes ingress and gateway api deployment pattern showing shared load balancer IP with parallel traffic paths during migration

Cutover-first is an anti-pattern. Both controllers run simultaneously against the same cluster, sharing the same external load balancer IP.

# Install Gateway API CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

# Deploy Gateway API controller alongside existing Ingress controller
kubectl apply -f https://github.com/nginx/nginx-gateway-fabric/releases/download/v1.5.0/nginx-gateway-fabric.yaml
Enter fullscreen mode Exit fullscreen mode 
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production-gateway
  namespace: nginx-gateway
spec:
  gatewayClassName: nginx-gateway
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: production-tls
        namespace: nginx-gateway
Enter fullscreen mode Exit fullscreen mode

The one rule: Never configure both an Ingress resource and an HTTPRoute for the same hostname and path simultaneously. The two controllers compete for the same traffic.

HTTPRoute Translation — Before and After 

# Before — Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 8080
Enter fullscreen mode Exit fullscreen mode 
# After — HTTPRoute
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-route
  namespace: production
spec:
  parentRefs:
  - name: production-gateway
    namespace: nginx-gateway
  hostnames:
  - "app.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /
    backendRefs:
    - name: api-service
      port: 8080
Enter fullscreen mode Exit fullscreen mode

Traffic splitting — native in HTTPRoute, no annotations needed:

rules:
- backendRefs:
  - name: api-stable
    port: 8080
    weight: 90
  - name: api-canary
    port: 8080
    weight: 10
Enter fullscreen mode Exit fullscreen mode

Adjacent Dependencies — Address Before First HTTPRoute

cert-manager: Requires v1.14.0+ for Gateway API support. Configuration moves from Ingress annotations to Gateway resource annotations.

ExternalDNS: Requires v0.14.0+ for Gateway API support. DNS records for HTTPRoute hostnames won't be created automatically on older versions — DNS resolution fails silently.

Prometheus/alerting: Gateway API controllers expose different metric structures than Ingress-NGINX. Dashboards keyed to Ingress-NGINX metric names won't work without updates.

DNS Cutover Sequence

  1. All services validated under load via HTTPRoutes in side-by-side state
  2. Keep Ingress resources — rollback safety
  3. Reduce DNS TTL to 60 seconds — 24 hours before cutover
  4. Update external DNS record
  5. Monitor error rates for 30 minutes
  6. Remove decommissioned Ingress resources after 24 hours of clean traffic — not before

Production Failure Modes — Works in Staging, Breaks in Prod

Four Gateway API migration production failure modes — header routing mismatch, ReferenceGrant missing, TLS handshake surprise, and implicit defaults disappearing

Header routing mismatch — HTTPRoute header matching is exact by default. Ingress-NGINX treats some header matching case-insensitively. Verify your Gateway implementation's behavior explicitly.

ReferenceGrant missing — the most common failure in multi-team clusters. An HTTPRoute in namespace frontend referencing a Service in namespace api requires a ReferenceGrant. Without it: accepted status, 500 response.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-frontend-routes
  namespace: api
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    namespace: frontend
  to:
  - group: ""
    kind: Service
Enter fullscreen mode Exit fullscreen mode

TLS handshake surprise — Ingress-NGINX's TLS defaults (cipher suites, protocol versions) live in the ConfigMap. Gateway API controllers start from their own defaults. Validate TLS behavior against legacy clients explicitly before cutover.

Implicit defaults disappearing — proxy timeouts, upstream keepalive, buffer sizes set in the Ingress-NGINX ConfigMap don't transfer. A service relying on a 600-second proxy timeout reverts to the controller's default silently. Audit the ConfigMap before any service migrates.

Architect's Verdict

ingress2gateway 1.0 handles straightforward migrations cleanly. The gap it cannot close is between syntax translation and architectural translation. Find the untranslatable annotations during the audit — not during the rollback.

The side-by-side pattern is the correct one. Both controllers running against the same load balancer IP costs nothing and eliminates the primary risk vector: the all-at-once cutover that discovers production failure modes under incident conditions.

The migration doesn't fail where you think it will. It fails in everything you assumed would just translate.

Part of the Rack2Cloud Kubernetes Ingress Architecture Series. Full post with interactive examples at rack2cloud.com.

Top comments (0)