DEV Community

Onur Bolatoğlu
Onur Bolatoğlu

Posted on

What Silently Breaks When You Migrate from Ingress NGINX to HAProxy

#devops #infrastructure #kubernetes #networking

Introduction

ingress-nginx was retired in March 2026. There are no more releases and no more security patches. It routes traffic into roughly half of all Kubernetes clusters, which makes this a problem for a lot of teams. Installing a retired project is still completely silent. I tried it last week and got no warning at any step. Most migration guides treat this as an annotation mapping exercise. But the real problem is not the annotations. It is everything that keeps working afterwards while quietly doing the wrong thing.

The environment

Everything below was tested on a single-node cluster:

Ubuntu 26.04 LTS (kernel 7.0.0-31)
k3s v1.36.4+k3s1
Helm v3.21.4
HAProxy 3.2.9 (host, LTS branch)
ingress-nginx chart 4.15.1 / app 1.15.1  (retired March 2026)
HAProxy Kubernetes Ingress Controller (HAProxy 3.2.23)
Enter fullscreen mode Exit fullscreen mode

Traffic path: Cloudflare (proxied) → HAProxy on the host → ingress controller → pod.

This is the chain I run in production, so it is the one I tested. Many teams run something close to this, with a CDN in front and a proxy on the host.

Step 1: Installing a retired project produces no warning 

bash
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=NodePort \
  --set controller.service.nodePorts.http=30080 \
  --set controller.service.nodePorts.https=30443
Enter fullscreen mode Exit fullscreen mode 
NAME: ingress-nginx
STATUS: deployed
REVISION: 1
Enter fullscreen mode Exit fullscreen mode

The chart repository still answers, the install succeeds, and no step mentions that the project is over. Automation makes this worse: a pipeline that installs this chart keeps working, and nobody reads the output anyway.

Step 2: Three layers, three wrong IPs

The test application is traefik/whoami, which prints the IP it sees and every header it receives.

Request from an external machine, through Cloudflare:

RemoteAddr: 10.42.0.6:36848
Cf-Connecting-Ip: 203.0.113.5
X-Forwarded-For: 10.42.0.1
X-Real-Ip: 10.42.0.1
X-Original-Forwarded-For: 203.0.113.5
X-Forwarded-Proto: http
Cf-Visitor: {"scheme":"https"}
Enter fullscreen mode Exit fullscreen mode

HAProxy log for the same request:

162.158.210.222:14219 [08/Sep/2026:08:54:27] fe_main~ be_ingress/ingress 200
Enter fullscreen mode Exit fullscreen mode

ingress-nginx access log for the same request:

10.42.0.1 - - [08/Sep/2026:08:54:27 +0000] "GET / HTTP/1.1" 200
Enter fullscreen mode Exit fullscreen mode
Layer IP it recorded Correct?
HAProxy (host) Cloudflare edge address No
ingress-nginx CNI gateway No
Pod (RemoteAddr) ingress pod No
CF-Connecting-IP real client Yes

Every hop rewrites the source address, so by the time the request reaches the pod, the original client is three layers away. Notice that the real address is not gone. ingress-nginx moved it to X-Original-Forwarded-For and wrote its own address into the header your application actually reads. The same request also arrives with X-Forwarded-Proto set to http, even though the client connected over HTTPS. This breaks redirect loops, secure cookies, and any absolute URL the application builds.

Step 3: Fixing the chain

Cloudflare IP list 

bash
{ curl -s https://www.cloudflare.com/ips-v4; echo; \
  curl -s https://www.cloudflare.com/ips-v6; echo; } \
  | sudo tee /etc/haproxy/cloudflare-ips.lst > /dev/null
Enter fullscreen mode Exit fullscreen mode

If you run these two curl commands one after another, the last IPv4 range and the first IPv6 range end up on the same line. HAProxy then refuses to load the config, and the error message shows the two ranges stuck together. Adding an echo between the two commands fixes it.

'131.0.72.0/222400:cb00::/32' is not a valid IPv4 or IPv6 address
Enter fullscreen mode Exit fullscreen mode

HAProxy configuration 

frontend fe_main
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/lab.pem

    acl from_cloudflare src -f /etc/haproxy/cloudflare-ips.lst
    http-request deny unless from_cloudflare

    http-request set-header X-Forwarded-For %[req.hdr(CF-Connecting-IP)] if from_cloudflare
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Forwarded-Proto http  if !{ ssl_fc }

    default_backend be_ingress
Enter fullscreen mode Exit fullscreen mode

I did not use option forwardfor here, because it appends the address of the connecting peer, which in this setup is a Cloudflare edge server. Instead I set X-Forwarded-For from the CF-Connecting-IP header, which Cloudflare adds and which holds a single address rather than a chain. The deny rule rejects any request that does not come from a Cloudflare address, because a header is only evidence when it comes from a party you trust.

Controller configuration 

bash
helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --reuse-values \
  --set controller.config.use-forwarded-headers="true" \
  --set controller.config.compute-full-forwarded-for="true" \
  --set controller.config.proxy-real-ip-cidr="10.42.0.0/16"
Enter fullscreen mode Exit fullscreen mode

Result:

X-Forwarded-For: 203.0.113.5, 10.42.0.1
X-Real-Ip: 203.0.113.5
X-Forwarded-Proto: https
Enter fullscreen mode Exit fullscreen mode

Step 4: A correct output does not mean a correct configuration

I want to describe a mistake I made here, because the output gave me no reason to look for it. haproxy -c had rejected the file because of the IP list bug from the previous section, so the reload failed and the old config stayed live. One of the two changes I made was doing all the work, and it was not the one I was looking at. On its own, that setting tells the controller to trust the incoming X-Forwarded-For without checking where it came from. A correct-looking output is not a verification. It only tells you that something produced the right value, not what.

Step 5: Proving the gap 

bash
curl -s -H "Host: lab.example.com" \
     -H "X-Forwarded-For: 198.51.100.99" \
     http://192.0.2.10/
Enter fullscreen mode Exit fullscreen mode 
X-Forwarded-For: 198.51.100.99, 10.42.0.1
X-Real-Ip: 198.51.100.99
Enter fullscreen mode Exit fullscreen mode

The forged address in the ingress-nginx access log:

198.51.100.99 - - [08/Sep/2026:09:02:34 +0000] "GET / HTTP/1.1" 200
Enter fullscreen mode Exit fullscreen mode

I sent a request straight to the origin IP, skipping Cloudflare, with an X-Forwarded-For header I made up. Both the header and the access log recorded 198.51.100.99, an address that belongs to nobody and had nothing to do with the machine sending the request. A wrong header is a bug. A wrong log line is a security control pointed at the wrong target. The fix is not a better header. It is restricting who is allowed to write one, which is what the deny rule from the previous section does.

With the deny rule in place, the same request:

HTTP/1.1 403 Forbidden
Request forbidden by administrative rules.
Enter fullscreen mode Exit fullscreen mode

Step 6: The migration 

bash
helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update

helm install haproxy-ingress haproxytech/kubernetes-ingress \
  --namespace haproxy-controller --create-namespace \
  --set controller.service.type=NodePort \
  --set controller.service.nodePorts.http=31080 \
  --set controller.service.nodePorts.https=31443
Enter fullscreen mode Exit fullscreen mode 
NAME      CONTROLLER
haproxy   haproxy.org/ingress-controller/haproxy
nginx     k8s.io/ingress-nginx
Enter fullscreen mode Exit fullscreen mode

I installed the HAProxy controller next to the existing one instead of replacing it, on a different node port. This is what makes the rest of this article possible: every difference below comes from sending one identical request to two ports.

A note on bulk conversion

I converted the existing Ingress with a sed command that renamed the resource and switched the ingress class. The name pattern was not specific enough. It rewrote the resource name and the backend service name in the same pass, and only one of those was supposed to change. A 404 after a migration looks like a routing rule you got wrong, so that is where you start looking, and that is not where it is.

lab.example.com
  /   whoami-haproxy:80 (<error: services "whoami-haproxy" not found>)
Enter fullscreen mode Exit fullscreen mode

Step 7: What disappeared

Same request, same forged header, sent directly to each controller.

ingress-nginx (port 30080)

X-Forwarded-For: 198.51.100.99, 10.42.0.1
X-Forwarded-Host: lab.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Original-Forwarded-For: 198.51.100.99
X-Real-Ip: 198.51.100.99
Enter fullscreen mode Exit fullscreen mode

HAProxy controller (port 31080)

X-Forwarded-For: 198.51.100.99
X-Forwarded-For: 10.42.0.1
Enter fullscreen mode Exit fullscreen mode

These three ConfigMap settings existed before the migration and were carried over by nothing:

yaml
controller:
  config:
    use-forwarded-headers: true
    compute-full-forwarded-for: true
    proxy-real-ip-cidr: 10.42.0.0/16
Enter fullscreen mode Exit fullscreen mode

X-Real-Ip, X-Forwarded-Host and X-Forwarded-Port are simply not there anymore. The X-Forwarded-For header did not disappear, which is worse. It arrives twice now, as two separate lines instead of one comma-separated value. Which value your application ends up using now depends on its HTTP library, and different frameworks make different choices here. Those three settings were the entire reason the client IP was correct before. The HAProxy controller has no equivalent for any of them, and the migration did not mention that. None of this produced an error. The site loads, the controller is Running, every check in a normal migration runbook passes, and the client IP is wrong.

The header that came back for the wrong reason

Through the full chain:

X-Forwarded-Proto: https
X-Forwarded-For: 203.0.113.5
X-Forwarded-For: 10.42.0.1
Enter fullscreen mode Exit fullscreen mode

Through the full chain X-Forwarded-Proto is present again, but the controller is not the one setting it. Most setups do not have that layer, because a cloud load balancer connects straight to the controller, and there the header is gone entirely. So where you run the test changes what you see, and testing through the full chain can hide a regression that a simpler setup would expose.

Step 8: Rolling back 

bash
sudo sed -i 's/127.0.0.1:31080/127.0.0.1:30080/' /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
Enter fullscreen mode Exit fullscreen mode

Because both controllers are still running, going back is a port number and a reload, not a reinstall. This is why I kept the old controller instead of deleting it. The cost is one unused node port; the benefit is that every step is reversible while you are still learning what changed.

When this does not apply

No CDN at all means one less hop and one less place to get this wrong. With a cloud load balancer there is no HAProxy on the host, so the trust boundary moves to the load balancer and its own IP ranges. If you use PROXY protocol instead of headers, none of this applies. The client address arrives at the TCP layer and there is no header to forge. This was a single-node cluster, so the CNI addresses you see here will be different on a multi-node setup.

The migration itself is easy. Finding out what changed afterwards is the actual work.

Top comments (0)