What Silently Breaks When You Migrate from Ingress NGINX to HAProxy

Onur Bolatoğlu — Tue, 08 Sep 2026 12:20:37 +0000

Introduction

ingress-nginx was retired in March 2026. There are no more releases and no more security patches. It routes traffic into roughly half of all Kubernetes clusters, which makes this a problem for a lot of teams. Installing a retired project is still completely silent. I tried it last week and got no warning at any step. Most migration guides treat this as an annotation mapping exercise. But the real problem is not the annotations. It is everything that keeps working afterwards while quietly doing the wrong thing.

The environment

Everything below was tested on a single-node cluster:

Ubuntu 26.04 LTS (kernel 7.0.0-31)
k3s v1.36.4+k3s1
Helm v3.21.4
HAProxy 3.2.9 (host, LTS branch)
ingress-nginx chart 4.15.1 / app 1.15.1  (retired March 2026)
HAProxy Kubernetes Ingress Controller (HAProxy 3.2.23)

Traffic path: Cloudflare (proxied) → HAProxy on the host → ingress controller → pod.

This is the chain I run in production, so it is the one I tested. Many teams run something close to this, with a CDN in front and a proxy on the host.

Step 1: Installing a retired project produces no warning

bash
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=NodePort \
  --set controller.service.nodePorts.http=30080 \
  --set controller.service.nodePorts.https=30443

NAME: ingress-nginx
STATUS: deployed
REVISION: 1

The chart repository still answers, the install succeeds, and no step mentions that the project is over. Automation makes this worse: a pipeline that installs this chart keeps working, and nobody reads the output anyway.

Step 2: Three layers, three wrong IPs

The test application is traefik/whoami, which prints the IP it sees and every header it receives.

Request from an external machine, through Cloudflare:

RemoteAddr: 10.42.0.6:36848
Cf-Connecting-Ip: 203.0.113.5
X-Forwarded-For: 10.42.0.1
X-Real-Ip: 10.42.0.1
X-Original-Forwarded-For: 203.0.113.5
X-Forwarded-Proto: http
Cf-Visitor: {"scheme":"https"}

HAProxy log for the same request:

162.158.210.222:14219 [08/Sep/2026:08:54:27] fe_main~ be_ingress/ingress 200

ingress-nginx access log for the same request:

10.42.0.1 - - [08/Sep/2026:08:54:27 +0000] "GET / HTTP/1.1" 200

Layer	IP it recorded	Correct?
HAProxy (host)	Cloudflare edge address	No
ingress-nginx	CNI gateway	No
Pod (`RemoteAddr`)	ingress pod	No
`CF-Connecting-IP`	real client	Yes

Every hop rewrites the source address, so by the time the request reaches the pod, the original client is three layers away. Notice that the real address is not gone. ingress-nginx moved it to X-Original-Forwarded-For and wrote its own address into the header your application actually reads. The same request also arrives with X-Forwarded-Proto set to http, even though the client connected over HTTPS. This breaks redirect loops, secure cookies, and any absolute URL the application builds.

Step 3: Fixing the chain

Cloudflare IP list

bash
{ curl -s https://www.cloudflare.com/ips-v4; echo; \
  curl -s https://www.cloudflare.com/ips-v6; echo; } \
  | sudo tee /etc/haproxy/cloudflare-ips.lst > /dev/null

If you run these two curl commands one after another, the last IPv4 range and the first IPv6 range end up on the same line. HAProxy then refuses to load the config, and the error message shows the two ranges stuck together. Adding an echo between the two commands fixes it.

'131.0.72.0/222400:cb00::/32' is not a valid IPv4 or IPv6 address

HAProxy configuration

frontend fe_main
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/lab.pem

    acl from_cloudflare src -f /etc/haproxy/cloudflare-ips.lst
    http-request deny unless from_cloudflare

    http-request set-header X-Forwarded-For %[req.hdr(CF-Connecting-IP)] if from_cloudflare
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Forwarded-Proto http  if !{ ssl_fc }

    default_backend be_ingress

I did not use option forwardfor here, because it appends the address of the connecting peer, which in this setup is a Cloudflare edge server. Instead I set X-Forwarded-For from the CF-Connecting-IP header, which Cloudflare adds and which holds a single address rather than a chain. The deny rule rejects any request that does not come from a Cloudflare address, because a header is only evidence when it comes from a party you trust.

Controller configuration

bash
helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --reuse-values \
  --set controller.config.use-forwarded-headers="true" \
  --set controller.config.compute-full-forwarded-for="true" \
  --set controller.config.proxy-real-ip-cidr="10.42.0.0/16"

Result:

X-Forwarded-For: 203.0.113.5, 10.42.0.1
X-Real-Ip: 203.0.113.5
X-Forwarded-Proto: https

Step 4: A correct output does not mean a correct configuration

I want to describe a mistake I made here, because the output gave me no reason to look for it. haproxy -c had rejected the file because of the IP list bug from the previous section, so the reload failed and the old config stayed live. One of the two changes I made was doing all the work, and it was not the one I was looking at. On its own, that setting tells the controller to trust the incoming X-Forwarded-For without checking where it came from. A correct-looking output is not a verification. It only tells you that something produced the right value, not what.

Step 5: Proving the gap

bash
curl -s -H "Host: lab.example.com" \
     -H "X-Forwarded-For: 198.51.100.99" \
     http://192.0.2.10/

X-Forwarded-For: 198.51.100.99, 10.42.0.1
X-Real-Ip: 198.51.100.99

The forged address in the ingress-nginx access log:

198.51.100.99 - - [08/Sep/2026:09:02:34 +0000] "GET / HTTP/1.1" 200

I sent a request straight to the origin IP, skipping Cloudflare, with an X-Forwarded-For header I made up. Both the header and the access log recorded 198.51.100.99, an address that belongs to nobody and had nothing to do with the machine sending the request. A wrong header is a bug. A wrong log line is a security control pointed at the wrong target. The fix is not a better header. It is restricting who is allowed to write one, which is what the deny rule from the previous section does.

With the deny rule in place, the same request:

HTTP/1.1 403 Forbidden
Request forbidden by administrative rules.

Step 6: The migration

bash
helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update

helm install haproxy-ingress haproxytech/kubernetes-ingress \
  --namespace haproxy-controller --create-namespace \
  --set controller.service.type=NodePort \
  --set controller.service.nodePorts.http=31080 \
  --set controller.service.nodePorts.https=31443

NAME      CONTROLLER
haproxy   haproxy.org/ingress-controller/haproxy
nginx     k8s.io/ingress-nginx

I installed the HAProxy controller next to the existing one instead of replacing it, on a different node port. This is what makes the rest of this article possible: every difference below comes from sending one identical request to two ports.

A note on bulk conversion

I converted the existing Ingress with a sed command that renamed the resource and switched the ingress class. The name pattern was not specific enough. It rewrote the resource name and the backend service name in the same pass, and only one of those was supposed to change. A 404 after a migration looks like a routing rule you got wrong, so that is where you start looking, and that is not where it is.

lab.example.com
  /   whoami-haproxy:80 (<error: services "whoami-haproxy" not found>)

Step 7: What disappeared

Same request, same forged header, sent directly to each controller.

ingress-nginx (port 30080)

X-Forwarded-For: 198.51.100.99, 10.42.0.1
X-Forwarded-Host: lab.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Original-Forwarded-For: 198.51.100.99
X-Real-Ip: 198.51.100.99

HAProxy controller (port 31080)

X-Forwarded-For: 198.51.100.99
X-Forwarded-For: 10.42.0.1

These three ConfigMap settings existed before the migration and were carried over by nothing:

yaml
controller:
  config:
    use-forwarded-headers: true
    compute-full-forwarded-for: true
    proxy-real-ip-cidr: 10.42.0.0/16

X-Real-Ip, X-Forwarded-Host and X-Forwarded-Port are simply not there anymore. The X-Forwarded-For header did not disappear, which is worse. It arrives twice now, as two separate lines instead of one comma-separated value. Which value your application ends up using now depends on its HTTP library, and different frameworks make different choices here. Those three settings were the entire reason the client IP was correct before. The HAProxy controller has no equivalent for any of them, and the migration did not mention that. None of this produced an error. The site loads, the controller is Running, every check in a normal migration runbook passes, and the client IP is wrong.

The header that came back for the wrong reason

Through the full chain:

X-Forwarded-Proto: https
X-Forwarded-For: 203.0.113.5
X-Forwarded-For: 10.42.0.1

Through the full chain X-Forwarded-Proto is present again, but the controller is not the one setting it. Most setups do not have that layer, because a cloud load balancer connects straight to the controller, and there the header is gone entirely. So where you run the test changes what you see, and testing through the full chain can hide a regression that a simpler setup would expose.

Step 8: Rolling back

bash
sudo sed -i 's/127.0.0.1:31080/127.0.0.1:30080/' /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy

Because both controllers are still running, going back is a port number and a reload, not a reinstall. This is why I kept the old controller instead of deleting it. The cost is one unused node port; the benefit is that every step is reversible while you are still learning what changed.

When this does not apply

No CDN at all means one less hop and one less place to get this wrong. With a cloud load balancer there is no HAProxy on the host, so the trust boundary moves to the load balancer and its own IP ranges. If you use PROXY protocol instead of headers, none of this applies. The client address arrives at the TCP layer and there is no header to forge. This was a single-node cluster, so the CNI addresses you see here will be different on a multi-node setup.

The migration itself is easy. Finding out what changed afterwards is the actual work.

DEV Community: Onur Bolatoğlu