DEV Community: Kazu

Kubernetes: iptables is empty after migrating to Cilium — what to look at next

Kazu — Tue, 15 Sep 2026 13:03:00 +0000

A Kubernetes Service stops responding. The first move is always the same: get on the node and run iptables-save | grep KUBE-SERVICES to see which Pod a given Service address maps to.

kube-proxy wrote those rules. It's a standard Kubernetes component that runs on every node and keeps Service forwarding rules registered in iptables. Looking up where a Service routes means reading those rules.

Cilium is a Kubernetes CNI plugin — the piece that wires up Pod networking so Pods can talk to each other. Flannel and Calico fill the same role; you pick one per cluster.

Cilium has a mode that goes beyond Pod networking and also handles Service forwarding, the job kube-proxy used to own. That mode is called kube-proxy replacement. With kubeProxyReplacement=true, Cilium runs on the assumption that kube-proxy is absent. Cilium itself does not remove kube-proxy; on an existing cluster you have to remove it separately. The test environment in this article is built with kind's kubeProxyMode: none, so kube-proxy was never present.

On a node in that Cilium environment, run the same iptables-save | grep KUBE-SERVICES from before. Nothing comes back. The same setup on a kube-proxy cluster returns 13 lines; here it returns 0. KUBE-SERVICES is the chain kube-proxy creates — the entry point for packets destined to a ClusterIP. It doesn't exist.

Broaden the grep to KUBE- and you get a few lines back, but they're KUBE-FIREWALL and KUBE-KUBELET-CANARY, chains created by kubelet that have nothing to do with Service forwarding. The nat table itself has some rules; it isn't empty. Only the Service forwarding rules are missing.

You try a different table, a different node. KUBE-SERVICES is nowhere, and yet the application is up and Services are reachable.

Nothing is broken. The information just isn't in iptables.

This article covers three things: where Cilium stores the forwarding rules that used to live in iptables; why tcpdump on a ClusterIP produces nothing and conntrack -L returns no matches; and what to check, in what order, when connectivity fails.

The test setup runs two kind clusters side by side — one with kube-proxy, one with Cilium — and runs the same commands against the same Service on both. All output shown is from real machines.

All tests run on kind v0.27.0, node image kindest/node:v1.32.2, Cilium 1.17.3, cilium-cli v0.20.0, kernel 6.8. Both clusters are single-node; multi-node-only behaviors are out of scope. ClusterIPs change each time a cluster is recreated, so IPs may differ between sections.

The reproduction repo is at https://github.com/shinagawa-web/kubernetes-cilium-kube-proxy-replacement-debug-guide

Where forwarding rules live, and when they get written

kube-proxy

Rules were registered in netfilter

A Service's ClusterIP doesn't exist on any machine. When a Pod sends a packet to it, the destination gets rewritten to a real Pod IP somewhere in the forwarding path. kube-proxy was what registered those rewrite rules in netfilter. netfilter is the Linux kernel's packet-processing framework; iptables is the command for reading and writing its rules.

That's why iptables-save worked. The forwarding rule — which Pod a ClusterIP packet goes to — was stored as a text rule registered in the kernel, so dumping everything was enough.

Here's what the packet path looks like when a Pod connects to a ClusterIP:

client Pod
   |  sends packet destined for 10.96.111.7:80 (ClusterIP)
   v
network stack
   |
   v
netfilter nat table
   KUBE-SERVICES → KUBE-SVC-xxxx → KUBE-SEP-xxxx
   |  rewrites destination to 10.244.0.6:80 (Pod IP) here
   v
Pod

tcpdump could capture the pre-DNAT packet

In this path, there's a stretch before the rewrite where the packet still carries the ClusterIP as its destination. Running tcpdump on the node with the ClusterIP as the filter caught those packets. Capturing before and after the rewrite showed exactly which Pod IP the destination changed to.

iptables-save for the rules, tcpdump for the live packets. Those two tools covered Service forwarding.

Cilium

Cluster setup

To enable kube-proxy replacement, you disable kube-proxy at cluster creation and install Cilium as the CNI. The test repo does this with kind. If you already have Cilium running, skip ahead to "No kube-proxy means no chains."

First, the kind cluster config disables both kube-proxy and the default CNI:

# cluster/kind-cilium.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.32.2
networking:
  disableDefaultCNI: true
  kubeProxyMode: none

kubeProxyMode: none keeps kube-proxy off the nodes. disableDefaultCNI: true removes kindnet so Cilium can take the CNI slot.

After creating the cluster, install Cilium via Helm. kubeProxyReplacement=true starts Cilium in the mode where it owns Service forwarding as well.

helm install cilium cilium/cilium \
  --version 1.17.3 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost="${API_SERVER_IP}" \
  --set k8sServicePort="${API_SERVER_PORT}"

k8sServiceHost and k8sServicePort are the address Cilium uses to reach the Kubernetes API server. Without kube-proxy, you have to supply these explicitly. The full setup script is in scripts/setup-cilium.sh.

No kube-proxy means no chains

With this setup, kube-proxy doesn't exist. No process registers rules, so KUBE-SERVICES is never created. That's why the initial grep returned 0 lines.

The forwarding table moves to eBPF maps

The destination mapping moves to eBPF. eBPF lets you load programs into the kernel and run them at predefined hook points — on packet receive, on system call entry — without modifying the kernel itself or rebooting. The attached program can read and modify the data passing through that point.

eBPF programs are stateless across invocations — local variables don't persist between calls. State lives in maps instead: a key-value store the kernel maintains. Programs read and write there. Cilium writes the ClusterIP-to-Pod-IP mapping into these maps; on each connection, the attached program looks up the map to decide where to forward.

The rewrite happens at connect()

If only the storage location changed, you'd just learn a new command. But there's one more difference: the rewrite happens earlier.

The same connection in a Cilium environment no longer goes through netfilter at all, and the rewrite point shifts:

client Pod
   |  calls connect() with 10.96.89.117:80 (ClusterIP) as the destination
   v
eBPF program attached to connect()
   |  rewrites destination to 10.0.0.183:80 (Pod IP)
   v
Pod

Cilium attaches an eBPF program to connect(), the system call where an application tells the kernel what it wants to connect to. At this point, not a single byte of data has been sent. The destination address gets replaced with the Pod IP right here.

The rewrite is done before any packet is constructed, so the resulting packet carries the Pod IP as its destination — no ClusterIP packet is ever created. That's why tcpdump on a network interface never shows the ClusterIP.

This applies to connections from within the cluster. Packets arriving from outside the cluster on a NodePort are handled differently, as covered in a later section.

Two kinds of "no output"

At this point there are two separate "nothing shows up" situations with different causes. iptables-save returning 0 lines is because the forwarding rules moved from iptables to eBPF maps. tcpdump returning 0 packets when filtering on a ClusterIP is because there are no packets with that ClusterIP as a destination in that path. The causes are different, so the replacement commands are different. The next section handles the first one.

Reading the forwarding table

The test environment has a single ClusterIP Service named demo, backed by a Deployment with 2 replicas. Here's how to read the forwarding table for that Service in each environment.

kube-proxy

Take another look at those 13 lines from iptables-save | grep KUBE-SERVICES.

:KUBE-SERVICES - [0:0]
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
:KUBE-SERVICES - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES -d 10.96.111.7/32 -p tcp -m comment --comment "default/demo cluster IP" -m tcp --dport 80 -j KUBE-SVC-73JNOU5FOFXIWZLE
-A KUBE-SERVICES -d 10.96.182.1/32 -p tcp -m comment --comment "default/demo-nodeport cluster IP" -m tcp --dport 80 -j KUBE-SVC-IECF2FXKD7A6IGZV
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS

The first 6 lines are chain definitions and hook points — no forwarding destination information. Lines 7 onward list each Service's entry. -d is the ClusterIP; -m comment --comment contains default/<ServiceName> cluster IP. Find the Service you want by that comment. demo is line 8 — "default/demo cluster IP" — and its ClusterIP is 10.96.111.7.

The -j KUBE-SVC-73JNOU5FOFXIWZLE at the end of that line is the jump to the backend-selection chain. Grep for that chain name to get the backend rules:

iptables-save -t nat | grep KUBE-SVC-73JNOU5FOFXIWZLE

-A KUBE-SVC-73JNOU5FOFXIWZLE ! -s 10.244.0.0/16 -d 10.96.111.7/32 -p tcp -m comment --comment "default/demo cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.6:80" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-4BM33MODP4BKIE3G
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.7:80" -j KUBE-SEP-C7KHQYF2QRUY23U4

The KUBE-SVC chain selects the backend. The first line — ! -s 10.244.0.0/16 — handles masquerading for traffic coming from outside the Pod CIDR; it's not part of backend selection. Lines 2 and 3 are the actual selection: the comment shows default/demo -> <Pod IP>:80 and --probability sets the distribution ratio. The -j KUBE-SEP-xxx at the end is the jump to the chain that executes the DNAT:

iptables-save -t nat | grep -E "KUBE-SEP-4BM33MODP4BKIE3G|KUBE-SEP-C7KHQYF2QRUY23U4"

-A KUBE-SEP-4BM33MODP4BKIE3G -p tcp -m comment --comment "default/demo" -m tcp -j DNAT --to-destination 10.244.0.6:80
-A KUBE-SEP-C7KHQYF2QRUY23U4 -p tcp -m comment --comment "default/demo" -m tcp -j DNAT --to-destination 10.244.0.7:80

--to-destination is the backend Pod IP and port.

Summary of how to read the forwarding table in a kube-proxy environment:

ClusterIP: the -A KUBE-SERVICES -d line (confirm the Service name in the comment)
Backend Pod IP: --to-destination in the corresponding KUBE-SEP chain
Weight: --probability in the KUBE-SVC chain

Cilium

First, get the ClusterIP from kubectl get svc:

kubectl get svc demo

NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
demo   ClusterIP   10.96.24.124    <none>        80/TCP    4s

ClusterIP is 10.96.24.124. cilium-dbg service list -o json shows the frontend (ClusterIP), Service name, and backend Pod IPs all at once:

kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | [
    (.spec.id | tostring),
    (.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
    .spec.flags.type,
    (.spec.flags.namespace + "/" + .spec.flags.name),
    (.spec["backend-addresses"] // [] | map(.ip + ":" + (.port|tostring)) | join(" "))
  ] | @tsv' | column -t

1   10.96.0.1:443      ClusterIP  default/kubernetes       172.18.0.2:6443
5   10.96.0.10:53      ClusterIP  kube-system/kube-dns     10.0.0.88:53     10.0.0.192:53
12  10.96.24.124:80    ClusterIP  default/demo             10.0.0.40:80     10.0.0.8:80
2   0.0.0.0:30080      NodePort   default/demo-nodeport    10.0.0.113:80    10.0.0.195:80
...

The 10.96.24.124 from kubectl get svc matches ID 12. The last column shows the backends: 10.0.0.40:80 and 10.0.0.8:80. The three-hop walk through KUBE-SERVICES → KUBE-SVC → KUBE-SEP in kube-proxy fits in a single line here.

In a multi-node cluster, kubectl -n kube-system exec ds/cilium execs into whichever Pod in the DaemonSet gets scheduled. In a multi-node cluster you only see the local state of that agent. The same applies to hubble observe flows — traffic on other nodes won't appear. If you know which node your target Pod is on, check kubectl get pod -n kube-system -o wide | grep cilium for that node's cilium Pod name and exec into it explicitly with kubectl -n kube-system exec <PodName>.

Packet tracing

Comparing how far tcpdump gets you when chasing ClusterIP traffic in each environment.

kube-proxy

Running tcpdump on the node and filtering on the ClusterIP captured packets. DNAT happened in netfilter, so there was a stretch before the rewrite where packets actually carried the ClusterIP as their destination. That "bracket the DNAT with two tcpdump captures" technique only worked because of that structure.

kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "host 10.96.104.4"

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
11:38:19.980617 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [S], seq 3530419686, win 64240, options [mss 1460,sackOK,TS val 4270529724 ecr 0,nop,wscale 10], length 0
11:38:19.980665 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [S.], seq 2901899611, ack 3530419687, win 65160, options [mss 1460,sackOK,TS val 4170938121 ecr 4270529724,nop,wscale 10], length 0
11:38:19.980675 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1, win 63, options [nop,nop,TS val 4270529724 ecr 4170938121], length 0
11:38:19.980726 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [P.], seq 1:76, ack 1, win 63, options [nop,nop,TS val 4270529724 ecr 4170938121], length 75: HTTP: GET / HTTP/1.1
11:38:19.980741 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [.], ack 76, win 64, options [nop,nop,TS val 4170938121 ecr 4270529724], length 0
11:38:19.983523 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [P.], seq 1:1545, ack 76, win 64, options [nop,nop,TS val 4170938124 ecr 4270529724], length 1544: HTTP: HTTP/1.1 200 OK
11:38:19.983538 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1545, win 67, options [nop,nop,TS val 4170938124 ecr 4270529724], length 0
11:38:19.983641 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], seq 76, ack 1545, win 67, options [nop,nop,TS val 4270529727 ecr 4170938124], length 0
11:38:19.983883 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [F.], seq 1545, ack 77, win 64, options [nop,nop,TS val 4170938124 ecr 4270529727], length 0
11:38:19.983898 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1546, win 67, options [nop,nop,TS val 4270529727 ecr 4170938124], length 0

10 packets captured
10 packets received by filter
0 packets dropped by kernel

Packets to the ClusterIP (10.96.104.4) are captured. SYN through FIN, all there. DNAT happens in netfilter after this, so the packets with the ClusterIP destination exist in this stretch.

conntrack -L also shows the DNAT in a single entry:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- conntrack -L | grep "10.96.104.4"

tcp  6 118 TIME_WAIT src=10.244.0.5 dst=10.96.104.4 sport=54462 dport=80 src=10.244.0.6 dst=10.244.0.5 sport=80 dport=54462 [ASSURED] mark=0 use=1

dst=10.96.104.4 is the ClusterIP (outbound direction); src=10.244.0.6 is the backend Pod IP (return direction). Before and after the DNAT, on one line.

Cilium

Same experiment on the Cilium cluster. Run tcpdump while curling ClusterIP 10.96.89.117 from the client Pod:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "host 10.96.89.117"

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
0 packets captured
0 packets received by filter
0 packets dropped by kernel

The curl succeeds. Connectivity is there. The empty capture is not a filter mistake — there are simply no packets with that ClusterIP as their destination.

connect() rewrites the destination to a Pod IP before any packet is created, so whatever hits the network already has the Pod IP. The ClusterIP never reaches the network layer.

conntrack -L filtered on the ClusterIP also returns nothing. The conntrack table contains only Pod-IP entries, but since no packets with the ClusterIP pass through netfilter, there's nothing to match.

kubectl debug node/<node> -it --image=nicolaka/netshoot -- conntrack -L | grep "10.96.89.117"

No output.

Seeing the ClusterIP translation with Hubble

tcpdump can't show the ClusterIP. So how do you trace ClusterIP traffic in a Cilium environment? Hubble is Cilium's observability component — it surfaces eBPF events as flow records. The test repo's scripts/setup-cilium.sh installs with Hubble Relay enabled. Curl the ClusterIP from the client Pod and then run hubble observe:

kubectl -n kube-system exec ds/cilium -- hubble observe --last 30 | grep -E "client|demo"

--last 30 outputs the 30 most recent flows.

Sep  7 07:23:31.106: default/client (ID:52940) <> 10.96.89.117:80 (world) pre-xlate-fwd TRACED (TCP)
Sep  7 07:23:31.106: default/client (ID:52940) <> default/demo-864f5f87b9-rvr95:80 (ID:10947) post-xlate-fwd TRANSLATED (TCP)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: SYN)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Sep  7 07:23:31.110: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Sep  7 07:23:31.110: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Sep  7 07:23:31.110: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, FIN)

Focus on the first two lines. pre-xlate-fwd TRACED is the pre-translation flow, destination still recorded as 10.96.89.117:80 — the ClusterIP from kubectl get svc demo. post-xlate-fwd TRANSLATED is after, destination now the actual Pod demo-864f5f87b9-rvr95:80.

This confirms why tcpdump couldn't see the ClusterIP: the translation is done before the packet reaches the network. Hubble records both sides of the translation as eBPF events, so you can read "which ClusterIP became which Pod" directly.

NodePort is visible to tcpdump

Try NodePort on the same cluster and the result is different. Send a curl from outside the cluster to the NodePort (172.18.0.2:30080) while running tcpdump for that port on the node:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "port 30080"

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
07:23:24.736193 eth0  In  IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [S], seq 3175489113, win 64240, options [mss 1460,sackOK,TS val 194961069 ecr 0,nop,wscale 10], length 0
07:23:24.736306 eth0  Out IP 172.18.0.2.30080 > 172.18.0.1.52306: Flags [S.], seq 2271513508, ack 3175489114, win 64308, options [mss 1410,sackOK,TS val 855246694 ecr 194961069,nop,wscale 10], length 0
07:23:24.736333 eth0  In  IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [.], ack 1, win 63, options [nop,nop,TS val 194961069 ecr 855246694], length 0
07:23:24.736421 eth0  In  IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [P.], seq 1:80, ack 1, win 63, options [nop,nop,TS val 194961069 ecr 855246694], length 79
...
10 packets captured
10 packets received by filter
0 packets dropped by kernel

10 packets captured. The difference is the path. NodePort packets arrive from outside the cluster as real packets, so they're not subject to socket LB. Processing happens at the tc layer (TCX), after the NIC, which is why tcpdump sees them.

cilium-dbg status --verbose shows which layers are active:

kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg status --verbose | grep -E "Attach Mode|Socket LB|XDP Acceleration"

Attach Mode:            TCX
  Socket LB:              Enabled
  XDP Acceleration:       Disabled

Socket LB: Enabled means Cilium rewrites destinations at the socket layer (the connect() hook). ClusterIP traffic gets translated to a Pod IP there, before any packet is created — which is why tcpdump never sees it.

XDP Acceleration: Disabled means NIC-driver-level (XDP) processing is off. XDP requires a compatible NIC, so it's disabled in kind by default. Instead, as Attach Mode: TCX shows, BPF programs are attached at the tc (Traffic Control) layer. NodePort traffic is real packets from outside the cluster, not subject to socket LB, so it gets processed at that tc layer and passes through the NIC — making it visible to tcpdump.

Packet tracing summary

	kube-proxy	Cilium
tcpdump on ClusterIP	captures (packet exists before DNAT)	nothing (translated at connect())
conntrack on ClusterIP	captures (DNAT before/after on one line)	nothing (doesn't go through netfilter)
tcpdump on NodePort	captures	captures (processed at tc layer)
translation visibility	tcpdump before/after DNAT	Hubble pre-xlate-fwd / post-xlate-fwd

E2E debugging flow

Starting from "Service is unreachable," walk through the investigation steps for a Cilium environment. By the end of this section, you'll have a clear path from symptom to cause.

The test repo's manifests/broken/ has two failure patterns: svc-wrong-selector.yaml (a Service with a selector that matches no Pods) and netpol-deny.yaml (a NetworkPolicy that blocks the client). The full debug script is in tests/debug-cilium.sh.

Case A — selector mismatch

Apply manifests/broken/svc-wrong-selector.yaml. The demo-broken Service has selector app: demo-v2, but every Pod in the cluster has the label app: demo. Nothing matches.

Check the ClusterIP first:

kubectl get svc demo-broken

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
demo-broken   ClusterIP   10.96.171.11    <none>        80/TCP    4s

Send a curl from the client Pod:

kubectl exec -n default client -- \
  curl -s --max-time 5 -o /dev/null -w '%{http_code} (%{time_total}s)' http://10.96.171.11/

000 (0.000063s)

%{http_code} returning 000 means no HTTP response. %{time_total} shows it returned immediately. When a Service has zero backends, Cilium's eBPF program rejects the connection right away — same speed as kube-proxy's REJECT --reject-with icmp-port-unreachable.

Step 1 — check Endpoints

Same command as before Cilium:

kubectl get endpoints demo-broken

NAME          ENDPOINTS   AGE
demo-broken   <none>      8s

<none> means no Pods are registered as backends. Compare against a working Service:

kubectl get endpoints demo

NAME   ENDPOINTS                     AGE
demo   10.0.0.117:80,10.0.0.119:80   34s

kubectl get endpoints works exactly the same as in a kube-proxy environment (from 1.33, kubectl get endpointslices is preferred).

Step 2 — check Cilium's internal state

The JSON output includes the Service name:

kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | [
    (.spec.id | tostring),
    (.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
    .spec.flags.type,
    (.spec.flags.namespace + "/" + .spec.flags.name),
    (.spec["backend-addresses"] | length | tostring)
  ] | @tsv' | column -t

12  10.96.171.11:80   ClusterIP  default/demo-broken  0
8   10.96.249.121:80  ClusterIP  default/demo         2

The last column is backend count. demo (ID 8) has 2; demo-broken (ID 12) has 0. A backend count of 0 indicates endpoint shortage. Cilium registers the Service but has nowhere to forward connections.

In a kube-proxy environment, the same situation shows up in the filter table with a comment:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- iptables-save -t filter | grep "10.96.171.11"

-A KUBE-SERVICES -d 10.96.171.11/32 -p tcp -m comment --comment "default/demo-broken has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable

has no endpoints is right there in the comment. Run that same command on the Cilium cluster and you get nothing. For endpoint-shortage cases (as opposed to NetworkPolicy), iptables stops explaining itself. Use cilium-dbg service list -o json | jq backend count instead. 0 means no endpoints.

Root cause

The most common reason Endpoints is empty is a selector mismatch between the Service and its Pods. Compare them directly:

kubectl get svc demo-broken -o jsonpath='{.spec.selector}'

{"app":"demo-v2"}

kubectl get pod -l app=demo -o jsonpath='{range .items[*]}{.metadata.name}  {.metadata.labels}{"\n"}{end}'

demo-864f5f87b9-dccbz  {"app":"demo","pod-template-hash":"864f5f87b9"}
demo-864f5f87b9-zzngq  {"app":"demo","pod-template-hash":"864f5f87b9"}

The selector points to app: demo-v2; the Pods carry app: demo.

Case B — NetworkPolicy deny

Apply manifests/broken/netpol-deny.yaml. The deny-client-to-demo NetworkPolicy allows Ingress to Pods with app: demo only from Pods labeled app: not-client. The client Pod doesn't match, so all connections to demo are denied.

kubectl exec -n default client -- \
  curl -s --max-time 5 -o /dev/null -w '%{http_code} (%{time_total}s)' http://10.96.249.121/

000 (5.001378s)

Same symptoms as Case A — but %{time_total} is 5 seconds. Case A (no backends) returned in 0.000063 seconds. Here, --max-time was exhausted. Packets are reaching the destination; nothing is sending a response back. Endpoints tell a different story.

Step 1 — check Endpoints

kubectl get endpoints demo

NAME   ENDPOINTS                     AGE
demo   10.0.0.117:80,10.0.0.119:80   34s

Endpoints are healthy. That's the branch point from Case A. Empty Endpoints → check selector match and Pod readiness. Endpoints present → keep going.

Step 2 — confirm Cilium sees active backends

kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | select(.spec["frontend-address"].ip == "10.96.249.121") | [
    (.spec.id | tostring),
    (.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
    .spec.flags.type,
    (.spec.flags.namespace + "/" + .spec.flags.name),
    (.spec["backend-addresses"] | length | tostring)
  ] | @tsv' | column -t

8  10.96.249.121:80  ClusterIP  default/demo  2

Backend count is 2; Cilium knows about them. Verify the eBPF map itself agrees with what the agent reports:

kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list | grep "10.96.249.121"

10.96.249.121:80/TCP (0)    0.0.0.0:0 (8) (0) [ClusterIP, non-routable]
10.96.249.121:80/TCP (1)    10.0.0.52:80/TCP (8) (1)
10.96.249.121:80/TCP (2)    10.0.0.167:80/TCP (8) (2)

service list is the agent's view of what it configured. bpf lb list reads the eBPF maps directly. They usually agree; when they diverge, the agent reports healthy backends the datapath never received. Here they match — the forwarding table is correct. Something else is dropping the packets.

Step 3 — check Hubble for packet drops

Cilium enforces NetworkPolicy directly in eBPF, so nothing shows up in iptables. Use Hubble:

kubectl exec -n default client -- curl -s --max-time 5 http://10.96.249.121/ > /dev/null
kubectl -n kube-system exec ds/cilium -- hubble observe --verdict DROPPED --last 20 | grep -E "client|demo"

Sep  9 08:07:46.753: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Sep  9 08:07:46.753: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) Policy denied DROPPED (TCP Flags: SYN)
Sep  9 08:07:47.777: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Sep  9 08:07:47.777: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) Policy denied DROPPED (TCP Flags: SYN)

Policy denied DROPPED. A NetworkPolicy is blocking the connection. policy-verdict:none INGRESS DENIED is the same verdict from a different angle. Either of these in hubble observe --verdict DROPPED points to NetworkPolicy as the cause.

Root cause

kubectl get networkpolicy deny-client-to-demo -o jsonpath='{.spec.ingress}'

[{"from":[{"podSelector":{"matchLabels":{"app":"not-client"}}}]}]

Ingress is only allowed from Pods labeled app: not-client. The client Pod doesn't qualify.

Debug flow summary

The two cases point to this sequence:

Run kubectl get endpoints (or kubectl get endpointslices from 1.33). <none> → check selector match and Pod readiness.
If Endpoints exist, run cilium-dbg service list -o json | jq and check the backend count. 0 → Cilium doesn't see the backends.
If backends exist but there's no connectivity, run hubble observe --verdict DROPPED. Policy denied → NetworkPolicy is the cause.

Tool mapping from kube-proxy to Cilium:

Old tool (kube-proxy)	New tool (Cilium)	What it shows
`iptables-save -t nat \	grep KUBE-SERVICES`	`cilium-dbg service list -o json \
{% raw %}`iptables-save -t filter \	grep `	backend count from `cilium-dbg service list -o json \
{% raw %}`conntrack -L`	`cilium-dbg bpf ct list global`	Connection tracking table
`tcpdump`	`hubble observe`	Per-flow packet visibility (see "Packet tracing")
(no equivalent)	`hubble observe --verdict DROPPED`	NetworkPolicy denials

Wrap-up

iptables-save returning 0 lines was not a broken cluster. kube-proxy doesn't exist, so nothing writes forwarding rules there — they moved to eBPF maps.

To read the forwarding table, use cilium-dbg service list -o json | jq. One command gets you the ClusterIP, Service name, and backend Pod IPs. The three-hop KUBE-SERVICES → KUBE-SVC → KUBE-SEP walk collapses to one line.

Parts of the troubleshooting workflow carry over unchanged. kubectl get endpoints works fine in a Cilium environment. After that, check cilium-dbg service list -o json | jq for backend count, and if backends are present but connectivity still fails, check hubble observe --verdict DROPPED for NetworkPolicy denials. The has no endpoints comment that kube-proxy wrote in the filter table is gone, but a backend count of 0 means the same thing.

Appendix — iptables evaluation cost grows with endpoint count

Measured while scaling demo replicas from 1 to 16. The kube-proxy column shows the number of KUBE-SVC chain rules; the Cilium column shows LB map backend slots.

replicas   kube-proxy SVC rules   Cilium backend slots
1          2                      1
2          3                      2
4          5                      4
8          9                      8
16         17                     16

Both grow linearly with endpoint count. The difference is lookup cost. The table shows equal growth in entry count; the asymmetry is in how those entries are evaluated per new connection.

kube-proxy evaluates KUBE-SVC chain rules top to bottom to pick a backend. Each rule's --probability check either hits or falls through to the next. When the last endpoint is selected, every rule before it was evaluated first. With 16 replicas, up to 16 evaluations happen. This only applies to the first packet of a new connection — subsequent packets are handled by conntrack and skip the rules entirely. The impact shows up in environments with high new-connection rates and many Services.

-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.10:80" -m statistic --mode random --probability 0.06250000000 -j KUBE-SEP-...
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.11:80" -m statistic --mode random --probability 0.06666666688 -j KUBE-SEP-...
...
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.9:80"  -j KUBE-SEP-...

Cilium computes random() % N to get a slot number and does one map lookup to find the backend. Slot 0 is the service master entry; slots 1+ are backends. With 16 Pods, it's still one lookup. This slot format is cilium-dbg bpf lb list output — it reads the eBPF maps directly, not the agent's in-memory state.

kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list | grep "10.96.210.178"

10.96.210.178:80/TCP (0)    0.0.0.0:0 (8) (0) [ClusterIP, non-routable]
10.96.210.178:80/TCP (1)    10.0.0.207:80/TCP (8) (1)
10.96.210.178:80/TCP (2)    10.0.0.48:80/TCP (8) (2)
...
10.96.210.178:80/TCP (16)   10.0.0.187:80/TCP (8) (16)

Migrating from ingress-nginx to Gateway API: a three-category approach to annotations

Kazu — Tue, 08 Sep 2026 13:02:00 +0000

The Kubernetes project announced the retirement of ingress-nginx. ingress-nginx retires in March 2026. No more patches, no more updates. No successor was named — just a recommendation to migrate to Gateway API.

The reason is straightforward: not enough maintainers. For years, effectively one or two people had kept the project running in their spare time. After the retirement announcement, there were plans to develop a successor controller (InGate) in collaboration with the Gateway API community, but no contributors came forward, and InGate was retired as well. No successor was named because none could be named.

Existing deployments won't break overnight. Helm charts and container images remain usable. But security patches stop. When new vulnerabilities are found, they'll go unfixed. The system keeps running while the risk accumulates slowly.

ingress-nginx had a kind of self-evidence: "if you need HTTP in Kubernetes, use this." Gateway API dissolves that.

This article covers how to choose a Gateway API implementation and how to migrate ingress-nginx annotations by sorting them into three categories — with before/after manifests throughout.

What "migrate to Gateway API" actually means

Gateway API is a Kubernetes specification. It is not software.

In the Ingress era, HTTP routing configuration looked like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service
            port:
              number: 8080

Apply this YAML and it worked. Assuming ingress-nginx was installed in the cluster — but since "using Kubernetes Ingress" and "using ingress-nginx" were nearly synonymous, that distinction rarely came up. Spec and implementation mapped almost one-to-one.

Gateway API is different. Writing an HTTPRoute does nothing unless you have separate software installed that reads those resources and actually routes requests.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-route
spec:
  parentRefs:
  - name: my-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: my-service
      port: 8080

Apply this YAML without a Gateway controller installed in the cluster, and nothing happens. Gateway API is a spec defining what resources exist — it's not a process that handles traffic.

"Migrating to Gateway API" meant picking an implementation and installing it. The question of which software was actually running — obvious in the Ingress world — is now something you have to decide yourself.

Too many implementations

"Pick one" sounds simple, but the options are many. In the Envoy-based family alone: Envoy Gateway, kgateway, Istio, Contour, Cilium. Outside Envoy: Traefik, Kong, NGINX Gateway Fabric. Cloud providers each offer their own. And all of them claim "Gateway API conformant."

It's tempting to think: "if they're all conformant, they'll behave the same within the standard, and I can switch later." That doesn't hold.

Three reasons "conformant" isn't uniform

First, conformance levels vary. Gateway API features are divided into three tiers: Core, Extended, and Implementation-specific. Core is the minimum every implementation must support. Extended is "recommended but optional." Implementation-specific is the open space for vendor extensions. Claiming "Gateway API conformant" only requires Core support. Extended and above vary across implementations.

Second, the features you actually need in production tend to land in vendor extensions. Auth, rate limiting, circuit breaking — these commonly belong to Extended or Implementation-specific. Envoy Gateway uses a SecurityPolicy CRD; Istio uses AuthorizationPolicy. The moment you write one of those resources, you're locked into that implementation.

Third, there's no default. When ingress-nginx was around, there was a clear first choice. Now there isn't. Every team is being forced to make that choice simultaneously, without a settled answer.

The first pick carries weight. The comparison below shows why.

Comparing the major implementations

	Envoy Gateway	Istio	Traefik	NGINX Gateway Fabric
Data plane	Envoy	Envoy	Traefik	nginx
Service mesh support	Partial	Full	None	None
Migration ease from ingress-nginx	Good	Moderate	Good	Excellent
Extended feature coverage	Excellent	Excellent	Good	Good
Development momentum	Excellent	Excellent	Good	Moderate

Istio is the only one with full service mesh support — east-west traffic control and sidecar injection included. The other three have no mesh capabilities. NGINX Gateway Fabric scores highest on migration ease because it shares the same nginx data plane. Istio scores lower because installing the entire service mesh stack is a prerequisite. The difference in Extended feature coverage becomes concrete once you read the Category B before/after examples. For development momentum, Envoy Gateway (CNCF incubating) and Istio (CNCF graduated) lead, while NGINX Gateway Fabric has a lower release cadence.

No recommendation here — the right answer depends on whether you already have a service mesh, how you weigh the annotation rewrite cost, and whether your team has Envoy operational experience. But before picking an implementation, there's work to do: inventory your current ingress-nginx annotations.

Sorting annotations into three categories

The before/after manifests and validation scripts used in this article are at shinagawa-web/ingress-nginx-to-gateway-api. before/ holds the ingress-nginx manifests, after/category-a/ the standard Gateway API equivalents, and after/category-b-envoy-gateway/ and after/category-b-istio/ the implementation-specific ones.

Envoy Gateway and Istio were chosen for the Category B manifests because they have broad coverage of features that can't be expressed in the standard spec — external auth, rate limiting, and so on. Traefik and NGINX Gateway Fabric can handle basic routing just as well, but their vendor extensions are less mature. Putting Envoy-based (Envoy Gateway) next to service-mesh-capable (Istio) also makes the architectural difference visible. Links to the relevant files appear at the end of each category section so you can run them locally as you read.

The goal is to inventory every annotation you have and sort each into one of three categories. The approach differs fundamentally by category. Note that Category C is evaluated line by line within the annotation, not annotation by annotation. A single configuration-snippet can contain lines that fall into Category A and lines that require Category B.

Category A: migrates directly to standard Gateway API

rewrite

A rewrite is the process of rewriting the URL path before the request reaches the backend. If a client hits /api/hello and the app should receive /hello, that's a rewrite. In ingress-nginx, you'd use regex capture groups.

# before/02-rewrite.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rewrite
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: echo-v1
            port:
              number: 8080

The pattern /api(/|$)(.*) captures the second group ($2) and forwards it as the path. A request to /api/hello puts hello into $2, delivering /hello to the backend.

Gateway API expresses the same thing with the URLRewrite filter in HTTPRoute. (URLRewrite is an Extended-tier feature — not Core. Verify your chosen implementation supports it; the four implementations in this article's comparison table all do.)

# after/category-a/02-rewrite.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: rewrite
spec:
  parentRefs:
  - name: eg
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /
    backendRefs:
    - name: echo-v1
      port: 8080

Read it as: replace the /api prefix with /. No regex — prefix replacement is explicit.

canary

Canary deployment means routing a small percentage of traffic to a new version, then gradually increasing it rather than doing an immediate cutover. In ingress-nginx, this required two Ingress resources: a primary route to v1 and a canary route to v2, with the canary's weight set in an annotation.

# before/03-canary.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-primary
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo-v1
            port:
              number: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo-v2
            port:
              number: 8080

Two Ingress resources define the same host: app.example.com and path /. Normally that's a conflict, but ingress-nginx treats any Ingress with canary: "true" as a weighted companion to the other. canary-weight: "20" sends 20% of traffic to echo-v2.

One routing decision split across two resources is the problem. Reading canary-primary alone gives you no visibility into the actual traffic split.

Gateway API fits it into a single HTTPRoute.

# after/category-a/03-canary.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: canary
spec:
  parentRefs:
  - name: eg
  rules:
  - backendRefs:
    - name: echo-v1
      port: 8080
      weight: 80
    - name: echo-v2
      port: 8080
      weight: 20

List v1 and v2 in backendRefs with weight ratios. 80:20 sends 80% to v1, 20% to v2. One resource, full picture.

The conversions in this category can be automated with the Ingress2Gateway CLI (1.0 GA). Worth using to reduce manual work.

before/02-rewrite.yaml → after/category-a/02-rewrite.yaml, before/03-canary.yaml → after/category-a/03-canary.yaml

Category B: requires implementation-specific extensions

external auth

External auth delegates authentication decisions to a separate service rather than the application itself. On each incoming request, the proxy asks the auth service whether to allow it. Only when the auth service responds OK does the request proceed to the app. Because the auth logic lives outside the app, the same auth service can cover multiple apps.

In ingress-nginx, two annotation lines did it.

# before/04-auth.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: auth
  annotations:
    nginx.ingress.kubernetes.io/auth-url: http://auth-service.default.svc.cluster.local/auth
    nginx.ingress.kubernetes.io/auth-response-headers: X-Auth-User
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /protected
        pathType: Prefix
        backend:
          service:
            name: echo-v1
            port:
              number: 8080

auth-url is the auth service endpoint. auth-response-headers lists headers from the auth service response to forward to the backend. When the auth service returns 200, the specified headers are added to the upstream request. Two annotation lines cover delegation and response propagation.

The sample repository includes a mock auth service in app/auth-service.yaml — a Python server that returns 200 for Authorization: Bearer valid-token and 401 for everything else.

With Envoy Gateway, HTTPRoute alone has no place to write auth configuration. You create a separate SecurityPolicy CRD — an Envoy Gateway-specific resource — and bind it to the target HTTPRoute.

# after/category-b-envoy-gateway/04-auth.yaml (excerpt)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: auth
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: protected
  extAuth:
    http:
      backendRefs:
      - group: ""
        kind: Service
        name: auth-service
        port: 80
      headersToBackend:
      - x-auth-user

targetRefs specifies which HTTPRoute the policy applies to — here, the HTTPRoute named protected. extAuth.http.backendRefs points to the auth service.

The intent is clearer than in ingress-nginx. Annotations were scoped to their Ingress resource, so tracking which route had which auth policy meant reading every Ingress. SecurityPolicy stands as an independent resource with its target made explicit via targetRefs.

To forward X-Auth-User from the auth service response to the backend, headersToBackend must be configured explicitly. Unlike ingress-nginx's auth-response-headers, Envoy Gateway does not forward auth response headers automatically — only the headers listed in headersToBackend are propagated.

Migrating to Istio changes the architecture. AuthorizationPolicy evaluates request attributes locally — headers, JWT claims, source principals — rather than calling out to an external service.

# after/category-b-istio/04-auth.yaml (excerpt)
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: auth
spec:
  selector:
    matchLabels:
      app: echo
      version: v1
  action: DENY
  rules:
  - when:
    - key: request.headers[authorization]
      notValues:
      - "Bearer valid-token"

action: DENY means "deny requests that match the condition." Requests where request.headers[authorization] is not Bearer valid-token are denied — only valid-token requests get through. Note that selector.matchLabels scopes the policy to the workload, not to the /protected path — the DENY rule applies to all traffic reaching the matched pods. To restrict it to a specific path, add to.operation.paths to the rule. Where Envoy Gateway's extAuth asks an external service for an OK/NG decision, Istio's AuthorizationPolicy evaluates the policy inside the cluster. The mechanism is fundamentally different from ingress-nginx's auth-url callback model, so migrating to Istio means rethinking the auth design.

AuthorizationPolicy evaluates request attributes locally. For logic that requires an external decision — like checking permissions against a database — Istio provides AuthorizationPolicy with action: CUSTOM plus an extensionProviders entry in meshConfig. This is the proper external auth delegation path in Istio, comparable in ergonomics to Envoy Gateway's extAuth. The EnvoyFilter route can also be used but writes Envoy internals directly.

Category A works the same way regardless of which implementation you choose. Category B changes fundamentally depending on which one you picked. This is the "operational features drift into vendor extensions" reality the comparison table pointed to: you can't form a migration plan for this category until you've chosen an implementation.

before/04-auth.yaml → after/category-b-envoy-gateway/04-auth.yaml / after/category-b-istio/04-auth.yaml

rate limit

ingress-nginx's rate limit was self-contained in annotations, but there's a trap: queueing behavior changes after migration.

# before/05-rate-limit.yaml (excerpt)
annotations:
  nginx.ingress.kubernetes.io/limit-rps: "2"
  nginx.ingress.kubernetes.io/limit-burst-multiplier: "1"

limit-rps: "2" allows at most 2 requests per second per IP. Requests over the limit don't get rejected immediately — they queue up to the limit set by limit-burst-multiplier. With limit-burst-multiplier: "1", the queue cap is limit-rps × 1 = 2. Once the queue fills, subsequent requests get a 503 immediately.

Envoy Gateway uses BackendTrafficPolicy. Like SecurityPolicy for external auth, it's a separate resource from HTTPRoute, with targetRefs pointing at the target.

# after/category-b-envoy-gateway/05-rate-limit.yaml (excerpt)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: rate-limit
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: rate-limited
  rateLimit:
    type: Local
    local:
      rules:
      - limit:
          requests: 2
          unit: Second

requests: 2, unit: Second expresses 2 requests per second, but there's no field corresponding to limit-burst-multiplier. The queueing behavior can't be reproduced.

Istio configures the token bucket directly via EnvoyFilter. There's no abstraction layer like BackendTrafficPolicy — you're writing Envoy internals directly.

# after/category-b-istio/05-rate-limit.yaml (excerpt)
token_bucket:
  max_tokens: 2
  tokens_per_fill: 2
  fill_interval: 1s

tokens_per_fill: 2, fill_interval: 1s replenishes 2 tokens per second (= 2 req/s). max_tokens: 2 is the burst cap — the maximum number of tokens the bucket can hold. This corresponds to ingress-nginx's limit-rps: "2", limit-burst-multiplier: "1". The algorithm differs from nginx's queuing model, but the behavior — 2 requests per second, burst cap of 2 — is equivalent. Whether reproducing limit-burst-multiplier is worth trading against implementation choice is a judgment call that varies by team.

before/05-rate-limit.yaml → after/category-b-envoy-gateway/05-rate-limit.yaml / after/category-b-istio/05-rate-limit.yaml

Category C: no equivalent

configuration-snippet

Probably the most useful annotation ingress-nginx had — and the most problematic.

nginx config has server blocks and location blocks. configuration-snippet lets you inject nginx directives directly into the location block, overriding or extending what ingress-nginx generates. It was effectively an escape hatch for anything the standard annotations couldn't express.

# before/06-configuration-snippet.yaml (excerpt)
annotations:
  nginx.ingress.kubernetes.io/configuration-snippet: |
    more_set_headers "X-Custom-Header: my-value";
    more_set_headers "X-Request-ID: $request_id";

more_set_headers is a directive from the headers-more module bundled with ingress-nginx, used to add or override response headers. "X-Custom-Header: my-value" adds a static header. "X-Request-ID: $request_id" attaches nginx's built-in variable $request_id — a unique ID auto-generated per request — as a response header.

Gateway API has no direct equivalent. But saying "no equivalent" oversimplifies it. Break down the annotation line by line, and the picture splits.

Adding a static header like X-Custom-Header migrates to HTTPRoute's ResponseHeaderModifier filter. That's standard spec — Category A.

# after/category-a/06-response-header.yaml (excerpt)
filters:
- type: ResponseHeaderModifier
  responseHeaderModifier:
    add:
    - name: X-Custom-Header
      value: my-value

Dynamic values like $request_id are a different story. Nothing in the Gateway API standard covers referencing nginx built-in variables. With Envoy Gateway, you can reproduce it by injecting a Lua script via the EnvoyPatchPolicy CRD — Category B.

# after/category-b-envoy-gateway/06-request-id.yaml (excerpt)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: request-id-lua
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
  - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
    name: "default/eg/http"
    operation:
      op: add
      path: "/default_filter_chain/filters/0/typed_config/http_filters/0"
      value:
        name: envoy.filters.http.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          default_source_code:
            inline_string: |
              function envoy_on_request(request_handle)
                local req_id = request_handle:headers():get("x-request-id")
                if req_id then
                  request_handle:streamInfo():dynamicMetadata():set("lua_ns", "req_id", req_id)
                end
              end
              function envoy_on_response(response_handle)
                local meta = response_handle:streamInfo():dynamicMetadata():get("lua_ns")
                if meta and meta["req_id"] then
                  response_handle:headers():add("x-request-id", meta["req_id"])
                end
              end

Envoy generates an x-request-id header automatically on incoming requests. The Lua script reads it during request processing, stores it in metadata, then attaches it as a response header during response processing. nginx's $request_id becomes Envoy's request ID, but the intent — a unique per-request ID on the response header — is the same.

The same Lua logic works in Istio via EnvoyFilter, but the apply context and metadata namespace differ.

# after/category-b-istio/06-request-id.yaml (excerpt)
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: request-id-response-header
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.lua
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          default_source_code:
            inline_string: |
              function envoy_on_request(request_handle)
                local req_id = request_handle:headers():get("x-request-id")
                if req_id then
                  request_handle:streamInfo():dynamicMetadata():set("lua_metadata", "req_id", req_id)
                end
              end
              function envoy_on_response(response_handle)
                local meta = response_handle:streamInfo():dynamicMetadata():get("lua_metadata")
                if meta and meta["req_id"] then
                  response_handle:headers():add("x-request-id", meta["req_id"])
                end
              end

The Envoy Gateway version uses EnvoyPatchPolicy with JSONPatch to insert the Lua filter at the Listener level, with state stored under metadata namespace lua_ns. The Istio version uses EnvoyFilter with applyTo: HTTP_FILTER and match.context: GATEWAY, with state stored under lua_metadata. The Lua logic is the same, but the configuration layer differs.

What truly has no equivalent in Category C is nginx-specific functionality that EnvoyPatchPolicy can't reach — directives that depend deeply on nginx internals, or snippets that assume concepts Envoy doesn't have. In practice, most configuration-snippet contents, when broken down line by line, land in Category A or Category B. The annotation as a whole isn't Category C — each line has to be read and evaluated individually.

before/06-configuration-snippet.yaml → after/category-a/06-response-header.yaml (X-Custom-Header) / after/category-b-envoy-gateway/06-request-id.yaml (X-Request-ID, Envoy Gateway) / after/category-b-istio/06-request-id.yaml (X-Request-ID, Istio)

Standardization didn't make migration easier

Gateway API gave us a unified spec, so "the interface is now shared." But more implementations means more decisions. The Core tier is consistent across implementations. But operational features drift into vendor extensions, so locking in to an implementation comes after, not before.

The frustration with ingress-nginx — "settings are scattered across annotations, hard to see the structure" — was a real problem. Gateway API was designed to address it. But from the migration side, new decisions piled on: which software to pick, which annotations map to the standard and which need vendor extensions and which get dropped.

"Migrate to Gateway API" wasn't a work order. It compressed two separate jobs into one phrase: choose an implementation, then classify each annotation into one of three categories and handle each accordingly.

Inspect HTTPS Traffic Without a Proxy — Built with Go and eBPF

Kazu — Tue, 01 Sep 2026 12:01:00 +0000

I built a CLI in Go that reads HTTPS traffic as plaintext — no CA certificate installation, no proxy, and no changes to the target app.

The original motivation was simple: spinning up mitmproxy every time I needed to debug something added one too many steps. I found an approach using eBPF, wrapped it into a CLI, and called it tinytap. This article is about how I built it.

Here's what it looks like in action.

Two terminals side by side. The left sends one HTTPS request; the right shows tinytap printing it in plaintext — request line, headers, Authorization token, JSON body, all of it.

What it captures

tinytap targets apps like this — ordinary code that hits an API, in this case the OpenAI API from Python:

from openai import OpenAI

client = OpenAI(api_key="sk-...", base_url="https://127.0.0.1:8443/v1")

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "what is a uprobe"}],
)

And here's what tinytap captures and outputs:

POST /v1/chat/completions HTTP/1.1
Host: 127.0.0.1:8443
Authorization: Bearer sk-demo-xxxx
Content-Type: application/json
User-Agent: OpenAI/Python 2.53.0
...
Request body (decoded): {"messages":[{"role":"user","content":"what is a uprobe"}],"model":"gpt-4o-mini"}

Request line, all headers, the Authorization: Bearer token, and the JSON body. The token in the gif is sk-demo-xxxx — a dummy — but when you point tinytap at a real app, the real token shows up exactly like that. No reformatting.

Nothing on the app side was touched. One command: tinytap.

How eBPF makes this work

A uprobe is a kernel mechanism for dynamically hooking into user-space functions. tinytap attaches a uprobe to SSL_write in libssl. The moment that function is called, tinytap reads the argument buffer — which is still plaintext at that point. The data travels through a ring buffer to user space and gets displayed in the TUI. No modifications to the app or to libssl.

This is not breaking TLS

One thing worth clearing up.

tinytap only looks at the buffer your own process hands to the TLS library — the moment before it gets encrypted. It is not a vulnerability that lets you intercept traffic on the network. What flows over the wire is properly encrypted. Other processes and other people's traffic are not accessible.

App
  │  plaintext
  ▼
TLS library  ◀── tinytap looks here
  │  ciphertext
  ▼
Network
  │  ciphertext
  ▼
API server

It's the same idea as curl printing its own request to your terminal.

Why Go

A few reasons I chose Go.

First, cilium/ebpf. It's a mature library for managing eBPF programs from Go — it absorbs the boilerplate of talking to the kernel, so I could focus on the actual hook logic.

Second, binary distribution. go build produces a single binary you can hand to someone and it just runs. No runtime or external dependencies to install. That matters a lot when you're shipping an open-source CLI.

Third, the TUI. I used Bubble Tea to build a live event display. Even when a burst of eBPF events comes in from the kernel, Go's goroutines and channels handle concurrent ingestion and rendering without trouble.

Try it

tinytap uses eBPF and is Linux-only. On Mac or Windows, set up a Linux container with Docker.

curl -fsSL https://raw.githubusercontent.com/shinagawa-web/tinytap/main/scripts/install.sh | sh
sudo setcap cap_dac_read_search,cap_perfmon,cap_bpf,cap_sys_admin=eip $(command -v tinytap)
sudo chmod +x $(ldconfig -p | grep libssl.so | awk '{print $NF}' | head -1)
tinytap

setcap grants the binary the capabilities it needs for eBPF. The chmod +x step is needed on Debian and Ubuntu because libssl.so.3 ships without the execute bit set there. With both done, tinytap runs without sudo.

If something isn't working, run tinytap doctor. It checks your kernel version, BTF availability, capabilities, and the libssl execute bit, and tells you exactly what's missing.

$ tinytap doctor
tinytap doctor — v0.6.1

[OK      ] kernel version               6.1.0 (>= 5.8 required)
[OK      ] kernel BTF                   present
[OK      ] cap_dac_read_search          present
[OK      ] cap_perfmon                  present
[OK      ] cap_bpf                      present
[OK      ] cap_sys_admin                present
[OK      ] cap_syslog                   not needed on arm64
[OK      ] perf_event_paranoid          4
[OK      ] unprivileged_bpf_disabled    2
[OK      ] RLIMIT_MEMLOCK               soft=unlimited hard=unlimited
[OK      ] syscall tracepoints          available
[OK      ] BPF dry-run load             ok
[DEGRADED] libssl execute bit           /usr/lib/x86_64-linux-gnu/libssl.so.3 not set
    Affects: TLS capture only. Plaintext HTTP capture is unaffected.
    Fix:     sudo chmod +x /usr/lib/x86_64-linux-gnu/libssl.so.3

12 ok, 1 degraded, 0 blocking, 0 info

Python, Ruby, PHP, and Node.js all work. Go apps are out of scope — the standard library's crypto/tls doesn't go through OpenSSL, so there's no SSL_write to hook. tinytap currently understands HTTP/1.1 only.

Targeting implementations that bypass OpenSSL — like Go's crypto/tls or Rust's rustls — would require attaching the uprobe at a different layer. Where exactly that should be is still an open question.

Verified compatibility details are in the docs:

Server compatibility — how much of the response body tinytap can surface for nginx, Caddy, Uvicorn, and other servers
TLS compatibility — the list of supported TLS implementations; OpenSSL-based stacks (Python, nginx, curl, Node.js, etc.)
Platform support — tested Linux distributions and kernel versions; kernel 5.8 or later is required (Ubuntu 20.04's default kernel 5.4 is too old)

Wrapping up

I built a CLI in Go that reads HTTPS plaintext using eBPF uprobes — no CA certificate, no proxy needed. By hooking into SSL_write in libssl, it works with any OpenSSL-based stack: Python, Ruby, Node.js, and others, regardless of language.

Code is on GitHub. Environment requirements and compatibility details are in the docs.

Nginx, Caddy, Traefik, or HAProxy: How to Pick the Right Reverse Proxy for Your Stack (2026)

Kazu — Tue, 25 Aug 2026 13:02:00 +0000

Have you ever heard "Caddy is great" and wondered what actually makes it different from Nginx? Or maybe you were evaluating Traefik for a Kubernetes setup, went down a rabbit hole of comparison articles, and gave up halfway through?

Part 1 ("For People Who've Configured a Reverse Proxy but Can't Explain How It Works") covered how reverse proxies work under the hood. Part 2 — this article — focuses on tool selection: the design philosophies behind Nginx, Caddy, Traefik, and HAProxy, and a mental model for deciding which one fits your use case. If you haven't read Part 1, one baseline is enough: a reverse proxy is software that sits between the browser and the app server, relaying traffic between them.

The four tools and what each one is

A quick one-liner for each:

Nginx: the most widely used general-purpose web server and reverse proxy
Caddy: simple config file, automatic TLS certificate management built in
Traefik: designed for dynamic routing config in container environments
HAProxy: a full-featured load balancer that operates at both the TCP (L4) and HTTP (L7) levels

All four can do load balancing, reverse proxying, and TLS termination. Nginx can handle load balancing. HAProxy can work as a reverse proxy. Capability overlap is the point — what differs is design emphasis, not capability ceiling. That emphasis is what drives the choice.

Consider what "emphasis" means in practice. Nginx was built as a general-purpose machine: serve static files, reverse proxy, act as a load balancer — all from one piece of software. Traefik was born from a specific frustration: having to rewrite config files every time a container starts or stops. The starting point differs, so even when both do the same job, one is working in its natural mode and the other is being stretched.

Read this article through that lens: where does each tool's center of gravity sit?

One tool you won't see in this comparison is Envoy. Envoy has native support for circuit breaking, distributed rate limiting, and gRPC load balancing, and it's usually encountered in the context of service meshes like Istio. The configuration complexity and operational overhead are in a different league. For someone who's only used Nginx, it's too big a jump. Envoy does appear in the bonus section at the end — not as "here's the next step to consider," but as "here's what these four can't reach."

Nginx vs Caddy: what's actually different

Put the minimal HTTPS configuration for each side by side.

The Nginx version:

server {
    listen 443 ssl;
    server_name example.com;

    ssl_certificate /etc/ssl/certs/example.com.crt;
    ssl_certificate_key /etc/ssl/private/example.com.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The equivalent Caddyfile:

example.com {
    reverse_proxy localhost:3000
}

The line count difference might look like Caddy is skipping things — it isn't. With Caddy, TLS certificate retrieval and renewal happen automatically. It fetches a certificate from Let's Encrypt and renews it automatically well before the 90-day expiry, with no manual steps needed. There's no ssl_certificate line because you don't supply the cert file yourself. There are no proxy_set_header lines because Caddy forwards those headers by default.

If you've ever wrestled with Nginx TLS config, Caddy's automatic TLS will be a real relief. Especially if you've had a service go down because a certificate renewal slipped through the cracks.

That said, Caddy's simplicity cuts both ways. The convenient defaults make it harder to reason about what's happening when you need to override something. Nginx's verbosity is a feature: every directive is explicit, which makes troubleshooting more straightforward.

The question to ask when choosing: do you want to control TLS yourself, or do you want to hand it off?

Caddy does have cases where it doesn't fit. Internal services — those not exposed to the internet — won't work with ACME-based automatic TLS. Let's Encrypt challenges require the domain to be publicly reachable, so for a service on an internal DNS, you'd need extra DNS-01 challenge configuration to get Caddy's automatic TLS working. Going in with the assumption that "Caddy means TLS is automatic" will cause problems for internal services.

There's also a debugging gap. caddy reload and caddy validate error messages can be less specific than Nginx's. Nginx's nginx -t gives you the exact line number of the problem; Caddy sometimes only tells you "configuration is invalid." For a team with deep Nginx experience, troubleshooting an unfamiliar tool takes longer than it should.

Why Traefik pairs well with Kubernetes

Traefik's starting point was: "I don't want to rewrite config files every time a container is added."

With Nginx, you write this:

upstream backend {
    server app1:3000;
    server app2:3000;
}

Adding app3 means opening the config file, adding server app3:3000;, and reloading Nginx. Manual every time.

Traefik works differently. With Docker, you attach labels to the container at startup and it joins the routing automatically.

services:
  app:
    image: my-app
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`example.com`)"
      - "traefik.http.services.app.loadbalancer.server.port=3000"

The moment that container starts, Traefik reads the Docker labels and adds the route. When the container stops, the route disappears. No config reload needed.

In Kubernetes, the same thing happens through Ingress annotations. Deploy a new service with kubectl apply, and Traefik detects it and updates routing. The "update config files on every deploy across 20 containers" workflow goes away.

If static config files are enough, Nginx works fine. When containers come and go frequently, that dynamic config automation is the reason to choose Traefik.

Going deeper into Kubernetes: Traefik has its own CRD called IngressRoute. It's more expressive than the standard Ingress resource, but it means writing Traefik-specific config into your K8s manifests. Nginx Ingress Controller is built on the standard Ingress resource and uses annotations to control fine-grained behavior.

Which you choose depends on whether you'd rather write in the proxy's config language or stay close to K8s manifests. If your team already knows Nginx, the Nginx Ingress Controller has a lower learning curve. If your service churn is high and you want routing definitions managed as code inside K8s, Traefik's IngressRoute is where it earns its keep.

Why HAProxy gets chosen as a dedicated load balancer

Ask the question "Nginx can load balance too, so why choose HAProxy?" and the answer surfaces.

A health check in HAProxy config looks like this:

backend api_servers
    balance roundrobin
    option httpchk GET /health HTTP/1.1
    http-check expect status 200
    default-server inter 2s fall 3 rise 2 maxconn 30
    server app1 10.0.0.1:3000 check
    server app2 10.0.0.2:3000 check

inter 2s fall 3 rise 2 means: check every 2 seconds, mark down after 3 consecutive failures, mark up after 2 consecutive successes. Nginx can do something similar, but HAProxy lets you express this level of control directly in the config.

Where HAProxy really stands out is observability — seeing what's happening in real time. The built-in stats dashboard shows connection counts, error rates, and latency per backend, live. If one backend starts spiking in latency because it's running out of memory, you can dig through Nginx access logs to find the cause eventually. Opening HAProxy's dashboard makes it immediately obvious which backend has the elevated error rate. When you're handling high traffic and need to know instantly which backend is the bottleneck, that data matters.

HAProxy also handles both TCP (L4) and HTTP (L7) load balancing. Beyond HTTP apps, you can balance database connections and manage WebSocket traffic in config.

HAProxy makes sense when traffic is heavy and you need fine-grained control over backend behavior. Reaching for it on a simple reverse proxy setup means absorbing configuration complexity without much benefit.

Picking one for your situation

Mapping the discussion to four scenarios:

Scenario	Choice	Main reason
Simple VPS + TLS termination	Caddy	Automatic TLS removes certificate management overhead
Small Docker microservices (stable topology)	Nginx	Proven track record, large community, explicit config
Small Docker microservices (high churn)	Traefik	Routing updates automatically as containers start and stop
Kubernetes cluster	Traefik / Nginx Ingress	Depends on what the team already knows
High-traffic load balancer	HAProxy	Health check granularity and built-in observability

For a simple VPS with TLS termination, Caddy is the pick. Offloading certificate management is a meaningful gain, and the short config file is easy to maintain. It suits personal projects and small services.

For small Docker-based microservices, Nginx or Traefik are both candidates. If the container count is stable and doesn't change much, Nginx is plenty. If you're adding and removing services regularly, Traefik's dynamic config earns its place.

For Kubernetes, Traefik or Nginx Ingress Controller. Traefik has a head start on K8s integration, but Nginx Ingress Controller is mature and fully viable. If your team has deep Nginx knowledge, there's no good reason to switch to Traefik just because it's the trendy choice.

For a high-traffic load balancer, HAProxy. That's where its design philosophy — fine-grained control and first-class observability — actually comes into play.

My last personal choice was Caddy on a VPS. The reason was simple: I'd taken a service down once because I forgot to renew a certificate. Since switching to Caddy, I haven't thought about certificates once. That alone was worth it.

Config files and benchmark

Working examples for all four proxies are in shinagawa-web/reverse-proxy-bench — each proxy has its own directory with a config file and a docker-compose.yml. Run any of them locally with make bench-nginx (requires Docker and k6).

The benchmark runs 60 seconds at 20 concurrent virtual users against a minimal Go backend (/ping → 200 OK), with default configurations:

Proxy	req/s	p50 (ms)	p95 (ms)	p99 (ms)
HAProxy	10,762	1.53	3.45	4.89
Traefik	7,946	2.15	4.62	6.48
Caddy	7,535	2.28	4.93	6.81
Nginx	4,386	4.37	5.43	6.83

These results reflect default configurations on a 2-vCPU GitHub Actions runner. Nginx in particular has significant headroom — worker_processes auto alone would narrow the gap considerably. The intent isn't to declare a winner, but to show how the design emphasis of each tool translates into behavior under the same load.

Bonus: what these four can't do

For intermediate and above: the areas where all four have the same blind spot.

Distributed rate limiting

Nginx, Caddy, Traefik, and HAProxy all have rate limiting, but all of them apply limits per instance. Scale out to four instances and your limit effectively quadruples. A "100 requests per second" rule becomes "400 requests per second" in practice. If you hit a situation right after a scale-out where rate limiting suddenly seems broken, the connection to instance count isn't always obvious.

You can share state via an external store like Redis — each proxy instance increments a Redis counter on every request and returns 429 when the threshold is hit. That turns Redis into a single point of failure, so you also need an availability design for Redis. Every request that touches Redis also adds the round-trip latency — typically 0.5–2ms on a local network, but under traffic spikes Redis itself can become the bottleneck, slowing the proxy down across the board.

The tools that provide this natively are Kong and Envoy. Kong's Rate Limiting Advanced plugin uses a hybrid approach: local counters per instance, synchronized to a shared store on a configurable interval. Requests don't need a Redis write on every hit, which reduces the latency overhead significantly. Envoy is designed to integrate with a dedicated external rate limit service from the start.

Circuit breaking

Suppose an upstream service starts responding slowly. The proxy holds connections open until timeout, and new requests keep trying to connect. The connection pool fills with waiting connections, other backends start backing up too, and failures cascade through the whole system. That's a cascading failure.

A circuit breaker cuts that chain. When it detects consecutive failures to an upstream, it stops sending new requests there and returns errors immediately. Envoy has this natively via outlier_detection — setting consecutive_5xx: 5 and base_ejection_time: 30s is enough to get it working. None of these four have an equivalent. Traefik's circuit breaker plugin uses similar configuration, but it operates at the route level rather than the stream level, which makes it less effective for long-lived streams like gRPC where a single stream can hold a backend connection open for minutes.

Native JWT verification / OIDC

Traefik and Caddy have plugin support for this, but many of the available plugins are community-maintained. How they handle JWKS endpoint caching and key rotation varies, and some don't document their behavior clearly. If a signing key rotates and the plugin doesn't refresh its cache in time, valid tokens start getting rejected. Nginx requires Lua or the NJS module. HAProxy is in a similar position.

If you want to handle authentication at the proxy layer reliably, Kong or Envoy are the more practical choices. Kong's JWT plugin caches JWKS with configurable TTL and handles key rotation predictably — which matters when authentication is in the critical path.

gRPC load balancing

With HTTP/1.1, a new connection is opened per request (or closed quickly), so the proxy distributes at the request level. gRPC runs over HTTP/2, which multiplexes requests over a single long-lived connection.

Put Nginx in front of a gRPC backend as a load balancer, and Nginx maintains one persistent connection to each backend. Those connections stay alive for a long time. Stream distribution becomes uneven, and specific backends end up with concentrated load. You can have 10 backends but effectively route to only 1 or 2 of them.

Nginx can partially work around this with keepalive on the upstream block combined with HTTP/2 proxying, but stream-level routing — deciding per-request which backend to send to — remains limited. Traefik opens multiple HTTP/2 streams per backend connection and rebalances more aggressively than Nginx's static upstream model, so load spreads more evenly across backends. Envoy is built gRPC-first, with stream-level load balancing as a baseline assumption.

You've Configured a Reverse Proxy, But Can You Explain Why It Works? (Part 1)

Kazu — Thu, 20 Aug 2026 12:01:00 +0000

You wrote proxy_pass http://localhost:3000;, checked it in the browser, saw it working, and called it done. But you couldn't actually explain why it works. What's happening between nginx and the app? Where is the request going? You probably couldn't answer that with any confidence.

Getting something working and understanding how it works are two different things. This article (Part 1) is about the how. Once you have that, the next question becomes which tool to use. In Part 2, I'll break down the design philosophies behind nginx, Caddy, Traefik, and HAProxy, and give you a framework for choosing the one that fits your stack.

Where the browser is actually connecting

When a browser sends a GET to https://api.example.com/users, it resolves api.example.com via DNS and opens a TCP connection to port 443 at that IP. Who's on the other end of that connection?

In a setup with a reverse proxy, it's not the app server — it's the proxy.

Browser  ──[TCP connection]──→  nginx (443)  ──[separate TCP connection]──→  App (3000)
         ←───────────────────                ←──────────────────────────────

This is the key point. The browser and the app server are not talking directly. The proxy holds two separate connections and shuttles requests between them. That's what "split communication" means: the browser has no idea what the app server's IP address is, what port it's on, or that it even exists.

This split is also the root cause of several problems I'll get to later. The reason the browser's IP disappears from the app's perspective, and the reason the Host header gets rewritten — both trace back to this structure.

How it differs from a forward proxy

"Proxy" is an overloaded term. Forward proxy and reverse proxy are easy to mix up. Both relay traffic, but they stand in for opposite sides, and they hide opposite things.

A forward proxy acts on behalf of the client. You see it in corporate networks, where internal machines can't go directly to the internet — they route through a proxy server. The destination server sees the proxy, not the individual machines behind it. What's hidden is the client.

Internal PC  ──→  Forward proxy  ──→  External server
                                       ↑
                             Internal PC is not visible

A reverse proxy acts on behalf of the server. The client (browser) connects to the proxy without knowing how many app servers are behind it or how they're arranged. What's hidden is the server infrastructure.

Browser  ──→  Reverse proxy  ──→  App server
                                   ↑
                         Server is not visible

Forward proxies hide the client; reverse proxies hide the server. That distinction comes back when we look at what disappears when you add a proxy.

What happens before a request reaches the app

Between the browser sending a request and the app server receiving it, the proxy does several things. Let's trace through a concrete nginx config.

server {
    listen 443 ssl;
    server_name api.example.com;

    location /users {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

You've probably written something like this. Let's go through each line and why it's there.

proxy_set_header Host $host; — The browser's HTTP request includes a header like Host: api.example.com. When the proxy forwards that request to the app server, the Host header gets rewritten to localhost:3000 by default, because that's the host of the connection the proxy opened. If your app needs to know the original domain — for virtual host routing, or to construct a redirect URL — it needs api.example.com, not localhost:3000. This line passes the original value through explicitly.

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; — X-Forwarded-For (XFF) is a header that records the list of IP addresses a request has passed through. When the proxy forwards a request to the app, the connection comes from the proxy, not the browser. From the app's perspective, the remote IP is the proxy's address. To tell the app what the original client IP was, the proxy adds this header. $proxy_add_x_forwarded_for is an nginx variable that appends to an existing XFF header if one is present, or creates a new one with the connecting IP if not.

proxy_set_header X-Forwarded-Proto $scheme; — When the proxy receives HTTPS on port 443 and forwards it to the app as HTTP, the app has no way of knowing the original request came over HTTPS. X-Forwarded-Proto carries that information.

Each of those settings exists because of the connection split from the first section.

What problems a reverse proxy solves

Consider what happens if there's no proxy and the app server has to handle everything itself.

To serve over HTTPS, you need to manage a server certificate and handle TLS handshakes. A Node.js app doing that on its own looks like this:

const https = require('https');
const fs = require('fs');

const options = {
  key: fs.readFileSync('/etc/ssl/private/server.key'),
  cert: fs.readFileSync('/etc/ssl/certs/server.crt'),
};

https.createServer(options, app).listen(443);

The app needs to know the certificate file paths. Every time you renew the certificate, you restart the app. If you're using Let's Encrypt for automatic 90-day renewals, that means the app restarts on every renewal cycle. Cipher suite selection and protocol version management live in your app code too — every time TLS 1.0/1.1 gets deprecated somewhere, you touch the app.

When nginx handles TLS termination (ending the encrypted connection with the client at the proxy), your app code becomes:

// Only needs to speak HTTP
const http = require('http');
http.createServer(app).listen(3000);

Certificate management and TLS configuration stay in the nginx config file. The app server speaks plain HTTP and never needs to restart because a cert was renewed.

When traffic grows and a single app server can't keep up, you add more. But if browsers connect to app servers directly, there's no way to spread requests across the new instances. With a proxy, you get load balancing across multiple upstreams:

upstream api_servers {
    least_conn;  # send to the server with fewest active connections
    server 10.0.0.1:3000;
    server 10.0.0.2:3000;
    server 10.0.0.3:3000;
}

least_conn sends each request to whichever server currently has the fewest active connections. It distributes load more evenly than the default round-robin behavior when response times vary — slow requests don't pile up on one server.

Routing /api/ requests to the backend cluster and /static/ requests to a file server is also more naturally expressed at the proxy layer:

server {
    listen 443 ssl;
    server_name api.example.com;

    location /api/ {
        proxy_pass http://api_servers;
    }

    location /static/ {
        proxy_pass http://file_server;
    }
}

The routing logic stays out of your app code. Where each request goes is visible in the nginx config.

These are usually described as "reverse proxy features," but what they really are is work that doesn't belong in the app server. The proxy takes on what would otherwise be scattered across every instance, and the app gets to stay focused on what it's actually for.

What disappears when you add a proxy

After adding a reverse proxy, you look at your app's access logs and every IP is 127.0.0.1. This trips people up. I was one of them.

Before the proxy:

# Before proxy
203.0.113.45 - - [12/Aug/2026:10:00:01 +0000] "GET /users HTTP/1.1" 200 512

After adding the proxy, without XFF headers configured:

# After proxy (no XFF config)
127.0.0.1 - - [12/Aug/2026:10:00:01 +0000] "GET /users HTTP/1.1" 200 512

The reason is straightforward: the app's connection comes from the proxy, and if the proxy and app are on the same server, that's the loopback address. What the app sees is "the IP of whatever connected to me," not "the browser's IP."

The fix is to read the X-Forwarded-For header that nginx is attaching:

# Value available when the app reads XFF
X-Forwarded-For: 203.0.113.45

But XFF is a header the client can freely manipulate, so trusting the wrong source means your IP can be spoofed.

If you don't notice that the log format changed after setting up the proxy, debugging gets confusing fast. If your app does any IP-based access control and isn't reading XFF, every request looks like it's coming from the same address.

Wrapping up

Everything in this article comes back to one thing: the communication is split into two separate TCP connections. Because of that split, the Host header has to be passed through explicitly, the client IP disappears, and TLS termination at the proxy becomes a viable pattern. Every line in that config exists because of this split. Which tool to use for the proxy is Part 2.

If what's in this article has stuck, you should be able to answer something like: "The proxy holds two TCP connections and splits the communication. That's why the Host header gets rewritten and the client IP disappears. TLS ends at the proxy." You can now explain what's happening between nginx and your app.

curl works, your app gets 400: read the exact request your SDK sent

Kazu — Tue, 18 Aug 2026 13:01:00 +0000

curl returns 200. The same request from your app returns 400. You followed the docs. The parameters look right. It still doesn't work.

When that happens, what you really want to know is what your app actually sent. But that's the one thing you can't see. The SDK assembles the final request below your code, and TLS encrypts it before it hits the wire — capture the traffic and you get ciphertext.

This post is about seeing that plaintext with one command, no CA certificate install, no proxy, no changes to your app. The tool is tinytap, something I built for exactly this. Here's what it looks like.

curl works, the app gets 400

Here's a concrete example of how this goes wrong. You're hitting a payment API that expects an integer amount in cents. Your code passes 1000. The app returns 400 invalid integer: amount. You hardcode the same value in curl and it goes through fine.

The usual suspects come to mind. Double-encoded body? Type coercion somewhere? Wrong Content-Type? Missing token? All plausible, none confirmable. Your app's logs show the 1000 you passed into the SDK — not the final form that went over the wire.

If you could see the request body with "amount": "10.00" — the value that silently became a string somewhere — you'd have exactly what you need to compare what you meant to send against what you actually sent.

The SDK's final form never appears in any log

The obvious move is to check the logs. But whatever you print or pass to your logger is the value before the SDK gets it. The SDK adds headers, serializes the body, and possibly retries or signs the request — all below the layer your logs can see. You get the intent, not the result.

Wireshark is another option. But the connection is TLS, so you get ciphertext. Decoding that requires setting up SSLKEYLOGFILE and a decryption step, and not every client supports it.

mitmproxy gives you plaintext — but it requires installing a CA certificate in your trust store, configuring your app to route through the proxy, and some SDKs ignore proxy environment variables entirely. Installing a CA cert on your machine is a heavy setup just for debugging.

So there was a gap: seeing the actual plaintext, without touching the app, without installing a certificate, without a proxy in the middle.

Peek at the plaintext before it gets encrypted

That's what tinytap does. The gif above shows it: run your app on the left, and the request's plaintext appears on the right in tinytap.

The left side is running this:

from openai import OpenAI

client = OpenAI(api_key="sk-...", base_url="https://127.0.0.1:8443/v1")

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "what is a uprobe"}],
)

And on the right:

POST /v1/chat/completions HTTP/1.1
Host: 127.0.0.1:8443
Authorization: Bearer sk-demo-xxxx
Content-Type: application/json
User-Agent: OpenAI/Python 2.53.0
...
Request body (decoded): {"messages":[{"role":"user","content":"what is a uprobe"}],"model":"gpt-4o-mini"}

The request line, every header, the Authorization: Bearer token, the JSON body — exactly what the SDK built and sent. The User-Agent: OpenAI/Python line confirms this is real bytes from the real SDK, not something I reconstructed. No proxy, no certificate, no changes to the app. The token in the gif is a dummy (sk-demo-xxxx), but when you point tinytap at your own app, your real token appears in plaintext just like this.

Your code on the left, what it actually sent on the right. The moment you see both side by side, the "I thought I sent X" question is answered on the spot.

This is not breaking TLS

Worth stating upfront before anyone raises an eyebrow: this is not a vulnerability that lets you intercept traffic.

tinytap looks at the buffer your process hands to the TLS library, just before encryption — and the buffer that comes back out, just after decryption.

App
  │  plaintext
  ▼
TLS library  ◀── tinytap looks here
  │  ciphertext
  ▼
Network
  │  ciphertext
  ▼
API server

It hooks into your process from the inside, at the moment the plaintext is still plaintext. TLS itself is untouched. Other people's traffic is not accessible. This isn't about someone with root on your machine — it's about the inside of your own process. It's the same idea as curl printing its own request to your terminal. The ciphertext on the wire is untouched. All you're doing is looking at it from your own side, one step before encryption.

Try it yourself

"Plaintext without a certificate or proxy" sounds like it might be overselling it. So here's how to reproduce it yourself. tinytap uses eBPF, which means Linux only. On Mac or Windows, spin up a Linux container with Docker.

# install
curl -fsSL https://raw.githubusercontent.com/shinagawa-web/tinytap/main/scripts/install.sh | sh

# grant the capabilities needed for eBPF and TLS capture
sudo setcap cap_dac_read_search,cap_perfmon,cap_bpf,cap_sys_admin=eip $(command -v tinytap)

# set the execute bit on libssl (Debian/Ubuntu ships it without one)
sudo chmod +x $(ldconfig -p | grep libssl.so | awk '{print $NF}' | head -1)

tinytap

setcap grants the binary the capabilities it needs for eBPF and TLS capture. chmod +x sets the execute bit on libssl — Debian and Ubuntu ship libssl.so.3 without it by default, so this line is required there. With both steps done, tinytap runs without sudo. Run your app normally and the requests show up.

If something doesn't work, run tinytap doctor. It checks your kernel version, BTF availability, capabilities, and the libssl execute bit, and tells you exactly what's missing.

$ tinytap doctor
tinytap doctor — v0.6.1

[OK      ] kernel version               6.1.0 (>= 5.8 required)
[OK      ] kernel BTF                   present
[OK      ] cap_dac_read_search          present
[OK      ] cap_perfmon                  present
[OK      ] cap_bpf                      present
[OK      ] cap_sys_admin                present
[OK      ] cap_syslog                   not needed on arm64
[OK      ] perf_event_paranoid          4
[OK      ] unprivileged_bpf_disabled    2
[OK      ] RLIMIT_MEMLOCK               soft=unlimited hard=unlimited
[OK      ] syscall tracepoints          available
[OK      ] BPF dry-run load             ok
[DEGRADED] libssl execute bit           /usr/lib/x86_64-linux-gnu/libssl.so.3 not set
    Affects: TLS capture only. Plaintext HTTP capture is unaffected.
    Fix:     sudo chmod +x /usr/lib/x86_64-linux-gnu/libssl.so.3

12 ok, 1 degraded, 0 blocking, 0 info

tinytap only speaks HTTP/1.1 — gRPC and HTTP/2 binary frames aren't readable as-is. Go is also out of scope: its crypto/tls bypasses OpenSSL entirely. Python, Ruby, PHP, and Node.js all work out of the box. The language is a quick guide: Python / Ruby / PHP / Node.js, you're good; Go, you'll need a different approach.

Instead of the self-signed local server in the demo, you can point tinytap at whatever API-connected app you normally run. The Authorization header and body that show up are what you need to debug "curl works, app gets 400."

Which SDK has bitten you? Silent retries, double-encoded bodies, headers disappearing for no reason — if any of that sounds familiar, it's probably the kind of thing that resolves the moment you can see what was actually sent.

So how does it actually see the plaintext?

This post has stayed focused on what tinytap shows, not how it works. The mechanism — eBPF and uprobes, hooking into the TLS library boundary — is a story for another post.

The server returned 400. It's telling you the request was wrong. You don't know what's wrong because you can't see what your code actually sent. If you could see it, you'd just compare what you meant to send against what went out. tinytap is built for that.

"nginx doesn't run your config top to bottom: two traps with rewrite, try_files, and if"

Kazu — Tue, 11 Aug 2026 13:01:00 +0000

You write a rewrite, then a try_files right under it, and you read it as "the rewrite on top runs first, the try_files below runs after." You put an if inside a location and read it as "when the condition is true, this stuff gets added to the outer config." Both readings trace the config file straight down, line by line, in the order it's written. And both have burned me.

Let me show you the two symptoms first. In each case the config reads cleanly, yet the behavior doesn't line up. Why that happens — the order in which nginx processes a request — is what this post is about.

Symptom 1: rewrite and try_files side by side, and you get a 404

In some location you want to rewrite the request once, then fall back to a static file. The naive version looks like this:

location /app/ {
    rewrite ^/app/(.*)$ /public/$1 last;
    try_files $uri $uri/ =404;
}

The intent is "rewrite /app/foo to /public/foo, serve the file if it exists, 404 otherwise." Since rewrite is on top and try_files is below, the rewrite takes effect first and then try_files handles the fallback — that's how it reads.

But throw /app/logo.png at it and you get a 404, even when /public/logo.png really exists.

$ ls /public/logo.png
/public/logo.png            # the file is definitely there

$ curl -sI http://localhost/app/logo.png
HTTP/1.1 404 Not Found      # and yet, 404

The try_files in this location is never evaluated once.

Symptom 2: config placed inside an `if` block doesn't take effect

In another location you want to add one header conditionally. The naive version looks like this:

location /download/ {
    add_header X-Served-By "nginx" always;

    if ($arg_debug = "1") {
        add_header X-Debug "on" always;   # only want this when ?debug=1
    }

    root /var/www;
    try_files $uri =404;
}

The intent is "always add X-Served-By, and add X-Debug on top of it only when ?debug=1." But hit /download/foo?debug=1 and X-Debug shows up while X-Served-By, which you set on the outside, has vanished. The moment the condition goes true, part of the outer config drops.

Both configs read cleanly when you scan the lines top to bottom. And both miss.

How it works: nginx's HTTP processing phases

nginx runs each request through a fixed sequence of processing phases. Which phase a given directive runs in is determined per directive type, and has nothing to do with where the line sits in the config file. Both symptoms come out of exactly this one fact.

Start with the big picture. Lay out the phases an HTTP request passes through, in order, and you get this (source: the official Phases section of the Development Guide).

post-read       right after the request is read
    ↓
server-rewrite  rewrite directly under server (before location selection)
    ↓
find-config     pick the location
    ↓
rewrite         rewrite / if inside the location
    ↓
post-rewrite    if the URI changed, go back to find-config
    ↓
preaccess       connection count, rate limiting, etc.
    ↓
access          allow / deny / auth
    ↓
post-access
    ↓
precontent      try_files
    ↓
content         serving files via proxy_pass / root
    ↓
log             access log

The same rewrite runs before location selection (server-rewrite) when written directly under server, and after it (rewrite) when written inside a location. Where you put it changes the phase, which changes the order it runs in. What decides the order is not the line sequence but "which block, and which type of directive" — that's how you read this list. Now let's follow how each of the two symptoms travels along this timeline.

Symptom 1, followed by phase

The part that matters for rewrite ... last and try_files is this stretch.

  ┌─▶ find-config     pick the location (first pass: /app/. with last, the second pass picks a different one)
  │       ↓
  │   rewrite         rewrite ... last rewrites to /public/foo
  │       ↓
  └── post-rewrite    URI changed → back to find-config
          ↓
      precontent      try_files runs here. we're now in the different location picked on the second pass.
                      the try_files written in the original /app/ is never reached → 404

The last in rewrite ... last means "once you've rewritten the URI, redo location selection with the new URI." So the instant /app/foo becomes /public/foo, the post-rewrite phase calls find-config back and picks a location all over again. The rewritten /public/foo no longer matches /app/, so it moves to a different location. Meanwhile try_files runs way later, in the precontent phase. It may get evaluated wherever the request landed, but the try_files written in the original /app/ is never reached again. If the destination has no file and no fallback, that's where the 404 comes from. It's not that your =404 fired — the request dead-ended somewhere else, never reaching the original try_files. "Written below, so it runs after" is not what happens; the phases differ, so regardless of line position the rewrite fires first and jumps out of the location.

Symptom 2, followed by phase

if and add_header take effect in two phases that are far apart.

find-config     pick the location (/download/)
    ↓
rewrite         if is true → switch the config context to the inner block
    ↓           (the switch carries over into all later phases)
preaccess       connection count, rate limiting, etc.
    ↓
access          allow / deny / auth
    ↓
post-access
    ↓
precontent      try_files
    ↓
content         apply add_header when the response is being sent
                it looks at the block we switched into (X-Debug only)
                → the outer X-Served-By is gone

An if inside a location is an "implicit nested location" — it creates a config block of its own. And that if is a rewrite-phase directive. When the condition goes true, this phase switches the request's config context into the inner block, and that state carries over into the phases that follow. Much later, when add_header is applied at the point of sending response headers, what it reads is the block we switched into. All that's inside is X-Debug, and on top of that, add_header does not inherit from the outer scope once there's even one in the same block (it replaces rather than accumulates), so X-Served-By disappears entirely. If the condition is false, no switch happens and X-Served-By shows up as usual. The outer header vanishing is not about line order — it's because the switch that happened in the rewrite phase is still in effect in the later phases. What people call if is evil is, at bottom, this "context switch in the rewrite phase."

Fixes

Symptom 1

If you want to keep processing in the same location after the rewrite, change last to break. With break, the post-rewrite return to find-config doesn't happen, so the request can go down to the precontent try_files while staying in the same location.

location /app/ {
    rewrite ^/app/(.*)$ /public/$1 break;   # last → break
    try_files $uri $uri/ =404;
}

If instead you want to route the request to a different location depending on the URI, leave last as is. In that case, put the fallback in the destination location.

location /app/ {
    rewrite ^/app/(.*)$ /public/$1 last;
}

location /public/ {
    root /var/www;
    try_files $uri $uri/ =404;   # put the fallback in the destination
}

Which one is right comes down to a design call: do you want to keep processing in the same location after the rewrite, or hand off to a different one?

Symptom 2

Don't put content-handling directives (add_header, proxy_pass, try_files, and so on) inside an if. Treat return and rewrite as about the only things that are safe inside if.

The condition here is a query argument (?debug=1), and location can't match on the query string. For branching on a value like this, push it out of if and into a map that builds a variable.

map $arg_debug $x_debug {
    default "";
    "1"     "on";
}

server {
    location /download/ {
        add_header X-Served-By "nginx" always;
        add_header X-Debug     $x_debug always;
        root /var/www;
        try_files $uri =404;
    }
}

The two add_header lines sit at the same level, so no nested block replaces anything and both take effect. When $arg_debug isn't 1, $x_debug is empty, and nginx won't emit an add_header with an empty value, so X-Debug shows up only during debugging.

Checking which location you're in right now

The feel for phases sticks once you get your hands dirty. The quick way is to plant one marker in a suspect location to tell whether the request settled there.

location /app/ {
    add_header X-Loc "app" always;   # marker that shows up if you stay in this location
    rewrite ^/app/(.*)$ /public/$1 last;
    try_files $uri =404;
}

Look at the headers with curl -sI http://localhost/app/foo, and if X-Loc: app isn't there, the request has already left this location via rewrite ... last (the add_header you wrote here has no effect once it's gone). Change last to break, hit it again, and if X-Loc: app appears, the request stayed in the same location and made it down to precontent. One marker header lets you see, across phases, which location the request is in right now.

Wrapping up

When you read an nginx config, your eyes follow the lines top to bottom. But what nginx actually does is run the request along a timeline of phases. When you're questioning a config, don't trace the line order — trace it in this order instead: "which phase does this directive run in? and at that point, which location is the request in?" That usually makes things line up.

More places where this phase-based behavior produces silent bugs — add_header inheritance dropping when if switches context, alias path traversal, location selection overrides — are catalogued with reproduction cases in Silent nginx Config Bugs That Pass nginx -t.

nginx Tools for When Your Config Gets Out of Hand

Kazu — Tue, 04 Aug 2026 13:02:00 +0000

nginx config files grow. What starts simple gets layers added on top: another reverse proxy, TLS settings, routing logic that branches and branches again.

Before long you're looking at 20+ location blocks and can't say off the top of your head which one handles which request. You add a single add_header line and some other header silently disappears. You fix an alias path and prod throws 404s while staging is fine. nginx -t says "syntax is ok." But you don't feel ok about it.

Every config change comes with a quiet dread: will this break something? And that dread usually has a basis. nginx config mistakes are silent. Nothing shows up in the error log. The browser gets a normal-looking response. You don't find out something is broken until much later.

nginx itself is well-documented, but the ecosystem around it is harder to survey. Validation tools, config generators, test frameworks — searching turns up scattered information. This article organizes the common pain points into six categories and maps the tools that address each one.

Problem 1: Prevent config mistakes upfront
Problem 2: Generate config files
Problem 3: Automate testing
Problem 4: Collect metrics
Problem 5: Run nginx on Kubernetes
Problem 6: Visualize logs
Gaps that still aren't filled

Problem 1: Prevent config mistakes upfront

nginx -t checks syntax. But syntactically valid config can still have security problems. SSRF, path traversal via alias (e.g., a request to /files/../../etc/passwd reaches files it shouldn't), HTTP splitting — none of these trigger a syntax error.

Gixy: static analysis focused on security

https://github.com/dvershinin/gixy

Originally built at Yandex, now actively maintained as a community fork. It checks for well-known security risks: SSRF, alias traversal, HTTP splitting. Install the gixy-ng fork from PyPI.

pip install gixy-ng
gixy /etc/nginx/nginx.conf

Output looks like this — it tells you which directive is the problem and why:

[WARN] [http_splitting] Possible HTTP-Splitting via HTTP header.
Directive: proxy_set_header X-Forwarded-For $http_x_forwarded_for;

A good fit when you've just inherited an nginx config and want a quick security audit before touching it. Detection is limited to known patterns and there's no custom rule support, but as a first security gate in CI it offers the best return on effort.

crossplane: treat config as data

https://github.com/nginxinc/crossplane

A Python library that parses nginx config files into JSON. Useful not just for validation but for dynamically generating config or loading it in tests.

import crossplane

payload = crossplane.parse('/etc/nginx/nginx.conf')

# Check for parse errors
for error in payload['errors']:
    print(f"Error: {error}")

# Walk the config tree to find specific directives
def find_directives(block, name):
    for item in block:
        if item.get('directive') == name:
            yield item
        if 'block' in item:
            yield from find_directives(item['block'], name)

config = payload['config'][0]['parsed']

# Example: flag any proxy_pass that uses plain HTTP
for d in find_directives(config, 'proxy_pass'):
    url = d['args'][0]
    if url.startswith('http://'):
        print(f"Warning: proxy_pass uses plain HTTP: {url}")

The parsed output is a tree of directives and their arguments. With a traversal like find_directives, you can express project-specific rules in Python: "are all proxy_pass values HTTPS?", "is server_tokens off set?" Where Gixy is limited to known patterns, crossplane lets you write your own.

Scripting against config files and automating structural checks in CI are where this earns its place. If you don't have a concrete need to generate or programmatically inspect config, skip it for now. Adding it "because it looks useful" tends to mean it sits unused.

Which to choose

Goal	Pick
Find security issues quickly	Gixy
Manipulate or validate config in code	crossplane

Gixy alone covers the security CI gate. If you're writing config generation or structural validation logic in Python, you need crossplane. They don't overlap, so using both is fine.

Problem 2: Generate config files

Writing TLS config from scratch means making a lot of small decisions: cipher suites, HTTP/2 support, HSTS headers. Starting from a known-good template beats hand-rolling it and getting something subtly wrong.

nginxconfig.io: generate best-practice config in the browser

https://www.digitalocean.com/community/tools/nginx

A web app by DigitalOcean that generates a complete nginx config from a series of UI choices. Select your TLS cipher suites, gzip settings, HTTP/2, security headers, and it outputs ready-to-use config files. Free, no account required.

Use it when bootstrapping a new server or bringing TLS settings up to current best practices. It's an excellent starting point, but don't deploy the output unchanged. Treat it as a template and adjust for your project's requirements.

nginx-proxy: auto-generate config in Docker environments

https://github.com/nginx-proxy/nginx-proxy

A container that watches Docker start/stop events and automatically updates nginx config. Set a VIRTUAL_HOST environment variable on your container and nginx-proxy generates the reverse proxy config for it.

services:
  app:
    image: my-app
    environment:
      - VIRTUAL_HOST=example.com
  nginx-proxy:
    image: nginxproxy/nginx-proxy
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro

When app starts, nginx-proxy automatically creates config to route example.com traffic to it.

In Docker Compose setups where manually updating nginx on every deploy is friction, this removes that burden. Traefik and Caddy do the same thing. If you're not already committed to nginx (existing config, team familiarity), compare them first. If you're already running nginx, nginx-proxy is the natural fit.

Which to choose

Environment	Pick
Regular Linux server	nginxconfig.io
Docker / Docker Compose	nginx-proxy

No Docker? nginxconfig.io is enough. In Docker environments, nginx-proxy's automation pays off.

Problem 3: Automate testing

Manually sending test requests after every config change doesn't scale. Automating it means the same checks run every time.

Test::Nginx: declarative test framework

https://github.com/openresty/test-nginx

A Perl-based test framework from the OpenResty community. It starts a real nginx instance, sends requests, and checks responses. The format is declarative — you describe what config to use, what request to send, and what response to expect. Tests read clearly as a result.

=== TEST 1: basic proxy
--- config
location /api {
    proxy_pass http://127.0.0.1:8080;
}
--- request
GET /api/users
--- response_body
[{"id":1}]

This fits nginx module development and verifying that config changes don't alter behavior. Perl is a real barrier for most teams today. If you're not writing custom nginx modules, hurl covers most of what you actually need.

hurl: a DSL for HTTP testing

https://hurl.dev/

hurl lets you write HTTP requests and assertions in a plain-text format, then run them from the shell. Easy to slot into CI. Not nginx-specific — it works for HTTP testing in general.

GET http://localhost/api/users
HTTP 200
[Asserts]
header "Content-Type" contains "application/json"
jsonpath "$.length" > 0

hurl --test api.hurl

Reach for it when you want to confirm API behavior hasn't broken after a config change, or when the team has no Perl experience. Because it's not nginx-specific, it doesn't go deep on nginx internals. It works well for endpoint reachability checks, but for testing precise nginx behavior (match precedence, path rewriting), Test::Nginx is the right tool.

Which to choose

Situation	Pick
Testing nginx config or modules directly	Test::Nginx
Checking that the API behind nginx still works	hurl
Team has no Perl experience	hurl

Test::Nginx for precise nginx behavior verification. hurl for HTTP interface testing with a lower adoption cost.

Problem 4: Collect metrics

Without visibility into how many requests nginx is handling and where the bottlenecks are, there's no basis for making informed decisions when problems surface.

nginx-module-vts: embedded stats module

https://github.com/vozlt/nginx-module-vts

A module that adds traffic statistics to nginx — request counts, error rates, latency per virtual host and upstream. Built in at compile time. Exposes a /status endpoint that returns JSON.

vhost_traffic_status_zone;

server {
    location /status {
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
    }
}

Detailed traffic stats are the payoff — when you have a build environment and can compile nginx from source. Requiring a source build is the biggest barrier. If you're using a packaged nginx install, this isn't an option. If you already have Prometheus, the overhead of a custom build may not be worth it.

nginx-prometheus-exporter: bridge stub_status to Prometheus

https://github.com/nginxinc/nginx-prometheus-exporter

Reads nginx's built-in stub_status data (connection counts, request totals) and exposes it in Prometheus format. Runs as a separate process alongside nginx — no changes to nginx itself required.

location /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
}

./nginx-prometheus-exporter -nginx.scrape-uri=http://localhost/nginx_status

In Prometheus + Grafana stacks, this is the easiest way to add nginx metrics without touching nginx itself. stub_status only gives you connection counts and request totals — no per-upstream breakdown. Confirm that's enough before committing.

Which to choose

Situation	Pick
Already running Prometheus	nginx-prometheus-exporter
Need per-host, per-upstream, per-status-code stats	nginx-module-vts
Using packaged nginx and don't want to rebuild	nginx-prometheus-exporter

If you're on Prometheus, nginx-prometheus-exporter is by far the easier path. If you need granular metrics and control your build, look at vts.

Problem 5: Run nginx on Kubernetes

When exposing services externally in Kubernetes, you use an Ingress resource — a rule that maps incoming paths to backend services. nginx-based Ingress controllers are widely used for this. Two controllers with similar names exist, and they're frequently confused.

ingress-nginx: the Kubernetes community controller

https://github.com/kubernetes/ingress-nginx

Maintained by the Kubernetes community (kubernetes/ingress-nginx). Used as the default by many Kubernetes distributions. Large amount of documentation and community knowledge available.

In March 2025, a set of serious vulnerabilities called IngressNightmare was disclosed (CVE-2025-1974 and others, CVSS 9.8). Remote code execution was possible through the admission controller component. Fixed in versions 1.12.1 and 1.11.5. If you're running an older version, upgrade now.

On a general-purpose Kubernetes cluster running standard Ingress, this is the obvious starting point. Switching Ingress controllers later is painful, so the initial choice matters. When in doubt, ingress-nginx has fewer dead ends from a documentation standpoint.

kubernetes-ingress: the NGINX Inc. controller

https://github.com/nginxinc/kubernetes-ingress

Maintained directly by NGINX Inc. (nginxinc/kubernetes-ingress). Supports both open-source nginx and NGINX Plus (the commercial version).

The case for it is NGINX Plus features (advanced load balancing, active health checks) or a support contract with NGINX Inc. If you don't need NGINX Plus, there's little reason to choose this over ingress-nginx. For OSS nginx, the community controller has better coverage.

Which to choose

Situation	Pick
Community resources matter	ingress-nginx
Planning to use NGINX Plus	kubernetes-ingress
Starting on EKS / GKE or similar managed cluster	ingress-nginx

If you go with ingress-nginx, run version 1.12.1 or 1.11.5 or later. Check your current version before anything else.

Problem 6: Visualize logs

nginx access logs pile up as text. Figuring out which paths are getting hammered, or where errors are spiking, is hard to do from raw log files.

GoAccess: real-time stats in the terminal

https://goaccess.io/

A TUI tool that aggregates nginx access logs and displays them in real time. Graphs and stats update live in the terminal. No log server or database needed — just install and run. Can also generate HTML reports for browser viewing.

goaccess /var/log/nginx/access.log -c
# Starts in interactive mode to select the log format

# Stream live logs in COMBINED format
tail -f /var/log/nginx/access.log | goaccess --log-format=COMBINED -

Reach for it when you need answers quickly, especially on a server with no log collection infrastructure in place. It reads local log files directly, so multi-server environments are awkward. Think of it as a per-server quick-look tool.

Loki + Grafana Alloy: long-term log storage and search

https://grafana.com/oss/loki/ / https://grafana.com/docs/alloy/latest/

Loki stores and queries logs efficiently, with Grafana for dashboards. Grafana Alloy handles log collection. Alloy replaces Promtail, which reached end-of-life in March 2026 — for new setups, use Alloy.

Example Alloy config (sending nginx logs to Loki):

loki.source.file "nginx" {
  targets = [{
    __path__ = "/var/log/nginx/access.log",
    job      = "nginx",
  }]
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

It earns its place when you need to aggregate logs from multiple servers, search historical logs by time range, or view nginx logs alongside other metrics in Grafana. Running the full stack yourself carries real operational overhead. Start with Grafana Cloud's free tier, or bring this in when someone on the team already knows how to operate Grafana.

Which to choose

Situation	Pick
Need to check something right now	GoAccess
Already running Grafana	Loki + Grafana Alloy
Need long-term log retention and cross-server search	Loki + Grafana Alloy
Debugging with direct SSH access to the server	GoAccess

GoAccess gives you immediate answers. Loki + Alloy is for ongoing operations. They're not mutually exclusive — GoAccess for fast situational awareness, Loki for accumulating and analyzing over time.

Summary: match tool to problem

Problem	Tool
Find security risks in config	Gixy
Manipulate config programmatically	crossplane
Generate correct config (Linux server)	nginxconfig.io
Auto-manage Docker reverse proxies	nginx-proxy
Test nginx config behavior	Test::Nginx
Add HTTP connectivity checks to CI	hurl
Collect metrics via Prometheus	nginx-prometheus-exporter
Need detailed per-upstream stats (module)	nginx-module-vts
Set up Ingress on Kubernetes	ingress-nginx (1.12.1+)
Need NGINX Plus features	kubernetes-ingress
Visualize logs immediately	GoAccess
Long-term log storage and search	Loki + Grafana Alloy

nginx itself is stable and mature, but the ecosystem moves fast — Promtail's EOL is a good example. Periodically check the maintenance status of tools you rely on, or you'll find out about a support cutoff at the worst time.

Gaps that still aren't filled

The tools covered here leave some problems unsolved or underserved. If you know tools that address any of these, drop them in the comments.

Dynamic upstream management

Updating upstreams without an nginx reload is a common requirement. With open-source nginx, the only options are OpenResty (Lua scripting for dynamic upstream control) or NGINX Plus (API-driven management). Vanilla nginx requires rewriting config and reloading.

Config drift tracking across environments

There's no standard tool for tracking and visualizing how nginx config differs between dev, staging, and production. Git-based management is the practical answer, but cross-environment diff review and detecting prod-only config still means manual work.

IDE / LSP support

VSCode extensions for nginx.conf exist, but completion and inline validation are incomplete. No editor today reliably explains location block match priority or how proxy_pass transforms paths.

WAF

ModSecurity has a solid track record as an nginx WAF, but rule management and false-positive tuning are heavy ongoing work. There's nothing you can drop in and operate at a low maintenance level.

https://github.com/owasp-modsecurity/ModSecurity-nginx

Distributed tracing

The nginx OpenTelemetry module (ngx_otel_module) shipped in 2023 but is still maturing. Trace collection with Jaeger or Tempo is possible, but documentation and real-world operational experience are thin.

https://github.com/nginxinc/nginx-otel

$request_time Looks Fast, But Users Say It''s Slow: Breaking nginx Latency into 4 Parts

Kazu — Tue, 28 Jul 2026 13:03:00 +0000

Your nginx $request_time looks fine in the dashboard — p95 is 120ms, totally reasonable.

But users keep reporting that things "occasionally freeze." You dig through the logs looking for slow requests and find nothing obvious. Overall p95 is healthy. The upstream app team says "our side is fast." nginx looks fast too. Yet the complaints are real. You don't know where to look.

$request_time collapses everything that happened during the request into a single number. Fast phases and slow phases all get averaged together, making it impossible to see where the time actually went.

nginx does record timestamps at each intermediate checkpoint, though. Subtract them, and latency breaks into 4 distinct parts.

The log variables covered here and what they measure are the same regardless of nginx version.

What one number hides

$request_time measures everything from when nginx reads the first byte from the client to when it finishes sending the last byte of the response. In a reverse proxy setup, that single number includes at least all of:

time to establish a connection to the upstream backend
time until the backend starts returning a response (backend processing time)
time to receive the full response from the backend
nginx's own processing and the time to send the response back to the client

All of this gets summed into one $request_time. So even if p95 is 120ms, you cannot tell from that number alone whether the breakdown is "20ms backend, 100ms sending" or "100ms connecting, 20ms everything else." Those scenarios call for completely different fixes, but $request_time erases the distinction.

Also worth noting: the TLS handshake on the client side happens before $request_time starts, since the clock starts at "first byte received." If you suspect TLS overhead and stare at $request_time, it won't show up there.

Before you start: add variables to your log format

The breakdown analysis below requires $upstream_connect_time, $upstream_header_time, and $upstream_response_time to be present in your access log. Open your own logs and there's a good chance none of these are there.

The reason is simple: nginx's default combined log format doesn't include any of them. combined logs eight fields — remote address, user, timestamp, request line, status, response bytes, referer, and user agent. $request_time isn't even there. Upstream timing variables are excluded by design.

Until you define a custom log_format and reload nginx, this breakdown is physically impossible. You can't subtract what doesn't exist. Fix the log definition first.

log_format latency '$remote_addr "$request" $status '
                   'rt=$request_time '
                   'uct=$upstream_connect_time '
                   'uht=$upstream_header_time '
                   'urt=$upstream_response_time';

access_log /var/log/nginx/access.log latency;

After adding this and running nginx -s reload, the raw data you need starts appearing in logs. Logs written before the reload won't have these fields, so there's no way to retroactively see the breakdown for historical requests.

There's also an approach that captures timing directly from the kernel without touching the nginx config. ngxray explores that direction (currently in development).

Breaking it into 4 parts

When nginx operates as a reverse proxy, it records three timestamps for the upstream interaction. These are the basis for the breakdown.

$upstream_connect_time: time until the connection to the upstream is established (includes TLS handshake if the upstream is HTTPS)
$upstream_header_time: time until nginx starts receiving the response headers from the upstream
$upstream_response_time: time until nginx has received the full response from the upstream

All three are measured from the same starting point — the moment nginx begins processing the upstream. They're not three independent intervals; they're cumulative values. $upstream_response_time contains $upstream_header_time, and $upstream_header_time contains $upstream_connect_time. They're nested.

Once you understand they're cumulative, subtraction gives you the individual intervals.

Part	Formula	What it measures
① Connect	`$upstream_connect_time`	Time to connect to upstream (TLS included)
② Backend processing	`$upstream_header_time − $upstream_connect_time`	Time until upstream starts responding (TTFB)
③ Response transfer	`$upstream_response_time − $upstream_header_time`	Time to receive the full response from upstream
④ Client + nginx internal	`$request_time − $upstream_response_time`	Request body read, nginx internals, sending to client

Don't read ④ as "nginx is slow." It captures request body reading, rewrite/access phases, and the time to finish delivering the response to the client. When users on slow connections are the only ones complaining, ①②③ will look fine while ④ carries all the time. Slow clients only become visible through this breakdown. This assumes proxy_buffering is on (the default), where nginx buffers the full upstream response before sending to the client. If buffering is off, slow-client impact can bleed back into ③, so suspect both in that case.

And don't look at averages. Break each of the four parts into p50 / p95 / p99 separately. Whichever part has a spiking p99 is your answer.

For example, laying out p50 / p95 / p99 per part might reveal a distribution like this (these numbers are for illustration):

Part             p50     p95     p99
① connect         1ms     2ms     3ms
② backend        15ms    40ms    60ms
③ transfer        3ms     8ms    12ms
④ client+nginx    2ms     5ms   380ms   ← here

Even when $request_time p95 looks clean at 120ms, the p99 for ④ alone can spike to 380ms. That's what "occasionally freezes" actually is — buried in the overall average and p95, but visible as soon as you look at p99 per part.

Once you know which part is the problem, the next action follows directly: ① is thick → upstream keepalive probably isn't configured (covered in the next section); ② is thick → backend processing is the bottleneck; ③ is thick → look at transfer size or upstream bandwidth; ④ is thick → suspect client connection speed or response body size.

Even the nginx official blog's explanation writes that $upstream_header_time measures the time "between establishing a connection and …" — it's ambiguous whether that excludes $upstream_connect_time or includes it. Rather than memorizing each variable's exact definition, it's more reliable to internalize the structure: all three accumulate from the same origin, and you subtract to get intervals.

Computing p50/p95/p99 per part

"Look at p99 per part" is easy to say, but how do you actually get those numbers? Using the latency format defined above, here's an awk example to compute percentiles.

To extract percentiles for part ④, client-side (rt − urt):

awk '{
  for (i = 1; i <= NF; i++) {
    if ($i ~ /^rt=/)  { rt  = substr($i, 4) }
    if ($i ~ /^urt=/) { urt = substr($i, 5) }
  }
  if (rt != "" && urt != "" && urt + 0 >= 0) a[++n] = rt - urt
}
END {
  asort(a)
  printf "p50=%.3f  p95=%.3f  p99=%.3f\n",
    a[int(n * 0.50)], a[int(n * 0.95)], a[int(n * 0.99)]
}' /var/log/nginx/access.log

The structure is the same for every other part. For ② backend processing use uht - uct; for ③ response transfer use urt - uht.

In production you can run equivalent aggregations in Loki, CloudWatch Logs Insights, or Datadog log queries. The key is not "overall p95" but "p99 per part, listed separately." No matter how good a single number looks, the problem stays hidden until you split the breakdown and look at each part's p99.

When ① is always non-zero

When you break down the parts, you occasionally see a pattern where ②③④ are all small but $upstream_connect_time registers as non-zero on every single request. The responses themselves are lightweight, but there's a small constant overhead on every request.

The TCP connection to the upstream isn't being reused. When connections are kept alive and reused, $upstream_connect_time should be near 0 (or -) after the first request. If it's consistently non-zero on every request, nginx is creating a new connection for each one — keepalive to the upstream isn't configured. This tends to get overlooked in long-running systems.

Two directives work together here:

upstream backend {
    server 127.0.0.1:3000;
    keepalive 32;
}

server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

keepalive 32 sets the maximum number of persistent connections the worker keeps in the pool. proxy_http_version 1.1 and proxy_set_header Connection "" are required because HTTP/1.0 sends Connection: close by default, which causes the upstream to close the connection immediately. Without those two lines, keepalive does nothing.

After adding the config and running nginx -s reload, check the uct= values in the access log. The first request will be non-zero since it's establishing the connection. If subsequent requests show 0.000 or -, connection reuse is working.

Repeatedly opening new connections also leads to TIME_WAIT buildup and ephemeral port exhaustion, but that's a different layer of concern. For this article, the key read is: "① consistently non-zero = connections aren't being reused."

When the breakdown breaks: commas and colons

There's another pattern that breaks the subtraction: looking at a $upstream_* value and finding more than one number.

uct=0.001, 0.003 uht=0.012, 0.045 urt=0.012, 1.230

This isn't a broken log. nginx logs multiple values when a single request involves multiple upstream interactions. The delimiter carries meaning.

Comma-separated (0.001, 0.003): the request hit multiple upstream servers in sequence. The typical case is the first server failed and the second was tried as a retry/failover.
Colon-separated (0.001 : 0.003): an internal redirect via X-Accel-Redirect or similar occurred, spanning multiple upstream groups.

If you treat this as a single value and compute $request_time − $upstream_response_time, the subtraction breaks completely. The string "0.012, 1.230" isn't a number.

More importantly, multiple values are a signal of something abnormal. Two comma-separated values mean there was a retry — the first upstream attempt failed. That failure was buried in the average and invisible. When a slow request shows comma-separated upstream times, investigate the retry before worrying about the interval breakdown.

Closing: where is your time going?

Rather than wrapping this up as a checklist of solutions, here are questions to bring to your own logs.

Does your access log include $upstream_connect_time, $upstream_header_time, and $upstream_response_time? Are you still running the combined format?
Are you looking at p50 / p95 / p99 separately for each of the 4 parts, not just $request_time as a whole?
Have you ever computed $request_time − $upstream_response_time (part ④, client + nginx internal)?
Is $upstream_connect_time non-zero on every request?
Can you interpret what it means when $upstream_* has multiple values separated by commas or colons?
Can you explain "numbers look fast but users say slow" using per-part p99 rather than averages?

If any answer is no, you haven't broken down latency yet. As long as that's the case, "fast numbers, slow experience" stays a permanent contradiction.

$request_time doesn't lie. It just compresses too much into one number. Split it four ways and you can always pinpoint where the time went. Start by opening your access log and checking whether $upstream_response_time is even being recorded.

For broader observability — metrics exporters, log visualization tools, and the validation and testing tools that sit around nginx's config — there's a roundup of the nginx ecosystem that maps what fits where.

References (official documentation)

Every variable and behavior discussed here is documented in the official nginx docs.

ngx_http_upstream_module: definitions of $upstream_connect_time, $upstream_header_time, $upstream_response_time; meaning of comma/colon separators; TLS handshake handling
ngx_http_log_module: log_format and the default combined format definition
Using NGINX Logging for Application Performance Monitoring: upstream timing variable usage
nginx source ngx_http_upstream.c: confirmation that all three variables share the same origin (u->start_time) and the nested structure

Silent nginx Config Bugs That Pass nginx -t: if, location, add_header, alias

Kazu — Tue, 21 Jul 2026 13:02:00 +0000

nginx -t returns syntax is ok and test is successful. nginx -s reload finishes without a complaint. Nothing shows up in the error log.

And yet the behavior is wrong.

I've done this to myself more times than I'd like to admit. The one that hurt most was a CSP header I shipped to production that never reached a single browser for a full day — I only noticed because I got suspicious that no violation reports were coming in. There have been others: a file I put under /static/ getting swallowed by a different location, old behavior lingering after a reload. In every case nginx -t passed, and nothing was grammatically wrong anywhere.

The nature of this gap is clear: nginx -t validates only syntactic correctness, and never looks at semantic correctness.

What `nginx -t` actually guarantees for you

What nginx -t does, roughly speaking, is check "is this config file in a shape that nginx can load?" Does the directive name exist? Is the argument count right? Are the {} blocks closed? Are the value types valid? In compiler terms, it goes about as far as syntax errors and maybe type checking.

Everything past that point — "will this config process requests the way you intended?" — is outside the scope of nginx -t. This is where the blind spot is. When the test passes, you quietly reread it as "the config is correct." But all that passed was the syntax test.

Outside nginx -t, there are three layers of breakage.

Semantic level — the config isn't read the way you intended. The grammar is perfect.
Security level — it works exactly as you intended. It just also works exactly as the attacker intended.
Runtime level — the config text is irrelevant. It's the behavior of the reload operation itself.

The higher layers are the ones you step on daily, and their cause is easy to look for inside the config file. The lower you go, the more it becomes the kind of incident where reading the config file gives you no answer at all. We'll go top to bottom. Reading this with your own nginx.conf open beside you should help.

Note that this article assumes the behavior of the nginx 1.31 (mainline) series. The stable series is 1.30, and the behavior covered here — if, location selection, add_header inheritance, alias, and reload — is the same across both. Where a version makes a difference, I'll call it out explicitly.

Semantic level: the grammar is correct, but it's read differently

The contents of `if` aren't evaluated in the order you think

When you want to "branch on a condition" in an nginx config, if is the first thing you reach for. But an if inside a location has long been called "if is evil" in the nginx community. It sounds like a religious argument if you only hear the name, but it's a statement about behavior.

First, why it breaks. if is a directive of the rewrite module, evaluated in an early phase of request processing (the rewrite phase). And the nastier part: when you write an if inside a location, that if block is treated as an implicit nested location. When the condition is true, nginx switches the request's entire configuration context into that inner location. Not every directive gets dropped (many settings like root or proxy_set_header carry over). But content-processing directives like try_files or proxy_pass may not carry over the way you expect. That's the trap.

Let's look at a concrete example. A common, simple setup: serve a static site, but route just the requests under /api/ to a backend. It's completely valid grammatically and passes nginx -t:

location / {
    root /var/www/html;
    try_files $uri $uri/ =404;      # serve static files

    if ($uri ~ ^/api/) {
        proxy_pass http://backend;  # route only /api/ to the backend...
    }
}

The author's intent is "normally serve static files, and only proxy to the backend when the path starts with /api/." But if creates an implicit nested location, so when the condition is true the config context switches wholesale into it. Inside there's only proxy_pass; the outer content processing like root and try_files isn't carried into the switched-to location. Worse, once there's an if in the location, try_files can stop being evaluated as expected even for requests where the condition is false. The grammar is correct and nginx -t says nothing, so you get "occasional 404s" and "occasional un-proxied requests" without ever noticing.

Detection and avoidance: The things that are genuinely safe inside an if in a location are, in practice, only return ..., rewrite ... last, set ... (and break). For anything else — try_files, proxy_pass, header manipulation — the moment you want to do it inside an if, that's a sign the design is wrong. When you want to "route by path" like here, the answer is not if but splitting the location. Routing in nginx is location's job; if breaks it by cutting in.

Rewriting the earlier example by splitting the location looks like this:

location / {
    root /var/www/html;
    try_files $uri $uri/ =404;      # static serving is this block's only job
}

location /api/ {
    proxy_pass http://backend;      # /api/ is a separate location from the start
}

Making /api/ its own location means no implicit nesting and no config-context switch. The static-serving location / and the proxying location /api/ each hold only their own processing. Replacing conditional branching with "location as routing" rather than "if as a control statement" is the basic form of avoiding if. When you want to branch on a value rather than a path, push the condition out into a map (a lookup table that maps an input value to an output value) with the same mindset, and use the resulting variable.

Location priority is not the order you wrote them in

When you write multiple location blocks, don't you assume "the one written higher up takes priority"? I did. And it bit me.

nginx's location selection runs on a rule that differs from the order you wrote. The procedure is this:

If an = (exact match) location matches the URI exactly, it's decided right there and nothing else is examined.
Otherwise, among the prefix locations (no modifier, or ^~), pick and remember the one with the longest match. Order within the config file is irrelevant here. The longest match wins.
If the selected prefix has ^~, it's decided there and regex is not checked.
Otherwise, check the regex locations (~, ~*) in written order, and the first that matches wins. Here, written order matters.
If no regex matches, use the prefix remembered in step 2.

"Prefixes are longest-match (order-independent), regexes are first-match by written order, and regexes take priority over prefixes." That last point is what gets people. A reproduction:

location /static/ {
    root /var/www;
}

location ~ \.(png|jpe?g|gif)$ {
    expires 30d;
    root /var/www/cache;   # <- a different root
}

Let's trace a request to /static/logo.png. As a prefix, /static/ is the longest match and gets "remembered." But /static/ has no ^~. So nginx proceeds to the regex check, \.(png|jpe?g|gif)$ matches, and that one wins. The result: /static/logo.png goes looking in /var/www/cache. The /static/ block is skipped past. The grammar is perfect, nginx -t passes. Only your assumption — "I'm handling all static files under /static/" — is off.

Detection: Don't guess which location you're hitting — confirm it. The easy way is to temporarily add a marker header to the suspect locations:

location /static/ { add_header X-Loc "static" always; ... }
location ~ \.(png|jpe?g|gif)$ { add_header X-Loc "regex" always; ... }

Then hit curl -sI http://localhost/static/logo.png | grep X-Loc and you'll know instantly which one answered. Nailing down "where am I hitting right now" as a fact comes first.

Once you have that, if you want to lock it down, put ^~ on the prefix to stop the regex check itself:

location ^~ /static/ {   # decided the moment this prefix is the longest match; regex is not examined
    root /var/www;
}

With `add_header`, adding one in the child makes all the parents disappear

This is the trap that most symbolizes the "semantic level" of the three layers, and the one I was stuck on the longest.

As a premise, many people write security headers like HSTS or CSP once in the server block, intending them to apply to every location.

server {
    add_header Strict-Transport-Security "max-age=63072000" always;
    add_header Content-Security-Policy "default-src 'self'" always;

    location /api/ {
        add_header Cache-Control "no-store" always;   # <- the moment you add this
        proxy_pass http://backend;
    }
}

You just added one cache-control header to /api/. But nginx's add_header is specified so that if there's even one add_header at the current level, it does not inherit any add_header from the parent level at all. It's replacement, not merging. The result: HSTS and CSP disappear from the /api/ response, and all that's left is Cache-Control. Only the security headers go silently missing.

This "array-style directives don't inherit (don't merge with) the parent once defined in a child" behavior is not unique to add_header. proxy_set_header and fastcgi_param have the same trap. Write this, for example, and Host and X-Forwarded-For stop being passed to the backend:

server {
    proxy_set_header Host            $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    location /api/ {
        proxy_set_header X-Request-Id $request_id;   # <- just added one
        proxy_pass http://backend;
        # neither Host nor X-Forwarded-For is inherited here
    }
}

Detection: Missing headers — the only sure method is to actually look at the response. At each endpoint where you "think" you set a header, compare against the real thing.

curl -sI https://example.com/         | grep -i -E 'strict-transport|content-security'
curl -sI https://example.com/api/foo  | grep -i -E 'strict-transport|content-security'

Present at the top but gone under /api/ is the classic signature of this trap.

How to deal with it: The most portable fix is to re-list every header you need in the child block. Split the common part into a separate file and include it, so you gather the duplication into one place while re-declaring it at each level:

# security-headers.conf
add_header Strict-Transport-Security "max-age=63072000" always;
add_header Content-Security-Policy   "default-src 'self'" always;

server {
    include security-headers.conf;

    location /api/ {
        include security-headers.conf;      # re-read and re-declare in the child too
        add_header Cache-Control "no-store" always;
        proxy_pass http://backend;
    }
}

The add_header_inherit merge; added in nginx 1.29.3 lets you change the behavior to "inherit the parent, then add the child" (it's in both the current stable 1.30 series and mainline 1.31 series). But it's not in the 1.28-and-earlier stable series, so it may not be available on your version — getting the premise "it's replacement" into your bones is more effective first.

Forget one trailing slash on `alias`, and you can climb outside the directory

Everything so far has been about "behavior that differs from intent." This alias trap is the same kind of mistake — a single-slash difference — but the quality of the outcome is different. The config looks like it works as intended and does, yet you can read files outside the published directory. It's not just a behavior bug; it's a security hole as-is.

A reproduction. It's the commonplace setup of "I want to serve static files from a different directory":

location /assets {           # <- no trailing slash
    alias /var/www/static/;  # <- this one has a trailing slash
}

/assets/logo.png is served normally. nginx -t passes, and as far as the browser is concerned there's no problem. It looks like it works as intended.

Why it's dangerous. alias builds the real file path by replacing the portion of the URI that matched the location (here, /assets) with the value of alias. When the location is /assets (no slash), the string following /assets gets stuck on directly at the end. So a request for /assets../ resolves to /var/www/static/ + ../ = /var/www/static/../ = /var/www/. That means with a little trick like /assets../../etc/passwd, you can climb outside the intended published directory. It's a classic path traversal (an attack that walks back up directories to read non-public files on the server). Normal access never notices it at all.

Detection and fix: The way to spot it is simple — line up the trailing slashes on the location and the alias. Either both have one, or neither does. In the example above, add a trailing slash to the location side too, like this:

location /assets/ {          # <- add the trailing slash so they match
    alias /var/www/static/;  # <- alias side has a trailing slash too
}

Now /assets../ no longer matches this location in the first place, and the climb is gone. Normal requests like /assets/logo.png are still served exactly as before.

If the location and the directory name match in the first place, the safest move is to drop alias and lean on root instead. root doesn't replace the location portion — it appends the URI as-is under the directory — so this trailing-slash problem simply can't occur:

location /assets/ {
    root /var/www/static;    # /assets/logo.png -> /var/www/static/assets/logo.png
}

(Note that root appends the URI path verbatim, so the real directory has to be laid out as /var/www/static/assets/ too. When you don't want to change an existing physical layout, take the alias + matched-trailing-slash route instead.)

To audit an existing config, use nginx -T (below) to dump all locations, and eyeball just the blocks containing alias for the trailing-slash correspondence. To reproduce it at hand, poke one level up with curl: if curl -sI 'http://localhost/assets../' returns not a 200 or 403 but "something that shouldn't be visible," you've got a hit.

Security level: it works exactly as intended, which is why it's dangerous

The semantic-level traps were "behavior that differs from intent." From here, the quality changes. The config works exactly as you intended. It just also works exactly as the attacker intended. So it passes not only nginx -t but even a visual review, on a "well, it's working" basis.

Passing a client-touchable variable straight into `proxy_pass`

Using a variable for the proxy_pass destination lets you decide the destination dynamically per request. Handy. But it's a different story if the source of that variable is client-derived.

location /fetch/ {
    proxy_pass http://$arg_target;   # ?target=... decides the destination
}

With a request like /fetch/?target=internal-admin:8080/, nginx can relay the request to any host inside your network. This is a textbook entry point for SSRF (server-side request forgery, an attack that uses your server as a stepping stone to send requests to places it otherwise couldn't reach). Including a variable in proxy_pass also changes how the URI is handled and how name resolution behaves (you need a resolver if you use a hostname), and while you're distracted by those details, it's easy to overlook the danger that the destination is client-controlled.

I won't chase this further here — SSRF is a topic that deserves its own article. The one thing I want to nail down in this piece: "nginx -t passes" and "it's safe" are entirely separate things. Whether a value the client can influence flows into proxy_pass, a rewrite destination, or a root/alias path — that perspective is completely outside the syntax check.

Runtime level: the config text is no longer relevant

The last layer has a different flavor. Here the contents of the config file are correct. What "looks" broken is the behavior of the reload operation itself. Stare at the config and the answer isn't written there.

After a reload, the old behavior lingers for a while

You fix the config and nginx -s reload. It succeeds. And yet, for a while, the old behavior is observed. It looks like a bug, but this is nginx's normal behavior.

When it receives a reload (SIGHUP to the master process), nginx doesn't suddenly switch everything over. It starts new worker processes with the new config and tells the old workers to shut down gracefully. Gracefully — that's the point. The old workers stop accepting new connections, but they take care of requests already in flight to the end before terminating.

Normally this finishes in an instant, so you don't notice. The problem is when there are long-lived connections. WebSockets, large file downloads, long polling. An old worker holding one of these stays put until that connection ends. In other words, a window forms where the old and new configs are alive at the same time. That's the true identity of "the old behavior lingers after a reload." The config has been updated. It's just that a worker with the old config is still running.

Detection: This one you look at processes, not the config.

ps -eo pid,ppid,command | grep '[n]ginx'

Old workers linger with the label nginx: worker process is shutting down. If this "shutting down" keeps sitting alongside normal workers (nginx: worker process), that's the substance of old-and-new coexisting. If you want to cap how long they linger, set worker_shutdown_timeout:

worker_shutdown_timeout 30s;   # force old workers to terminate after at most 30 seconds

The important thing here is not to chase this as a bug. The cause isn't written in the config file — it's in the single fact that "reload is that kind of operation."

Also, confirming worker generations with ps requires being able to log into that server. I'm also working on a small tool that peeks — from the outside, without touching the running nginx at all — at which worker generation is holding connections after a reload, using eBPF (ngxray, work in progress).

Summary: how to fill in the space outside the syntax check

nginx -t doesn't lie. It just answers a narrow question. "Is it in a loadable shape?" — it answers that. "Is it read as intended?" "Can it be abused by an attacker?" "What happens during a reload?" — it was never answering those to begin with.

Each of the three layers is filled in differently.

Semantic level (if, location priority, add_header inheritance drops) can be caught by looking at the actual response. Marker headers and the difference in curl -I. Get into the habit of comparing behavior with curl -sI rather than reading the config and convincing yourself. Flattening the "config that's actually in effect" once with nginx -T (uppercase; it dumps the final config with all includes expanded) also helps.
Security level is about tracing where client-derived values flow. Whether external input reaches proxy_pass, a destination, or a path.
Runtime level is about looking at processes and connections, not the config. Check worker generations with ps.

The syntax check is only the first sheet of the layers. From the second sheet down, for now, I fill it in with human observation and discipline. Having a machine watch this space — reading the config not just by syntax but by "meaning," warning about location overtakes and header inheritance drops — static analysis, and a mechanism that continuously observes the actual responses, is the topic I want to think about next.

For now, one line is all I want you to take home. When nginx -t passes, the next thing to hit is curl -sI. That the syntax is correct and that it works as intended are two things you confirm separately.

The static analysis tools that automate parts of this gap — Gixy for security checks, crossplane for structural validation, Test::Nginx for behavioral testing — are mapped in a companion article.

References (official documentation)

The behavior in this article can all be backed up by official documentation. When you suspect your config, this is the first place to check.

ngx_http_rewrite_module (if) and If Is Evil — the story of if creating an implicit location
The location directive — the selection algorithm (longest prefix and regex priority)
ngx_http_headers_module (add_header) — inheritance rules and add_header_inherit, added in 1.29.3
The alias directive — how the real file path is assembled
proxy_pass — behavior when using a variable in the destination, and resolver
Controlling nginx and worker_shutdown_timeout — reload and graceful shutdown

"How I Designed a Go Linter's Rule Engine So Adding a Rule Is One Line"

Kazu — Tue, 14 Jul 2026 13:02:00 +0000

"Just add one more check" — and before you know it, a switch grows another branch, the config grows another case, and you're copy-pasting the same setup you already wrote somewhere else. If you've ever worked on code where rules or checks keep piling up, you probably recognize this kind of rot.

gomarklint is a Markdown linter, and it currently ships 24 checks (rules). Yet adding a new one touches just a function, one line in a table, and two spots in the config. The switch doesn't grow. No wiring to add. The cost of adding a rule stays flat no matter how many there are. This post takes that apart from the inside — how that state is kept.

First, let's run gomarklint

Before we go inside, let's see what the tool actually does, once. gomarklint is simple: it reads Markdown files, finds structural and link problems, and points at them with a file:line. Say you have a README with a link that goes nowhere.

# My Project

See the [docs]() for details.

Run gomarklint on it and you get:

$ gomarklint README.md

Errors in README.md:
  README.md:3: [error] no-empty-links: link has empty destination: [docs]()

✖ 1 issues found
✓ Checked 1 file(s), 3 line(s) in 1ms

Because an [error] was printed, the exit code is non-zero. In CI, this is where the build fails.

When this post says "rule," it means one of those checks that produces a single line of output like that. "Find empty links." "Check that heading levels don't skip." "Check for duplicate headings." There are 24 such checks today, and each one returns "which line, and what's wrong." gomarklint's own job is to collect those findings (the LintError you'll see in code shortly), sort them, and print them.

Here's the real question. How do you keep those 24 checks at "adding one is a one-line change"? We'll open it up in order — the diagnostic type, how rules are laid out, the shared preprocessing that skips code and HTML, and how severity wires into the exit code — and finish by adding a real rule through an actual PR (#106).

Even if you're not writing a linter — a validator, a CI check, a rule-based plugin system — if you write "a mechanism where items keep growing" in Go, this should be a template you can copy directly.

Rules pile up, and code usually rots

Take a check like that "find empty links" one, and now imagine 24 of them. Built naively, the dispatch and configuration rot a little more with every rule you add. The usual four traps:

Every new rule makes some switch longer. You can't tell where a new rule should be registered, so you get missed registrations and double registrations.
Small setup logic like "don't flag [x]() inside a code block" gets reimplemented in each rule. Fix it in one, and another stays stale.
The items you can write in the config file and the default values in code become two sources of truth, and one gets updated while the other drifts.
The "error or warning?" decision — the same one that just set our exit code — plus its counting and its effect on the exit code get scattered around, and CI fails or passes not quite as intended.

gomarklint would have stepped on every one of these if left alone. This post walks through how each is avoided, in order.

First, express diagnostics, config, and severity as minimal types

At the center of the engine sit three small types. Everything wobbles if these wobble, so they're kept deliberately plain.

The first is the violation itself.

type LintError struct {
    File     string
    Line     int
    Rule     string
    Message  string
    Severity string
}

The point is that Rule and Severity are in there but are filled in by the engine, not the rule — as we'll see later. A rule function only has to return "which line, and what happened."

The second is per-rule configuration.

type RuleConfig struct {
    Enabled  bool
    Severity RuleSeverity
    Options  map[string]interface{}
}

The third is severity, which has only three values.

const (
    SeverityError   RuleSeverity = "error"   // presence of this makes the exit non-zero
    SeverityWarning RuleSeverity = "warning" // reported, but doesn't fail the run
    SeverityOff     RuleSeverity = "off"     // disabled
)

If there's an error, CI fails. Hold onto just that one fact, and the rest of the branching comes back to it.

Rules are functions, not an interface

This is what changed the most in the engine.

The obvious design is to define a Rule interface and make each rule implement Check(). gomarklint doesn't do that. A rule is just a function, and the functions are laid out in a table.

They even split into two families by signature. One takes just the raw slice of lines — the simple rules.

var simpleRules = []struct {
    name string
    fn   func(string, []string, int) []rule.LintError
}{
    {"final-blank-line", rule.CheckFinalBlankLine},
}

The other takes a shared preprocessing result, *preprocess.Context, which we'll get to. Most rules live on this side now.

var contextRules = []struct {
    name string
    fn   func(string, *preprocess.Context, int) []rule.LintError
}{
    {"no-bare-urls", rule.CheckNoBareURLs},
    {"single-h1", rule.CheckSingleH1},
    {"duplicate-heading", rule.CheckDuplicateHeadings},
    {"unclosed-code-block", rule.CheckUnclosedCodeBlocks},
    {"empty-alt-text", rule.CheckEmptyAltText},
    {"no-empty-links", rule.CheckNoEmptyLinks},
    // …and more
}

Running the rules is then just iterating that table.

for _, r := range contextRules {
    if l.config.IsEnabled(r.name) {
        errs = append(errs, l.withSeverity(r.fn(path, ctx, offset), r.name)...)
    }
}

Only the rules that take options — like heading-level's minLevel or the style of consistent-*, and max-line-length — are kept out of the table and listed below with explicit ifs, because they take extra arguments. The trade is: "the option-free majority goes in a table, the option-taking minority is spelled out."

There's a cost to not using an interface. You can't do dynamic extension like dropping in a rule as a plugin at runtime. But the payoff was bigger.

Rules become stateless pure functions. Give input, get output — so tests fall straight out as table-driven.
Which rules run is fully visible by reading one slice. Nothing hides behind an abstraction.
Registering a new rule is, literally, one line in a slice.

Write "skip inside code blocks" once

Of those traps, the one that hurts most quietly is "the logic to skip inside code blocks scattered across every rule."

The heading rule, the link rule, the emphasis rule — all of them want to say "but not inside a fenced code block." Done naively, each rule starts counting "am I inside a fence right now?" on its own. And then one rule misses indented code blocks, another misses HTML comments — inconsistencies creep in. This actually happened in gomarklint, and we took inventory of it in issue #337.

The answer is preprocess.Scan. It walks the file's lines exactly once, classifies each line into "fenced code / indented code / HTML block / HTML comment," and hands it to every rule as a *preprocess.Context.

ctx := preprocess.Scan(lines)
allErrors := l.collectLineErrors(path, lines, ctx, offset)

The rule side no longer counts anything itself. It just asks the shared verdict.

if inBlockContext(ctx, i) {
    continue // inside code/HTML, so this rule ignores it
}

The clever bit here is that Context deliberately does not offer a single "is this line skippable?" convenience method. It exposes the four contexts individually. There's a reason: max-line-length and no-hard-tabs, for instance, do want to look inside fenced code (line length and tabs should be flagged even there). So "what to skip" is the rule's choice. The inBlockContext helper for the majority is just a thin OR over those four.

func inBlockContext(ctx *preprocess.Context, i int) bool {
    return ctx.InFencedCode(i) || ctx.InIndentedCode(i) ||
        ctx.InHTMLBlock(i) || ctx.InHTMLComment(i)
}

Context is trimmed hard for speed. It borrows the input line slice rather than copying it. The context flags are packed one byte per line, and the "sanitized line" with inline code blanked out is stored in a map only for the lines that differ. Since Scan runs over every file, that one byte matters. But that's a story about designing for speed, so I'll leave the deep dive to the benchmarking post. Here the design win — "write the skip logic once and share it with everyone" — is the real point.

Severity and exit code are wired on the engine side

I said the rule functions fill in neither Rule nor Severity. What fills them is this withSeverity.

func (l *Linter) withSeverity(errs []rule.LintError, ruleName string) []rule.LintError {
    sev := l.config.RuleSeverity(ruleName)
    for i := range errs {
        errs[i].Rule = ruleName
        errs[i].Severity = sev
    }
    return errs
}

The rule body returns only the "facts," and whether it's an error or a warning is decided by the config. The same rule can be an error for one team and a warning for another. Finally, the number of errors is counted, and if there's even one, the exit is non-zero. The basis for the CI gate is consolidated into this one place.

Prevent config/code drift with a contract

The flexibility of .gomarklint.json comes from RuleConfig.UnmarshalJSON accepting three shorthands.

{
  "rules": {
    "no-empty-links": true,                        // enabled / error
    "no-trailing-punctuation": "warning",          // enabled / warning
    "max-line-length": { "enabled": true, "lineLength": 120 }  // full spec
  }
}

true means "enabled / error," "warning" means "enabled / warning," and an object is a full spec. The writer's convenience (the human) is absorbed by one method on the parser side.

Here's the two-sources-of-truth trap. There's a default definition in the Go code too, config.Default(), and the JSON string that gomarklint init emits (DefaultConfigJSON) carries the same list. These two are kept in sync by a manual contract — and the code says so in a comment. It is not auto-generated. When you add a rule, you update both; that's the promise.

Actually adding one: no-empty-links (MD042)

With the tooling in place, let's trace it through a real PR. It's no-empty-links (MD042 in markdownlint terms, empty-link detection), added in issue #106. It catches links whose destination is empty, like [text]().

1. Write one rule function

In internal/rule/no_empty_links.go, write one function with the contextRules signature. The parts we've built so far just work.

func CheckNoEmptyLinks(filename string, ctx *preprocess.Context, offset int) []LintError {
    var errs []LintError

    for i := 0; i < ctx.Len(); i++ {
        if inBlockContext(ctx, i) {
            continue // ignore links inside code/HTML
        }

        // Line with inline code blanked out, so a link-looking
        // `[x]()` inside a code span isn't a false positive.
        line := ctx.Sanitized(i)
        if !strings.Contains(line, "](") {
            continue
        }

        for _, match := range findEmptyLinks(line) {
            errs = append(errs, LintError{
                File:    filename,
                Line:    offset + i + 1,
                Message: fmt.Sprintf("no-empty-links: link has empty destination: %s", match),
            })
        }
    }
    return errs
}

All that's written here is the judgment of "what counts as an empty link" (findEmptyLinks catches [](), [](#), [](<>), and the image variants). "Skip code blocks," "blank out inline code," "adjust the line number by offset" — those all just call the shared parts that already exist. Rule and Severity aren't filled in — the engine adds them later.

2. Add one line to the table

var contextRules = []struct { /* … */ }{
    // …
    {"no-empty-links", rule.CheckNoEmptyLinks}, // ← just this
}

Dispatch is done. No switch, no wiring to add.

3. Register in config (two spots)

Add no-empty-links to both config.Default() and DefaultConfigJSON. This is the "keep in sync by hand" contract in practice.

4. Add tests

Write internal/rule/no_empty_links_test.go table-driven, like the other rules. Because the rule is a pure function, you just line up input lines and expected violations.

Of the four steps, the only one where you really "think" is step 1. The rest is one line, two spots, and a boilerplate test. Even at 24 rules, this shape hasn't broken.

Extensibility isn't adding layers

Many linters get slower and harder to extend as they grow, because every rule leans on a shared, complex framework. gomarklint went the other way. Each rule is an independent, stateless function, and only the thing everyone needs — the code/HTML context check — was pulled out into a single preprocessing pass. Simplicity scales better than abstraction. Even now, with the engine carrying 24 rules, that still holds.

The result is:

Adding a rule is one function + one line in a table + two config spots,
Which rules run is visible by reading a slice,
It scales linearly with repository size.

Extensibility is often assumed to mean stacking layers — but sometimes it's about not stacking them. If you ever want to enforce your own documentation style guide, you can copy this shape directly. Write one function, add one line to the table. That's it.

For the background on parsing and the overall architecture, my earlier post Inside gomarklint: Architecture, Rule Engine, and How to Extend It is also worth a look (the engine has since evolved into the shape described here).

shinagawa-web / gomarklint

Catch broken links before your readers do.

gomarklint

English | 日本語

Catch broken links before your readers do — and keep your Markdown clean while you're at it. 100,000+ lines in ~170ms, single binary, no Node.js required.

Quick install (macOS / Linux):

curl -fsSL https://raw.githubusercontent.com/shinagawa-web/gomarklint/main/install.sh | sh

Download binary (no Go required):

Download the latest binary for your platform from GitHub Releases.

# macOS / Linux
tar -xzf gomarklint_Darwin_x86_64.tar.gz
sudo mv gomarklint /usr/local/bin/
# or install to user-local directory (no sudo required)
mkdir -p ~/.local/bin && mv gomarklint ~/.local/bin/

# Windows (PowerShell)
Expand-Archive -Path gomarklint_Windows_x86_64.zip -DestinationPath "$env:LOCALAPPDATA\Programs\gomarklint"
# Add to PATH (run once)
[Environment]::SetEnvironmentVariable("PATH", $env:PATH + ";$env:LOCALAPPDATA\Programs\gomarklint", "User")

Via Homebrew:

brew install shinagawa-web/tap/gomarklint

Via npm:

npm install -g @shinagawa-web/gomarklint

Via go install:

go install github.com/shinagawa-web/gomarklint/v3@latest

Catch broken…

View on GitHub

Tired of re-recording my README demo GIF by hand, I turned the whole demo environment into code

Kazu — Tue, 07 Jul 2026 13:01:00 +0000

The demo GIF at the top of an OSS README. That little clip that shows you what a tool does in five seconds, before a single line of prose does. That one.

I used to re-record mine by hand. Every single time.

Fire up the screen recorder, tidy up the terminal, type the command, stop at a good moment, export to GIF. Three minutes if it goes well. It usually doesn't. I fat-finger one character in the command. I hit that final q a beat too early and the recording ends before the screen I wanted to show even appears. Again. This time I notice, after exporting, that the terminal font size is different from last time. Again.

That's how "just one GIF" would casually swallow 30 minutes.

Hand-recording produces a slightly different picture every time

The biggest problem with recording by hand is that the picture changes every time you redo it.

Font, size, color theme, shell prompt, window width. All of it depends on the state of the machine you recorded on, at that moment. My prompt is set up to show the current git branch, and once, after a re-record, the README GIF alone had some unrelated repo name baked into it.

The quieter problem is staleness. You tweak the UI a little. You add a line to the header. You change a keybinding. Every time, the GIF gets a bit more out of date. But re-recording is a pain, so it stays stale. The very top of the README goes on proudly showing a screen that no longer exists in the current version. The first thing a visitor sees is the thing that gets updated least.

Anyone who's shipped a CLI tool as OSS has probably felt this one.

A TUI demo is a whole other level of annoying

pgincident, a tool I maintain, is a TUI for the first response to a Postgres incident. It's the full-screen-in-the-terminal kind.

A TUI demo is hard in a way a single CLI command isn't. If there's no "state" on the screen, the demo means nothing.

For an ls GIF, you just run it in a directory with a few files and it looks fine. But what pgincident wants to show you is a screen of "what's happening in this Postgres right now." A long query is running. A lock wait queue is forming. There's a session that opened a transaction and walked away. Only when all of that is on the screen at once does the viewer go "ah, so that's what this tool is."

Connect to a clean, quiet Postgres where nothing is happening, and all you record is a screen of empty rows that says nothing. That's not a demo.

But manufacturing a real long query, a real lock, and a real idle-in-transaction all at once, conveniently, just for the demo — doing that by hand every time isn't realistic. To create a lock artificially you open two sessions, deliberately make one wait, keep that state alive while you start recording in a third terminal, and... just the choreography is a headache. It's a pain to do once, and the thought of reproducing it every time I change the UI means the GIF stays stale even longer.

The `.tape` file — writing terminal operations down

This is where vhs comes in. It's a Charm tool, and in one line: you write a script of terminal operations, and it runs them exactly as written and exports a GIF or MP4.

The script goes in a text file with a .tape extension. Type types characters, Enter presses enter, Sleep waits. Set pins down the appearance. This isn't a full reference article, so it's faster to just look at the real thing. Here's pgincident's demo.tape.

# pgincident demo
# Run via: ./scripts/record-demo.sh

Output docs/demo.gif

Set FontSize 14
Set Width 1200
Set Height 700
Set Theme "Dracula"
Set Shell "bash"

# Start pgincident with the dev config
Type "./pgincident --config dev/pgincident-dev.toml"
Sleep 500ms
Enter

# Overview screen — let it render and refresh once
Sleep 4s

# Navigate to Dashboard screen
Type "o"
Sleep 4s

# Move cursor down to a Long-running query row and open SQL detail
Type "j"
Sleep 300ms
Enter
Sleep 3s

# Close detail and quit
Type "q"
Sleep 500ms

You can pretty much read what it does. Font size, window width, height, color theme — all pinned explicitly by the Set lines up top. None of my machine's settings leak in. Type "o" moves to the Dashboard screen, Type "j" moves the cursor down a row, Enter opens the SQL detail — the gaze of a README reader, turned straight into a script.

Even this alone almost entirely kills the "slightly different picture every time" problem. Same script, same picture. No re-shoots for typos. If I mistype something, I fix one character in the .tape and run it again.

But `.tape` alone is only half of it

So far the operations have become code. But remember the TUI difficulty from earlier. The "state" that shows up on screen still isn't set up anywhere.

A script that launches ./pgincident is useless if the Postgres it connects to is empty — all you record is an empty screen. The script only knows how to operate. It doesn't know what should be on the screen.

So I wrapped the .tape in one more script from the outside. That's scripts/record-demo.sh.

#!/usr/bin/env bash
set -euo pipefail

cd "$(dirname "$0")/.."

echo "==> Building pgincident..."
go build -o pgincident ./cmd/pgincident

echo "==> Starting Postgres..."
docker compose up -d postgres

echo "==> Waiting for Postgres to be ready..."
until docker compose exec -T postgres pg_isready -U postgres -q; do
  sleep 1
done

echo "==> Starting load generator..."
docker compose exec -T postgres psql -U postgres < dev/loadgen_setup.sql
go run ./cmd/pgincident-loadgen &
LOADGEN_PID=$!
trap 'kill $LOADGEN_PID 2>/dev/null || true' EXIT

echo "==> Waiting 5s for load to build up..."
sleep 5

echo "==> Recording demo..."
vhs demo.tape

echo "==> Done! -> docs/demo.gif"

Reading top to bottom, here's what it does. First it go builds pgincident. It starts Postgres with Docker. It waits until pg_isready says the connection is up. It pipes in the load-generation SQL and runs the load generator in the background. It waits 5 seconds for the load to build up. Only then does it call vhs demo.tape. When the recording finishes, the trap kills the load generator and cleans up.

The .tape records the operations, and this script sets up the state that shows on screen. Build, Postgres startup, load generation, recording, cleanup — all folded into a single script, so the whole demo environment is reproducible. This is the part I most wanted to say. Unless you widen "record the demo" into "assemble the demo environment and record it," you can't fully turn a stateful TUI demo into code.

You type ./scripts/record-demo.sh. After that, with a real long query, a real lock, and a real idle-in-transaction lined up on screen, one GIF gets generated. The manual choreography that used to give me a headache became a single command.

What actually creates that "state"?

Inside record-demo.sh, the line that looks the most unremarkable and does the most work is go run ./cmd/pgincident-loadgen. That's the part that artificially stokes the "real incident" you see on screen.

The prep is dev/loadgen_setup.sql. It's an unremarkable script that just creates two small tables for the load to hit.

CREATE TABLE IF NOT EXISTS loadgen_accounts (
    id         bigint      PRIMARY KEY,
    balance    numeric(12,2) NOT NULL DEFAULT 0,
    touched_at timestamptz   NOT NULL DEFAULT now()
);

-- 10k rows gives cache hit ratio room to move
INSERT INTO loadgen_accounts (id, balance)
SELECT i, (random() * 10000)::numeric(12,2)
FROM   generate_series(1, 10000) AS g(i)
ON CONFLICT DO NOTHING;

-- pgincident_dev only has pg_monitor privileges. Grant DML so the simulator can run
GRANT SELECT, UPDATE ON loadgen_accounts TO pgincident_dev;

Here's where it gets interesting. The load generator itself (Go) deliberately produces, one by one, the three categories that pgincident's Dashboard shows.

Long-running queries come from analytical queries with a pg_sleep baked in. It runs short ones that finish in a few seconds and long ones that squat for 5–8 minutes, with staggered start times.

WITH paused AS (SELECT pg_sleep($1))
SELECT a.id, a.balance, b.id, b.balance
FROM   paused, loadgen_accounts a
JOIN   loadgen_accounts b ON b.id != a.id
WHERE  a.id = $2 AND b.id = $3

Lock waits come from the classic wait queue: one session grabs a row with FOR UPDATE and holds it, while another goes for the same row and gets stuck waiting.
Idle in transaction comes from a BEGIN, one query, and then just sitting there without committing or rolling back. pgincident's threshold is 30 seconds, so it sleeps long enough to cross it.
On top of that, a light OLTP load (a mix of SELECT / UPDATE) runs in the background so that TPS and cache hit ratio look "alive" on screen.

And here's the biggest trick for the sake of the recording: each worker is written to start on a staggered schedule. The idle ones so that "at least one is always visibly past the 30-second threshold," the long queries so that "two cycles half-overlap." The point is that the load itself is designed so that whenever vhs demo.tape runs, the Dashboard's three categories are all filled in at once, just right.

"Turning the whole demo environment into code," taken to its conclusion, includes this. Writing operations in a .tape will never produce this "broken just right" state.

The README just embeds this GIF as the top image.

![pgincident demo](docs/demo.gif)

What changed after turning it into code

Honestly, I didn't invent anything dramatic. Neither vhs nor a shell script is a novel technology. Even so, the day-to-day feel clearly changed.

Anyone who runs it gets the same look and the same composition. Font, size, theme, the way load is applied — it's all in the script. It won't match down to which query lands in the top row on any given run, but the look and composition of the picture don't depend on my machine's mood. When a future contributor wants to update the GIF, they can read the .tape and know what's on screen.

It reviews in a diff and lives in git. A GIF is binary, so you can't read the diff of its contents, but .tape and record-demo.sh are just text. "Bumped a Sleep from 3s to 4s," "reordered the operations" — that shows up in the PR diff. Demo changes ride on the same rails as code changes.

Change the UI, regenerate. Added a line to the header. Changed a keybinding. Before, it was "re-recording is a pain, so leave it." Now it's just typing ./scripts/record-demo.sh again. I won't claim staleness is gone — forget to run the regen and it stays stale. But the weight of the "it's a pain" excuse dropped from 30 minutes of manual work to a single command.

Let me be honest about one thing too. Right now record-demo.sh is a local script I run by hand. I'm not auto-regenerating the GIF in CI. When I change the UI, I still have to notice "ah, gotta regenerate the GIF" myself and type it myself. Ideally there's room to get to where any UI-touching change makes CI re-bake the GIF and commit it. That's the next move. What I've got so far is that regeneration is one command.

Gotchas and tips

A few spots that actually took some doing.

Tuning Sleep takes the most work. A TUI refreshes the screen periodically. Right after launch, the data isn't drawn yet. If Sleep is too short, you record an empty or half-drawn screen. The reason demo.tape waits a generous Sleep 4s after launch is to let the Overview screen refresh once and fill in. Too long, though, and the GIF drags. I went back and forth a few times here — "shorten it, record, didn't make it, lengthen it a bit." It's an adjustment a single CLI command never needs; it's a TUI-specific chore.

How well the load is crafted decides how convincing the demo is. If the load the generator throws is too weak, not a single long query crosses the threshold and the screen stays sparse. Too strong and the screen fills up so much you can't tell what to look at. Creating the "broken just right" state is unglamorous but the single biggest thing for demo-ness.

Don't mix in secrets. The script is text and goes into git. And of course the GIF bakes in exactly the characters you typed and the screen you showed. Be careful not to write a production DSN or password into a Type, or to record a screen connected to production. That pgincident's demo uses a dev config (dev/pgincident-dev.toml) and connects to a local Docker Postgres is partly for this reason.

GIF size. Crank up Set Width / Set Height and the file naturally gets heavier. This image sits at the top of the README, so slow loading defeats the purpose. Width 1200 × height 700 was the sweet spot where the text isn't crushed and the file isn't too heavy.

Demos go from something you "record" to something you "write and generate"

Looking back, what changed is the kind of work. From the manual labor of "recording" a demo — moving your hands, failing, re-shooting — to "writing and generating" one: turning the script and the environment into code, and baking it with a command.

And for a stateful TUI demo, writing the operations in a .tape isn't enough. Only when you go all the way to setting up the "state" on screen — a real long query, a real lock — and turn the whole demo environment into code does it become reproducible. That, I think, is the key thing when you turn a TUI demo into code.

If you're re-recording your own OSS README demo GIF by hand every time, start with a single .tape. And if the screen you want to show needs "state," try folding the setup of that state into the script too.

By the way, what pgincident actually is as a tool, I wrote up in another article. It's the story of that first 30 seconds when Postgres slows down at 2 a.m.