DEV Community: Kazu

Slow 200s in nginx, fast in the application server's logs

Kazu — Fri, 30 Oct 2026 13:02:00 +0000

An nginx latency alert fires, yet the logs and metrics of the application server behind nginx can show the same requests finishing quickly. This happens when the application server stops entirely for a few seconds, for example during a full GC¹, and connections that arrive in the meantime are kept waiting.

sequenceDiagram
    participant N as nginx
    participant K as Application server's kernel
    participant U as Application server
    Note over U: Stopped by a full GC or similar
    N->>K: SYN (open a connection)
    Note over K: The accept queue is full, so the SYN is dropped

The dropped SYN is resent every second and connects after the application server resumes. From nginx, these look like 200s that took about 1, 2, or 3 seconds to connect. The three lines below are from the nginx access log, which records $upstream_connect_time (the seconds nginx took to finish connecting to the application server) as uct. The columns are time, request ID, status, uct, $upstream_header_time (uht), $upstream_response_time (urt), and $request_time (rt). uht and urt are the seconds from when nginx starts connecting to the application server until it receives the start of the response, and until it has received the whole response. rt is the seconds from when nginx receives the request until it finishes sending the response.

1792402370.463 487aefe88a1328206a8e93c7c242399a 200 uct=1.002 uht=1.002 urt=1.002 rt=1.002
1792402370.464 cfdd11ceca67677e3601df796850a712 200 uct=2.067 uht=2.067 urt=2.067 rt=2.068
1792402371.040 dd774240d10448d9dc41bfcaaa7d8dfe 200 uct=3.054 uht=3.055 urt=3.055 rt=3.054

The application server receives the request ID that nginx assigned, and its log records time, request ID, and processing time in milliseconds. These are its lines for the same request IDs:

1792402370.463 487aefe88a1328206a8e93c7c242399a 0.092
1792402370.463 cfdd11ceca67677e3601df796850a712 0.104
1792402371.040 dd774240d10448d9dc41bfcaaa7d8dfe 0.084

Nearly all of the time went to connecting, and processing on the application server took about 0.1 ms. The application server starts timing only after it takes the connection off the queue, so the wait to connect does not appear in its logs or metrics.

The logs above were taken from an application server that receives 200 requests per second, stopped for 3 seconds².

What is happening

When the application server stops, the accept queue fills up. SYNs that arrive after that are dropped, and connections are made through retransmissions by the nginx-side kernel.

The accept queue

The kernel carries out the exchange that sets up a TCP connection (the handshake). The connecting side sends a SYN to request a connection, the receiving side replies with a SYN-ACK to accept it, and the handshake completes when the connecting side sends an ACK. A completed connection waits in the kernel's accept queue until the server process takes it with the accept system call.

Even while the application server is stopped, the kernel on its side keeps completing handshakes and queuing the connections. Nothing takes connections off the queue during the stop, so the queue fills up.

sequenceDiagram
    participant N as nginx
    participant NK as nginx-side kernel
    participant UK as Application server's kernel
    participant U as Application server
    Note over U: Stopped
    N->>NK: Connect to the application server
    NK->>UK: SYN
    UK-->>NK: SYN-ACK
    NK->>UK: ACK
    NK-->>N: Connected
    Note over UK: Puts the connection in the accept queue
    Note over U: On resuming, takes connections off the queue in order

The maximum number of connections the queue can hold is the backlog. The application server passes the backlog to the kernel when it starts listening for connections (when it calls listen). The value comes from the runtime's or framework's default, or from configuration.

SYNs are dropped when the queue is full

When the number of connections in the queue exceeds the backlog, the application server's kernel drops newly arriving SYNs without replying³.

Retransmissions get through

When a SYN gets no reply, the nginx-side kernel waits and sends it again. This is SYN retransmission. If the nginx-side kernel is Linux 6.5 or later, the first few retransmissions go out at 1-second intervals⁴.

Retransmitted SYNs are dropped too while the queue is full. Once the application server resumes and the queue has room, the next retransmission gets a SYN-ACK and the connection is made. Depending on which retransmission gets through, $upstream_connect_time is about 1, 2, or 3 seconds. The three log lines at the top are these cases.

sequenceDiagram
    participant N as nginx
    participant NK as nginx-side kernel
    participant UK as Application server's kernel
    participant U as Application server
    Note over U: Stopped
    Note over UK: The accept queue is full
    N->>NK: Connect to the application server
    NK->>UK: SYN
    Note over UK: Drops it without replying
    loop Every second while the queue is full
        NK->>UK: Resend the SYN
        Note over UK: Drops it again
    end
    Note over U: Resumes and takes connections off the queue
    NK->>UK: Next retransmission
    UK-->>NK: SYN-ACK
    NK->>UK: ACK
    NK-->>N: Connected ($upstream_connect_time is about 1, 2, or 3 seconds)

Verification

I stopped the application server for 3 seconds to confirm this sequence. Requests that arrived during the stop split into those that connected immediately and those that connected after waiting about 1, 2, or 3 seconds.

Setup

nginx and the application server ran as containers on the same Docker network, and nginx was sent 200 requests per second for 15 seconds. Five seconds in, the application server process was stopped from outside with SIGSTOP, then resumed with SIGCONT 3 seconds later. This mimics a full GC stopping the whole process.

nginx is 1.30.6, with a single application server in the upstream block. proxy_connect_timeout (how long nginx waits to finish connecting to the application server) is left at its default of 60 seconds
The application server is an HTTP server that returns 200 immediately for every request, and listens with a backlog of 100

Time to connect

In the 3 seconds between the stop and the resume, 600 requests reached nginx, and all of them returned 200. The counts by $upstream_connect_time are:

`$upstream_connect_time`	Requests
0 seconds (connected immediately)	102
About 1 second	204
About 2 seconds	205
About 3 seconds	89

This holds beyond the three lines at the top: for all 600 requests, the processing time in the application server's log was under 1 ms.

Which row a request falls into depends on when it arrived. The queue was full about 0.5 seconds after the stop. Connections that arrived before that went into the queue and connected immediately. Connections that arrived later had their SYNs dropped, the nginx-side kernel resent them every second, and they connected on the first retransmission after the application server resumed. The earlier a connection arrived, the more times its SYN was resent and the longer it waited.

Spotting it in your own logs

To tell whether the accept queue overflowed, match the nginx access log against the application server's log. The access log shows that SYNs were dropped and the connections were made by retransmission. The application server's log shows that the application server was stopped.

Access log

In the access log, look at $upstream_connect_time on the 200 lines from when the alert fired. If the values cluster around 1, 2, and 3 seconds, SYNs were dropped and the connections were made by retransmission.

$upstream_connect_time is not part of the default combined format, so add it with log_format⁵. The verification used this format:

log_format lab '$msec $request_id $status uct=$upstream_connect_time uht=$upstream_header_time urt=$upstream_response_time rt=$request_time';

Application server log

In the application server's log, find the lines with the same IDs as the requests whose $upstream_connect_time came in 1-second steps, and look at how the lines are spaced around that time. If those requests finished quickly and the lines stop for a while around that time, the application server was stopped. The lines after the gap are bunched at the time it resumed.

To match the two logs, pass the ID that nginx assigns to each request ($request_id) to the application server. On the nginx side, this takes one line:

proxy_set_header X-Request-Id $request_id;

On the application server side, its access log needs to be configured to include the received X-Request-Id header.

Fix

If the application server's backlog is set larger than the number of requests that arrive during a stop (requests per second × seconds stopped), the queue does not overflow⁶. With a backlog of 4096, the 200s of about 1, 2, and 3 seconds no longer appear.

The default backlogs of JVM servers, which can stop for a full GC, are:

Tomcat and Spring Boot: 100⁷
Jetty: 50⁸

Even without an overflow, requests that arrive during a stop wait in the queue until the application server resumes. That wait shows up in $upstream_header_time, so 200s slowed by the length of the stop remain.

How this varies by environment

With a shortened connect timeout

If proxy_connect_timeout is shorter than the stop, nginx gives up on connections whose retransmissions have not gotten through in time and returns 504.

When the stop is not 3 seconds

A stop of a different length should change the values $upstream_connect_time clusters around. This article tests a 3-second stop.

Kernels older than 6.5

If the nginx-side kernel is older than 6.5, $upstream_connect_time clusters around 1 and 3 seconds, and about 2 seconds does not appear. This is because the retransmission interval doubles each time: after 1 second, then 2 seconds after that, then 4 seconds after that.

The stock kernel of Ubuntu 24.04 (6.8) is 6.5 or later. Those of Ubuntu 22.04 (5.15), Debian 12 (6.1), and RHEL 9 (5.14) are older. Whether a distribution has backported this change into its own kernel has not been checked.

With a proxy in front of the application server

If a proxy that accepts connections first sits between nginx and the application server (Docker's userland-proxy, which handles published ports, or a sidecar proxy), that proxy answers the SYNs from nginx. Even if the application server's queue overflows, no retransmissions happen on the nginx side, so the method in this article should not apply. This article does not verify this case.

Ordinary GC pauses are not this long. The JVM's G1 collector has a default pause-time goal of 200 ms (G1Arguments::initialize() in JDK 25). Pauses of several seconds come from things like a full GC under heap pressure, which this article does not verify. ↩
The results are here, and the code is in show-your-work. It ran on GitHub Actions ubuntu-24.04 (x86_64, kernel 6.17) with Docker 28.0.4. ↩
In the Linux 6.17 source, tcp_conn_request() drops the SYN when the queue is full. sk_acceptq_is_full() decides whether it is full, and treats the queue as full when the number of connections in it is greater than (>) the backlog. ↩
The number of retransmissions sent at 1-second intervals is set by net.ipv4.tcp_syn_linear_timeouts (default 4; see the Linux 6.17 documentation). ↩
Adding variables with log_format is covered in nginx request_time is fast, but users say it's slow. ↩
To go above 4096, also raise net.core.somaxconn in the application server's kernel. The kernel caps the backlog at somaxconn (__sys_listen_socket() in Linux 6.17), and its default is 4096 (preinit_net_sysctl(), documentation). ↩
The Tomcat 11 documentation on HTTP connector configuration, and TomcatServerProperties in the Spring Boot 4.1.1 source. ↩
Jetty passes 0, and the JDK replaces it with 50. The default is 0 in ServerConnector in the Jetty 12.1.14 source, and ServerSocketChannelImpl in the JDK 25 source turns any value below 1 into 50. ↩

HPA does not keep the replicas you set with kubectl scale

Kazu — Tue, 27 Oct 2026 13:00:00 +0000

Sometimes you want more Pods even though CPU usage is low: when the workers in each Pod are all tied up waiting on a downstream response, or when you want capacity in place before an event. In those cases, people sometimes raise replicas with kubectl scale, even on a Deployment that has an HPA (HorizontalPodAutoscaler). HPA is a controller that decides the number of Pods from metrics such as CPU usage and rewrites the Deployment's replicas.

Right after the scale-up, READY in kubectl get deploy matches the new count. HPA does not revert it right away either, so the count looks like it will hold.

A few minutes later, it is back to the original count. No errors or warnings appear. If HPA reverted it, it is not clear why it took a few minutes to do so.

I reproduced this state and checked whether HPA was really what reverted it, and why the revert happens a few minutes later.

Reproduction

I reproduced the state above on a Kubernetes v1.37.0 cluster created with kind.¹

I attached an HPA to a Deployment that runs 2 Pods using almost no CPU. The HPA targets 50% CPU utilization, with minReplicas 2 and maxReplicas 5. HPA decides the count again every 15 seconds. Below, each of these runs is called a sync.

$ kubectl get hpa c1-1
NAME   REFERENCE         TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
c1-1   Deployment/c1-1   cpu: 0%/50%   2         5         2          81s

I scaled this Deployment from 2 to 4 with kubectl scale.

$ kubectl scale deploy/c1-1 --replicas=4
deployment.apps/c1-1 scaled

Right after the scale-up, READY reached 4/4.

$ kubectl get deploy c1-1
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
c1-1   4/4     4            4           107s

About 5 minutes later, it went back to 2.

$ kubectl get deploy c1-1
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
c1-1   2/2     2            2           7m17s

Narrowing it down

Deployment replicas can be written back by kubectl apply run by another person or by CI, by GitOps tools, and by controllers such as HPA.

Kubernetes objects carry managedFields, a record of which manager (writer) wrote each field of the object.² Viewing the Deployment after it went back to 2 with kubectl get deploy c1-1 -o yaml --show-managed-fields showed that the manager of spec.replicas was kube-controller-manager.

  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:replicas: {}
    manager: kube-controller-manager
    operation: Update
    subresource: scale

kube-controller-manager is the in-cluster process that runs HPA and the other controllers. If kubectl apply or a GitOps tool had written the field, the manager would be some name other than kube-controller-manager.

Events show which controller wrote it. An Event is a record of something that happened, which Kubernetes keeps per object. kubectl events --for hpa/c1-1 showed a SuccessfulRescale at the time of the revert.³ New size: 2 means HPA rewrote replicas to 2. HPA was what reverted it.

$ kubectl events --for hpa/c1-1
LAST SEEN               TYPE      REASON                         OBJECT                         MESSAGE
16s                     Normal    SuccessfulRescale              HorizontalPodAutoscaler/c1-1   New size: 2; reason: All metrics below target

What remains of the opening question is why HPA takes about 5 minutes to revert.

What is happening

What a sync does

On each sync, HPA does the following.

Reads the Pods' CPU utilization
Divides the average utilization by the target to get a ratio, and computes a Pod count from the ratio. Below, this count is called the computed count. If the ratio is between 0.9 and 1.1, the computed count is the current count (the Deployment's spec.replicas). Outside that range, it is the current number of Pods multiplied by the ratio, rounded up.⁴
When scaling down, does not go below the maximum computed count over the past 5 minutes. It then clamps the result to the range from minReplicas to maxReplicas.⁵
If the count decided in step 3 differs from the Deployment's spec.replicas on the cluster, rewrites spec.replicas to that count

The "5 minutes" in step 3 is the value used when the HPA has no behavior, and it comes from the default of kube-controller-manager's --horizontal-pod-autoscaler-downscale-stabilization.⁶ It can be changed per HPA with behavior.scaleDown.stabilizationWindowSeconds.

From the scale-up to the revert

The table below shows the syncs from the scale-up to the revert. Seconds are the time elapsed since running kubectl scale. The computed counts were taken from kube-controller-manager logs in the reproduction environment, with --v=4 (a setting that makes the logs more detailed).

Since the operation	Metrics for the 2 new Pods	Computed count	replicas
8 s	none	4	4
23 s	none	4	4
38 s	present (0%)	0	4
(same every 15 s)
323 s	present (0%)	0	2

At the 8 s and 23 s syncs, the 2 new Pods are running but have no metrics yet. metrics-server is the component that collects Pod CPU and memory usage from each node and passes it to HPA. Values for a Pod that has just started do not appear right away.

When the ratio computed from only the Pods that have metrics is below 1, HPA recomputes it, treating the Pods without metrics as 100% utilized.⁷ Here the 2 existing Pods are at 0% and the 2 new ones at 100%, so the average is 50%. The ratio to the 50% target is 1.0, so in step 2 the computed count is the current count, 4.

From the 38 s sync on, metrics for the 2 new Pods are available and the computed count becomes 0. In step 3, though, HPA does not go below 4, the maximum computed count over the past 5 minutes, so replicas stays at 4.

At the 323 s sync, the computed count of 4 from 23 s is now more than 5 minutes old, and the maximum computed count over the past 5 minutes becomes 0. HPA clamps this to minReplicas, 2, and rewrites spec.replicas to 2.

To HPA this is an ordinary scale-down, so the only Event is a Normal SuccessfulRescale, with no errors or warnings.

How long the kubectl scale count lasts

HPA takes the replicas a person wrote with kubectl scale into its calculation as the current count. When scaling down, HPA preserves only the computed counts from the past 5 minutes.

The count a person wrote lasts only until 5 minutes after the computed count last matched it. In the reproduction, the 4 Pods lasted about 5 minutes because the computed count was 4 at the syncs before the 2 new Pods had metrics, and that 4 stayed as the maximum computed count over the past 5 minutes.

If the computed count never matches that number, the first sync reverts it. In the same setup, scaling from 2 to 3 went back to 2 at the sync about 10 seconds later. Even with the 1 Pod without metrics treated as 100% utilized, the average across 3 Pods is about 33%. The ratio to the 50% target is below 0.9, and the computed count is 2.

Whether the revert comes a few minutes later or right away, HPA is what does it.

Remedies

The two places to change are the HPA's lower bound (minReplicas) and its scale-down rules (behavior.scaleDown).

Raise `minReplicas`

Raise the HPA's minReplicas to the count you want. replicas goes up immediately. Only the lower bound goes up, so HPA can still add more Pods if load increases.

$ kubectl patch hpa <hpa> --type=merge -p '{"spec":{"minReplicas":4}}'
horizontalpodautoscaler.autoscaling/<hpa> patched

If the count you want is above maxReplicas, raise maxReplicas in the same patch.

Disable scale-down, then scale up

Set the HPA's behavior.scaleDown.selectPolicy to Disabled, then scale up with kubectl scale. HPA no longer reduces replicas.⁸

$ kubectl patch hpa <hpa> --type=merge -p '{"spec":{"behavior":{"scaleDown":{"selectPolicy":"Disabled"}}}}'
horizontalpodautoscaler.autoscaling/<hpa> patched
$ kubectl scale deploy/<deployment> --replicas=4
deployment.apps/<deployment> scaled

Which to choose

I choose to raise minReplicas. The only thing it stops is scaling down below the lower bound. HPA keeps adding Pods as load rises and keeps scaling down as far as the lower bound. Disabling scale-down also stops HPA from removing the Pods it added in response to load.

When you no longer need the extra Pods, set minReplicas back to its original value.

The results are here, and the code is in show-your-work. ↩
Field management in the Kubernetes documentation. ↩
Events are deleted after 1 hour by default, because the default of kube-apiserver's --event-ttl is 1h0m0s. ↩
Algorithm details in the Kubernetes documentation. ↩
normalizeDesiredReplicas in kube-controller-manager. ↩
kube-controller-manager's defaults. ↩
replica_calculator.go in kube-controller-manager. ↩
Example: disable scale down in the Kubernetes documentation. ↩

301 redirect change doesn't take effect for some users

Kazu — Fri, 23 Oct 2026 13:03:00 +0000

On the documentation site, /docs redirects to the pages for the latest version. Links on other sites, such as the product site, point to /docs. For the v4 release, the nginx config was edited to change the target from /docs/v3/ to /docs/v4/.

location = /docs { return 301 /docs/v4/; }

301 is the status code meaning the requested URL has moved permanently to another URL. The /docs URL itself will not change, so it was set up as a 301.

After the change, reports keep coming in: "When I open it from the link on the product site, I get v3." Running curl against /docs locally returns a 301 to v4.

$ curl -sI https://<host>/docs
HTTP/1.1 301 Moved Permanently
Location: https://<host>/docs/v4/

I reproduced this with nginx and browsers and measured why some users land on v3.

Impact

Some of the users who opened /docs from the product site link while it pointed to v3 keep reading the v3 pages after the v4 release. No error appears, so only the users who notice send a report. The ones who don't notice go on configuring and working by the v3 instructions.

This does not go away, however many days pass. It happens again every time a release changes the target.

Reproduction

I reproduced the scenario above with nginx 1.31.6 and three browsers driven by Playwright 1.63.0: Chromium, WebKit (the engine behind Safari), and Firefox.¹

A page containing only a link to /docs is served on a host name separate from the documentation site, and /docs is opened from that page. /docs responds with the same return 301 as above.

While the target is v3, open /docs from the link page. Then change the nginx config to v4 and reload. After that, open /docs from the link page three times: right after the switch, 70 seconds later, and after restarting the browser. Each time, record which version's page the browser landed on.

Chromium and WebKit landed on v3 all three times. Firefox landed on v4 from right after the switch.²

When /docs was opened while the target was v3, all three browsers left a /docs line in the nginx access log.

<client IP> - - [15/Oct/2026:23:13:51 +0000] "GET /docs HTTP/1.1" 301 169 ...

After the switch, there was no /docs line for any of the three times Chromium and WebKit landed on v3. Firefox, which landed on v4, left a /docs line all three times. Chromium and WebKit went to v3 without asking the server.

Why v3 keeps showing

The /docs 301 does not say how long it may be stored and reused. How to treat a 301 without that lifetime is up to the browser. Chromium treats it as never expiring and WebKit as valid for one year, and both reuse the stored 301 without asking the server.³⁴ So users who stored the 301 to v3 still land on v3 after the switch. Firefox does not reuse the stored 301 as is. It asks the server every time the page is opened.

These are the headers of the /docs 301:

$ curl -sI https://<host>/docs
HTTP/1.1 301 Moved Permanently
Server: nginx/1.31.6
Date: Thu, 15 Oct 2026 23:14:03 GMT
Content-Type: text/html
Content-Length: 169
Location: https://<host>/docs/v4/
Connection: keep-alive

Cache-Control: max-age=<seconds> is how a response tells the browser it may store the response and use it without asking for that many seconds. This response has no such header. Nor does it have Last-Modified, the header that gives the time the file was last modified.

Firefox computes the lifetime of a 301 without a stated lifetime from Last-Modified.⁵ This response has no Last-Modified, so the lifetime is 0, which means the response is already expired when it is stored. That is why Firefox asks the server every time.

Fix

Switch to a 302, or give the 301 a lifetime with max-age. Either way, users who have already stored the 301 to v3 stay on v3. Those users are covered in the notes at the end.

Use a 302

302 is the status code meaning the requested URL has moved temporarily to another URL. In nginx:

location = /docs { return 302 /docs/v4/; }

For a 302 with neither a stated lifetime nor Last-Modified, Chromium, WebKit, and Firefox all set the lifetime to 0 and ask the server for /docs every time it is opened.³⁴⁵

Add max-age to the 301

You can also keep the 301 and state a lifetime with max-age in Cache-Control. The verification used 60 seconds.

location = /docs {
    add_header Cache-Control "max-age=60";
    return 301 /docs/v4/;
}

Headers added with add_header are also attached to the 301 sent by return.

The tradeoff is that users who opened /docs just before a release land on the previous version for up to max-age seconds after the release.

Which to choose

I choose the 302. At the start, /docs got a 301 because its URL does not change. A 301, however, says that the move from /docs to its target is permanent. The target changes with every release, so the move is temporary.

I don't choose max-age. It saves one round trip to /docs during the lifetime, but in exchange, users who opened /docs just before a release land on the previous version.

Effect of the fixes

With a 302

With the config already using return 302 while the target was v3, I opened /docs with the same steps as the reproduction.²

Time	Chromium	WebKit	Firefox
Right after the switch	v4	v4	v4
70 seconds later	v4	v4	v4

All three browsers landed on v4 from right after the switch.

With max-age on the 301

With the config already returning the 301 with max-age=60 while the target was v3, I opened /docs with the same steps as the reproduction.²

Time	Chromium	WebKit	Firefox
Right after the switch	v3	v3	v3
70 seconds later	v4	v4	v4

Right after the switch, Firefox also used the stored 301 and landed on v3. At 70 seconds, after the 60-second lifetime had passed, all three landed on v4.

Notes

Affected browsers

Chromium-based browsers and Safari keep landing on v3. Firefox 139 and later are not affected. Firefox 138 and earlier, and the ESR releases based on them, are. Firefox 138 and earlier treated a 301 without a stated lifetime as never expiring.⁶

Chromium stores the 301 separately depending on whether the page was opened from a link on another site or typed into the address bar.⁷ In the verification, after the 301 to v3 was stored by opening /docs from the link page and the target was switched to v4, opening /docs directly, without going through a link on another site, took Chromium to v4 and WebKit to v3.²

To check this in your own Chrome, open /docs from a link on another site, the way users do.

Users who still land on v3 after changing to 302

I also ran the case where the 301 to v3 is stored first and the config is changed to return 302 at the switch, opening /docs with the same steps as the reproduction. Chromium and WebKit landed on v3 all three times: right after the switch, 70 seconds later, and after restarting the browser.² They keep using the stored 301 to v3, so the change to 302 never reaches these users.

Clear-Site-Data is a header that tells the browser receiving it to delete that host's cache and other stored data. In the verification, the v4 page, on the same origin as /docs, returned this header.

location /docs/v4/ { add_header Clear-Site-Data '"cache"'; }

For users who opened this page, the stored 301 was deleted, and Chromium and WebKit also landed on v4 from /docs.² The header only reaches users who open a v4 page. Users being sent to v3 never get to a v4 page from /docs.

Note that the header tells the browser to delete the entire cache for that host, not just the v4 page. The header also works only over HTTPS (or on localhost).⁸

Setups that give the redirect a lifetime

If expires is set at the server level, the 301 sent by return in a location also gets a Cache-Control header.⁹ In the verification, with expires 30d at the server level and only return 301 in the /docs location, the 301 for /docs got Cache-Control: max-age=2592000 (30 days).² Users land on the previous version for up to 30 days after the release.

Setups not tested

Sites with a registered Service Worker that answers requests to /docs with a locally held response
Setups with a CDN in front of nginx that stores the /docs 301 and serves it

In both, the request does not reach nginx, so the /docs lines in the access log cannot tell you whether the browser used a stored 301.

Run on GitHub Actions (ubuntu-24.04). Browser versions: Chromium 153.0.8010.12, WebKit 26.6, Firefox 155.0. The logs and curl output in the text have the per-condition paths and host names rewritten to /docs and <host>. The config, scripts, and raw logs are in show-your-work. ↩
Verification results: CI job ↩
Chromium's http_response_headers.cc (GetFreshnessLifetimes). A 301 without a stated lifetime never expires. A 302 gets 0. ↩
WebKit's CacheValidation.cpp (computeFreshnessLifetimeForHTTPFamily). A 301 without a stated lifetime gets one year. Anything else gets a lifetime computed from Last-Modified, or 0 if there is none. ↩
Firefox's nsHttpResponseHead.cpp (ComputeFreshnessLifetime). A 301 without a stated lifetime gets a lifetime computed from Last-Modified, or 0 if there is none. A 302 without a stated lifetime gets 0. ↩
Mozilla Bug 1957619. ↩
Chromium's http_cache.cc. Top-level navigations coming from another site are stored under a separate cache key. ↩
W3C's Clear Site Data. "cache" deletes the cache for the same host as the response URL (4.2.3). If the response URL is not treated as secure (HTTPS and localhost URLs are), nothing is deleted (4.2). ↩
nginx's ngx_http_headers_module. expires adds Expires and Cache-Control to responses with status code 301 as well. ↩

K8s upgrade removes cgroup paths and changes how you read the values

Kazu — Tue, 20 Oct 2026 13:01:00 +0000

After a Kubernetes version upgrade replaces the node image, procedures and scripts that read container CPU and memory values from under /sys/fs/cgroup stop working.

$ kubectl exec <pod> -- cat /sys/fs/cgroup/cpu/cpu.stat
cat: can't open '/sys/fs/cgroup/cpu/cpu.stat': No such file or directory
command terminated with exit code 1

Running ls inside the container shows no cpu/ and no memory/. cpu.stat and memory.stat, which used to live in separate directories, are listed directly. The node's /sys/fs/cgroup looks the same.

$ kubectl exec <pod> -- ls /sys/fs/cgroup
cgroup.controllers
cgroup.events
...
cpu.stat
...
memory.current
memory.events
...
memory.stat
...

The layout with one directory per controller is cgroup v1, and this single flat layout is cgroup v2. The switch from v1 to v2 happened when the node image changed.

Removing cpu/ from the script's path opens a file with the same name.

$ kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat
usage_usec 768869
user_usec 684571
system_usec 84297
nice_usec 0
core_sched.force_idle_usec 0
nr_periods 38
nr_throttled 37
throttled_usec 1592698
nr_bursts 0
burst_usec 0

The throttled_time line from the old cpu.stat is missing, and there is a throttled_usec line in its place. A script that collected throttled_time finds no matching line and returns an empty result without raising an error.

Even once the files open, some values read the v1 way come out different. This article covers three of them: CPU throttle time, memory usage, and OOM count.¹

It is written for people who read the files under /sys/fs/cgroup with their own scripts or runbooks. If you see these values through a monitoring agent such as cAdvisor, the agent handles v2 itself, and that case is outside the scope here. An explanation of v2 itself and the migration steps are on the Kubernetes blog.

Three reads that change

Throttle time

throttled_time, missing from the cpu.stat above, became throttled_usec. It is the total time the cgroup was stopped for hitting its CPU limit, and its unit changed from nanoseconds to microseconds.² If you change only the key name and keep the /1e9 that converts nanoseconds to seconds, you get a value 1/1000 of the real one.

$ kubectl exec <pod> -- awk '$1=="throttled_usec"{printf "%.6f\n", $2/1e9}' /sys/fs/cgroup/cpu.stat
0.001593

With /1e6 instead, it comes out to 1.592698 seconds.

nr_throttled and nr_periods, which give the fraction of periods that hit the limit, keep the same names and meanings as in v1, so the division works unchanged.

Memory usage

In v1, memory usage was calculated with this formula.

memory.usage_in_bytes − total_inactive_file (a line in memory.stat)

When a process reads or writes a file, the kernel keeps the contents in memory as page cache, and the cgroup's usage includes it. The part of the page cache that hasn't been used recently is inactive_file, which the kernel reclaims as soon as memory runs short. Usage minus inactive_file is the part that can't be released right away: the working set. The kubelet exposes this value as container_memory_working_set_bytes, and the memory column of kubectl top pod shows the same value.³

In v2, the formula becomes:⁴

memory.current − inactive_file (a line in memory.stat)

v2's memory.stat has no lines starting with total_. The aggregate lines from v1 appear in v2 under their names with total_ removed.

$ kubectl exec <pod> -- cat /sys/fs/cgroup/memory.current
108158976
$ kubectl exec <pod> -- grep ^inactive_file /sys/fs/cgroup/memory.stat
inactive_file 104906752

The difference, about 3 MiB, is the working set.

The corrected value matches container_memory_working_set_bytes from the kubelet.

$ kubectl get --raw "/api/v1/nodes/<node>/proxy/metrics/resource" | grep ^container_memory_working_set_bytes

OOM count

In v1, a container's OOM count came from oom_kill in memory.oom_control.⁵ In v2, the same oom_kill is in memory.events, a file that counts the memory events in that cgroup by type. However, oom_kill in the container's memory.events reads 0 even after the container has restarted due to OOM. The count is kept in the memory.events of the Pod's slice, one level above the container.

The rest of this section covers why it reads 0, how to find the Pod's slice, and how to read the slice's memory.events, in that order.

Why the container's oom_kill reads 0

In a Pod that restarted due to OOM, the container's memory.events reads like this.

$ kubectl exec <pod> -- grep oom /sys/fs/cgroup/memory.events
oom 0
oom_kill 0
oom_group_kill 0

On v2 nodes, the kubelet sets the container's memory.oom.group to 1.⁶ When OOM occurs in this cgroup, the kernel kills every process in the container, the container is recreated along with its cgroup, and the counters start again from 0.

To check whether your nodes behave this way, see whether the container's /sys/fs/cgroup/memory.oom.group is 1. If the kubelet's singleProcessOOMKill is true, it stays 0.

Finding the Pod's slice

The Pod's slice is the cgroup that groups all of the Pod's containers, and its memory.events adds up the counts of the containers under it. The slice survives container restarts, so the count survives too.

/sys/fs/cgroup/
└── kubepods.slice/
    └── kubepods-burstable.slice/
        └── kubepods-burstable-pod<uid>.slice/         ← Pod's slice
            └── cri-containerd-<container-id>.scope/   ← container's scope

To locate the slice, search for the scope by container ID. The container ID is the part of the Pod's containerID after containerd://. When you enter the node with kubectl debug node, the node's files appear under /host.

$ kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].containerID}'
containerd://<container-id>
$ kubectl debug node/<node> -it --image=busybox
# find /host/sys/fs/cgroup -type d -name 'cri-containerd-<container-id>.scope'

The parent of the directory it finds is the Pod's slice. cri-containerd- is the prefix used when the runtime is containerd. If you can log in to the node as root, drop /host from the path.

Reading the slice's memory.events

In the slice's memory.events, oom_group_kill is the OOM count, and it matches restartCount. oom_group_kill is the number of times all processes in the cgroup were killed together because of memory.oom.group.⁷ For a Pod with a restartCount of 3, it looks like this.

# grep oom <slice>/memory.events
oom 3
oom_kill 9
oom_group_kill 3

oom_kill goes up each time the kernel kills a process, so a single OOM raises it by more than one. In this Pod, three OOMs brought it to 9.

I checked these on a Kubernetes 1.37.0 cluster started with kind v0.33.0 on a GitHub Actions runner (ubuntu-24.04). The results are here, and the code is in show-your-work. ↩
The cpu.stat entry in the kernel's cgroup v2 documentation. ↩
The memory entry in the Kubernetes Resource metrics pipeline documentation. ↩
cAdvisor's handler.go subtracts total_inactive_file for v1 and inactive_file for v2. The memory-available-cgroupv2.sh example in the Kubernetes documentation writes the v2 formula the same way. ↩
The memory.oom_control entry in the kernel's cgroup v1 memory documentation. oom_kill is the number of processes in that cgroup killed by OOM. ↩
In the kubelet's kuberuntime_container_linux.go, memory.oom.group is set to 1 on v2 nodes unless singleProcessOOMKill is true. This was added in kubernetes/kubernetes#117793. ↩
The memory.events entry in the kernel's cgroup v2 documentation. ↩

Docker exit code 137: telling the causes apart from the logs

Kazu — Fri, 16 Oct 2026 13:02:00 +0000

A container has stopped with exit code 137. docker ps -a shows Exited (137), and docker inspect shows "ExitCode": 137.

This value does not tell you whether the container hit its memory limit, the host as a whole ran out of memory, or Docker or some other process stopped it.

I reproduced this on a test VM, changed how the container was stopped, and checked what each case left in the kernel log and the Docker daemon log.¹

Which log points to which cause

The OOM killer is the mechanism the kernel uses when memory runs short: it picks a process and kills it. For each way the container stopped, the kernel log and the Docker daemon log contained the following.²

What happened	Kernel log	Docker daemon log
The container hit its memory limit and the OOM killer killed it	`Memory cgroup out of memory: Killed process`	none
The host as a whole ran out of memory and the OOM killer killed it	`Out of memory: Killed process`	none
The container did not exit on `docker stop`, so Docker killed it	none	`using the force`
Someone ran `docker kill`	none	`hasBeenManuallyStopped=true` (if the container has a restart policy; no `using the force`)
Cannot be told from the logs	none	none

The last row covers a process on the host running kill -9, the app returning 137 itself, and a container without a restart policy being stopped with docker kill.

For a container with a restart policy, docker stop also logs hasBeenManuallyStopped=true. If a using the force line comes before it, it was docker stop. If not, it was docker kill.

When a container without a restart policy is stopped with docker kill, the Docker daemon log has no line that identifies it as docker kill. The only trace is the kill event in docker events, and the Docker daemon keeps only the most recent 256 events in memory, counted across all containers on the host.³

Read the kernel log with journalctl -k. A line starting with oom-kill: appears just before the Killed process line. The task_memcg= value on that line contains the container ID, which tells you which container it was.⁴ When the host as a whole runs out of memory, the container that gets killed is not necessarily the one that used up the memory. In the test, the OOM killer killed the app container, which held more memory than the container whose memory kept growing.

journalctl -k | grep -E 'Killed process|oom-kill:'

Read the Docker daemon log with journalctl -u docker. The string after container= is the container ID.

journalctl -u docker | grep -E 'using the force|hasBeenManuallyStopped=true'

Using the table in production

In production, there are situations where the values in docker inspect cannot be relied on. The logs in the table still tell the causes apart in those situations.

The container has restarted

If RestartCount in docker inspect is 1 or more, the container has restarted. When a container with a restart policy restarts, ExitCode in docker inspect goes back to 0 and OOMKilled back to false, and nothing there shows that it stopped with 137.

After a restart, FinishedAt in docker inspect still holds the time of the previous stop. If the kernel log has a Killed process line at about the same time, the previous stop was a kill by the OOM killer. FinishedAt is in UTC, so use journalctl -k --utc to line up the timestamps.

`OOMKilled` is false

State in docker inspect has a value called OOMKilled, which Docker sets to true when a process in the container is killed by the OOM killer. But it sometimes stays false even though the OOM killer did the kill. Decide whether the OOM killer killed it from the Killed process line in the kernel log (journalctl -k).

What exit code 137 indicates

PID 1 is the first process started in the container, the one given in ENTRYPOINT or CMD. When PID 1 exits, the container exits, and the exit code is determined by how PID 1 ended.

When PID 1 is ended by a signal, the exit code is 128 plus the signal number. SIGKILL is 9, so the code is 137.⁵

The exit code can be 137 even when PID 1 itself did not receive SIGKILL. If an init is started ahead of the app and runs the app as its child, as with docker run --init, then when the app is ended by SIGKILL, init exits with 137.⁶ With init in place, the logs for a kill at the container's memory limit and for kill -9 from a host process looked the same as without init.

SIGKILL is sent to a container's processes by one of the following:

The kernel's OOM killer
Docker (when the container does not exit on docker stop, and on docker kill)
Another process on the host

Every kill in the table is this SIGKILL. The exit code does not tell you who sent it, so use the logs in the table to tell the senders apart.

What was not checked

I did not check mechanisms outside the kernel that watch for memory shortage and kill processes (systemd-oomd and earlyoom).

Appendix: each way of stopping and what it left

`--memory 128m`, app allocates 256MiB

Kernel log: Memory cgroup out of memory
Docker daemon log: none
docker events: oom, die
OOMKilled: true (false in 0 to 1 of 3 tries per full test run)

Another container with no limit uses up the host's memory

Kernel log: Out of memory
Docker daemon log: none
docker events: oom, die (order varies between tries)
OOMKilled: true (false in 0 to 2 of 10 tries per full test run)

`docker stop -t 2`

Kernel log: none
Docker daemon log: using the force
docker events: kill (15), kill (9), stop, die
OOMKilled: false

`docker stop -t 2`, with a restart policy

Kernel log: none
Docker daemon log: using the force, hasBeenManuallyStopped=true
docker events: kill (15), kill (9), stop, die
OOMKilled: false (does not restart)

`docker kill`

Kernel log: none
Docker daemon log: none
docker events: kill (9), die
OOMKilled: false

`docker kill`, with a restart policy

Kernel log: none
Docker daemon log: hasBeenManuallyStopped=true
docker events: kill (9), die
OOMKilled: false (does not restart)

`kill -9` from the host to the app

Kernel log: none
Docker daemon log: none
docker events: die
OOMKilled: false

The app returns 137 itself

Kernel log: none
Docker daemon log: none
docker events: die
OOMKilled: false

`--memory 128m`, with a restart policy

Kernel log: Memory cgroup out of memory
Docker daemon log: none
docker events: oom, die, start
OOMKilled: false after the restart (ExitCode is 0)

I tested on a VM started with KVM inside a GitHub Actions runner (Ubuntu 24.04, kernel 6.8, 512MiB memory, no swap, cgroup v2), using Docker 29.1.3 from Ubuntu's docker.io package (cgroup driver: systemd). The scripts and results are in show-your-work and in the GitHub Actions run. ↩
The kernel log strings come from kernel 6.8: the call to oom_kill_process() and the line that prints Killed process. The Docker daemon log lines come from Docker 29.1.3: the line that logs using the force and the lines that log the warning containing hasBeenManuallyStopped=true. ↩
eventsLimit in the Docker 29.1.3 source. ↩
Kernel 6.8: where the oom-kill: line is printed and where task_memcg= is printed. ↩
runc's ExitStatus(). ↩
The init that --init starts is tini, which returns 128 plus the signal number when its child is ended by a signal (tini v0.19.0). ↩

nginx 504 Gateway Timeout with zero 429s from limit_req

Kazu — Wed, 14 Oct 2026 13:03:00 +0000

nginx uses limit_req to cap the number of requests per second for each IP address. Requests over the limit get a 429 (Too Many Requests) through limit_req_status 429.

At some point, 504s (Gateway Timeout) start to increase. A 504 is the status nginx returns when it forwards a request to the application server behind it (an upstream, in nginx terms; called the upstream below), waits for the time set in proxy_read_timeout without receiving a response, and gives up. In the access log, $request_time (the time from when nginx receives the request until it finishes sending the response) on every 504 line equals the proxy_read_timeout value. Each time nginx gives up, error.log records the following string.

upstream timed out (110: Connection timed out) while reading response header from upstream

There are zero 429s, and the number of requests per IP address is the same as usual.

The upstream is not down, and its access log keeps flowing.

I reproduced this state and measured the conditions under which the upstream falls behind while traffic stays within the limit_req limit, how to tell the causes apart, and a remedy that needs only nginx configuration.

Impact

This article covers upstreams built this way: the upstream has a cap on how many requests it can process at the same time, and connections that arrive beyond that cap wait in an OS queue. Even after nginx closes the connection, the upstream finishes processing the request. Handlers that detect the disconnect and abort processing are out of scope.

With this design, if the upstream cannot keep up with even a fraction of the incoming requests and that continues, no response succeeds. Every user waits until the timeout and then receives a 504.

The upstream keeps processing at full capacity, and that capacity goes into building responses nobody will receive. Almost every request it finishes is one for which nginx has already returned a 504.

When 504s increase, nothing is recorded on the rate limiting side. 429s stay at zero, which leaves nothing to base a decision on: whether to lower the limit_req limit, and if so, by how much.

What limit_req looks at

limit_req records one excess value per key value. The key is the value given to limit_req_zone; here it is the IP address. Two settings determine the limit. rate is the average speed allowed for requests from one IP address. Requests that arrive faster than rate are recorded as excess (how many requests' worth above rate have arrived), and the excess decreases over time at the speed of rate. burst is how much excess is allowed. When the excess exceeds burst, that request gets a 429.¹

Heavy and light requests alike, however many seconds they take on the upstream, each add the same one request's worth to the excess.

rate and burst are fixed when the configuration is loaded and do not change while nginx is running. Whether to return a 429 is decided before the request is forwarded to the upstream. When upstream responses slow down, or the number of requests in progress on the upstream grows, limit_req does not see the change.

In a zone keyed by IP address, rate and burst apply to requests from one IP address. If the number of IP addresses sending requests triples while requests per IP address stay the same, every IP address stays within the limit and the requests reaching the upstream triple.

Conditions under which the upstream falls behind within the limit

Setup

I ran nginx 1.30.5 and the upstream.

The upstream is a small HTTP server written in Go, built the way described in the Impact section. It processes up to 20 requests at a time, and connections beyond that wait in the queue where the OS holds connections until it hands them to the server (the accept queue). Processing takes 50ms per request on the light endpoint and 300ms on the heavy endpoint.

The upstream closes the connection after each response. In nginx 1.30.5, keepalive to the upstream is enabled by default, but because the upstream closes the connection, connections are not reused and one request is one connection.² The handler processes each request to the end without watching for nginx disconnecting. In Go's net/http, the request context is canceled when the client disconnects, and this upstream does not check it.³

Load came from the load testing tool k6: 200 requests per second at even intervals, all to the light endpoint. I prepared 10 key values and assigned 20 requests per second to each.

limit_req was set to rate=50r/s and burst=20 nodelay. nodelay forwards the excess requests to the upstream immediately instead of holding them back, as long as the excess stays within burst.⁴ Each key value receives 20 requests per second, fewer than the 50 per second of rate, so every key value stays within the limit.

Because load came from a single machine, I used a request header value as the key in place of the IP address. limit_req treats the key as a string, so the behavior is the same whether the key is an IP address or a header value.

proxy_read_timeout was shortened to 1 second to keep the test short. The mechanism is the same with the default of 60 seconds.

At this starting point, about 10 requests are in progress on the upstream at a time (200 per second × 0.05 seconds), and the upstream can handle 400 per second (20 ÷ 0.05 seconds).

Load ran for 25 seconds, and one item was changed at the 8-second mark. I also ran a pass with nothing changed as the baseline.⁵

Results

In every condition, 429s stayed at zero, and the success rate was 0% from 6 seconds after the change onward. This is the same state as the symptom described at the start.

Condition	Changed at 8 seconds	Success rate
Baseline	Nothing	100%
More heavy requests	Share of heavy requests 0% → 40%	0%
Longer processing time	Processing time of light requests 50ms → 120ms	0%
Fewer concurrent slots	Requests the upstream can process at the same time 20 → 8	0%
More key values	Number of key values 10 → 30 (600 per second in total)	0%

The success rate is the fraction of requests k6 sent between 14 and 25 seconds (from 6 seconds after the change) that got a 200.

Reading the conditions

More heavy requests corresponds to heavy requests mixing in while the number of incoming requests stays the same. The number of requests the upstream can handle drops, and limit_req adds one request's worth to the excess regardless of weight.

Longer processing time corresponds to the upstream, or the DB or external API behind it, slowing down. The upstream can no longer keep up with the same number of requests, and limit_req does not see the change.

Fewer concurrent slots corresponds to a reduction in upstream nodes or workers. The number of requests the upstream can handle drops, while rate and burst stay at their configured values.

More key values corresponds to more IP addresses sending requests. Every key value stays within the limit while the total number of requests reaching the upstream exceeds its capacity.

Why no 200s come back

Mechanism

In each of the four conditions in the previous section, after the change, the number of requests the upstream can handle per second falls below the number arriving. The upstream processes at most 20 at a time, so the connections it cannot handle line up in the accept queue. While the amount reaching the upstream exceeds what it can handle, the number of connections in the accept queue keeps growing, and requests that arrive later wait longer.

sequenceDiagram
    participant C as Client
    participant N as nginx
    participant Q as accept queue
    participant U as upstream

    C->>N: Request
    N->>Q: Connects and sends the request
    Note over Q,U: Waits while the upstream processes<br/>the connections ahead of this one, in order
    Note over N: proxy_read_timeout elapses
    N->>C: 504
    N->>Q: Closes the connection
    Q->>U: Hands over the connection once it reaches the head
    Note over U: Processes it without noticing<br/>the connection was closed
    U->>N: Response nobody receives

When the time waited in the accept queue plus the processing time on the upstream exceeds proxy_read_timeout, nginx gives up on that request, returns a 504, and closes the connection to the upstream. This is the symptom seen at the start, where $request_time on 504 lines equals the proxy_read_timeout value.

The upstream does not notice that nginx closed the connection. When the connection reaches the head of the accept queue, the upstream accepts it, processes it, and writes a response nobody receives. This is why records of finished processing keep appearing in the upstream access log.

The accept queue is processed from the front and keeps growing in the meantime, so everything that reaches the head is a request nginx has already given up on. All of the upstream's capacity goes to these, and by the time a request that is still waiting gets its turn, it too is past proxy_read_timeout.

In the logs

For the Longer processing time condition (light request processing time 50ms → 120ms, changed at 8 seconds), I counted the requests in each second from the start of the load, using the upstream access log and the nginx access log.

Second	Requests the upstream finished	200s returned by nginx	504s returned by nginx
9	167	167	0
10	167	167	0
11	162	162	0
12	166	166	0
13	168	52	139
14	164	0	200
15	165	0	200
16	167	0	200

Through second 12, the number of requests the upstream finished matches the number of 200s nginx returned. The difference from the 200 per second arriving accumulates in the accept queue.

Requests that arrive after 12.3 seconds spend more than the 1-second proxy_read_timeout on waiting plus processing, so 504s start at 13.3 seconds.

From second 14 on, the upstream keeps processing at the same speed, yet only 504s come back. nginx returns a 504 one second after a request arrives, so the count equals the 200 per second arriving.

Checking your own logs

When 504s increase and 429s are zero, first check whether the wait in the accept queue is growing. If it is, the upstream has fallen behind within the limit_req limit. Then identify which cause applies.

Check whether the accept queue wait is growing

The accept queue wait shows up as the difference between the upstream response time as seen by nginx and the processing time in the upstream access log. If this difference grows second by second, waits are building up in the accept queue. The difference starts growing before the 504s appear. On 504 lines $upstream_header_time is -, so look at the period before the 504s.

The upstream response time as seen by nginx is $upstream_header_time, which can be recorded in the access log. It is the time from when nginx starts connecting to the upstream until it receives the response header.⁶ It includes both the accept queue wait and the processing time on the upstream. The processing time in the upstream access log is the time from when the upstream accepts the connection until it returns the response, and does not include the accept queue wait.

To get the difference, take the per-second, per-endpoint median of $upstream_header_time and of the upstream processing time, then subtract one from the other. Record the nginx access log in the following format.

log_format rl '$msec $status $uri $remote_addr $upstream_addr $upstream_header_time';

Identifying the cause

Check in this order.

In the upstream access log, look at processing time per endpoint. If it has grown, the upstream or the DB or external API behind it has slowed down (Longer processing time). However, an increase in heavy requests can put pressure on shared resources such as the DB and make even light endpoints slower. Even if processing time has grown, still check the share in the next step. In the test, the heavy processing is a sleep that does not affect other requests, so this interference was not verified.
If it has not grown, look at each endpoint's ($uri) share of the total count in the nginx access log. If the share of endpoints with long processing time has increased, heavy requests have increased (More heavy requests).
If the shares have not changed, look at the total requests per second in the nginx access log. If it has increased, more requests are arriving. Break it down by how many distinct key values there are (the number of distinct $remote_addr values) and by requests per second per key value. If the number of distinct key values has increased, more IP addresses are sending requests (More key values). The case where requests per key value increase within rate was not tested, but the total reaching the upstream increases in the same way.
If none of these apply, look at the number of upstream nodes or workers. If it is lower than usual, the number of requests that can be processed at the same time has dropped (Fewer concurrent slots).

Remedies

First stop the damage with max_conns, then fix the cause.

Cap concurrent requests with max_conns

Put max_conns on the upstream server line. Set it to the number of requests the upstream can process at the same time. zone is needed to share the concurrent connection count across nginx workers.⁷ If you run several nginx instances side by side, divide this value by the number of instances to get the per-instance value.

upstream app {
    zone app 64k;
    server app:8081 max_conns=20;
}

Connections nginx sends to the upstream at the same time no longer exceed max_conns. Requests over that are not sent to the upstream, and nginx returns a 502 (Bad Gateway) on the spot. nginx OSS has no mechanism to make them wait until a slot frees up. The upstream processes only requests nginx is still waiting on, and the accept queue does not grow.

In this test the upstream closes the connection, so keepalive connections are not reused. The nginx documentation states that with keepalive to the upstream, multiple workers, and zone, the total of active and idle connections can exceed max_conns.⁸ From nginx 1.29.7 on, keepalive is enabled by default, so if the upstream keeps connections open, you get this setup even without a keepalive directive. If the upstream limits concurrent processing by the number of connections, idle connections also take up slots. The case where keepalive connections are reused was not tested.

For each of the four conditions, I compared no remedy against max_conns=20. p99 is the value at the 99% position when the response times of successful responses are sorted from shortest to longest. p99 in the run with nothing changed was 51.6ms.

Condition	Success rate (no remedy)	Success rate (max_conns=20)	p99 of successful responses (max_conns=20)
More heavy requests	0%	66%	301ms
Longer processing time	0%	80%	122ms
Fewer concurrent slots	0%	79%	132ms
More key values	0%	64%	52ms

There were zero 504s in every condition. The success rate is roughly equal to what the upstream can handle divided by what arrives. Successful responses come back without waiting, and p99 matches the upstream processing time. In Fewer concurrent slots it was 132ms because the amount by which max_conns=20 exceeds the upstream's 8 (up to 12 requests) lines up in the accept queue, and that wait is added.

Even if the max_conns value is larger than the number of requests the upstream can actually process at the same time, the accept queue does not keep growing. In the Fewer concurrent slots run, max_conns stayed at 20 while the upstream's concurrent capacity dropped to 8. There were zero 504s, and the accept queue stopped at a maximum of 12. A connection is established when the OS accepts it, so the connections nginx counts include those waiting in the accept queue. The connections in the accept queue never exceed max_conns minus the upstream's concurrent capacity.

Users beyond what can be handled receive a 502 without waiting. You cannot choose which requests get the 502. This 502 is recorded in error.log with the following string.

no live upstreams while connecting to upstream

In the access log, $upstream_addr, which normally holds the upstream's IP address and port, becomes the upstream name (app here).⁹ 502s returned without sending the request to the upstream can be counted by this value. The error.log message is the same as for a 502 when all upstream servers are down, but with a 502 from max_conns, 200s keep coming back in the same time window.

Options ruled out

Shortening proxy_read_timeout is ruled out. Changing it from 1 second to 500ms still gave a 0% success rate. nginx gives up sooner, and the accept queue grows at the same speed.

Stacking a second limit_req to cap the total of requests from all IP addresses is also ruled out. A zone keyed by a fixed string (limit_req_zone all zone=fixed:1m rate=160r/s;) is stacked on top of the IP address zone. rate=160r/s is set to match the 167 per second the upstream can handle under Longer processing time (20 ÷ 0.12 seconds). With this setting, the success rate under Longer processing time was 80%, and the accept queue did not grow. However, about 19% got a 429 even in normal operation, when the upstream had spare capacity. rate has to be fixed to match the number of requests per second the upstream can handle, and that number changes with processing time. max_conns caps the number of requests in progress at the same time, so when processing time grows, it automatically lets fewer requests through, and in normal operation it turns nothing into a 502.

Fixing the cause

max_conns only stops the damage. What the upstream can handle does not recover. Where to act for each cause:

For Longer processing time, fix the DB or external API that slowed down
For More heavy requests, move heavy processing to a separate upstream or to asynchronous processing
For Fewer concurrent slots, restore the lost nodes or workers
For More key values, add upstream capacity to match the increase

Summary

What limit_req caps is the number of requests arriving per IP address. What the upstream can handle is determined by processing time and the number of requests it can process at the same time, and changes in these do not reach limit_req.

What cannot be handled accumulates in the accept queue and turns into 504s once waiting plus processing exceeds proxy_read_timeout. The upstream keeps processing requests nginx has already given up on, so no response succeeds.

Protect the upstream with a cap on how many requests nginx sends to it at the same time (max_conns). Regardless of cause, only what cannot be handled becomes a 502, and the rest comes back at the usual speed.

This applies to upstreams that make connections arriving beyond their concurrent capacity wait in the accept queue, and whose handlers do not abort processing when nginx closes the connection.

In the nginx 1.30.5 source, ngx_http_limit_req_lookup() makes this decision. ↩
Keepalive to the upstream has been enabled by default since nginx 1.29.7 (CHANGES; nginx documentation for keepalive). ↩
Go documentation for Request.Context. ↩
nginx documentation for limit_req. ↩
I ran it on a GitHub Actions runner (ubuntu-24.04). The results are here, and the code is in show-your-work. ↩
The nginx documentation for $upstream_header_time does not say where the measurement starts. In the 1.30.5 source, it is computed from u->start_time, which is recorded when nginx starts connecting to the upstream. ↩
nginx documentation for max_conns. Without zone, the limit applies per worker. ↩
nginx documentation for max_conns. ↩
nginx documentation for $upstream_addr. When no server can be selected, it holds the upstream name. ↩

High p99 latency in Node.js at low CPU usage: a blocked event loop

Kazu — Mon, 12 Oct 2026 13:04:00 +0000

The latency of a Node.js service suddenly gets worse. p99 (the 99th percentile of response time: when 1000 requests are sorted from fastest to slowest, the value of the 990th) jumps. CPU looks low when you check overall usage. Memory is normal too. Nothing points to a cause. Endpoints that respond immediately, like a health check, are slow in the same way.

Node.js runs its event loop on a single thread. When one piece of work blocks, every other request waits in the queue. This congestion rarely shows up on an overall CPU graph.

I reproduced event loop starvation on GitHub Actions (ubuntu-24.04, 4 vCPU)¹ and show, with the code I ran, which metric catches the degradation that an overall CPU graph tends to miss. In the same environment I also measured how much an overload guard (load shedding: returning 503 immediately once event loop delay exceeds a threshold) improves p99, and what it does not improve.

How Node.js handles requests

In an architecture that assigns a thread to each request, more concurrent connections mean more threads, and the cost of context switches (the overhead incurred each time the CPU switches from one thread to another) grows. Node.js avoids this by handling every request on an event loop that runs on one thread.

The event loop works through a to-do list (a queue) in order on a single thread. Each incoming request is added to the end of the queue, and the event loop takes items from the front and runs them. Work that has to wait for I/O (reading a file, network communication) leaves the queue while it waits and comes back when the I/O completes. In the meantime the loop can pick up the next item, so it can serve many connections without adding threads.

If one piece of work keeps hold of the thread, though, the rest of the queue has to wait until it finishes. This is event loop blocking. Reading a file with fs.readFileSync inside a request handler, running a shell command with execSync, and calling JSON.parse on a response several MB in size are all examples.²

How far this blocking backs up the queue is called lag. setTimeout(fn, 100) is a reservation to call fn after 100ms, but while the event loop is blocked, the call to fn is held back too. If fn should have been called after 100ms and is called after 450ms, the lag is 350ms.

Why overall CPU looks flat

A CPU graph shows usage for the whole system. Because Node.js runs on a single thread, a blocking operation that consumes CPU uses only one core's worth. On a server with 4 vCPU (virtual CPUs, the unit in which cloud environments divide physical cores and allocate them), one core at 100% shows up as only 25% overall.

I checked this by measurement, with two endpoints. /sync-cpu occupies one core with a busy loop (a loop that keeps hold of the CPU until its completion condition is met) that repeats JSON.parse until a deadline, blocking the event loop for a specified time. /light returns a response immediately with no computation. Ten concurrent connections sent a mix of requests, 1/3 to /sync-cpu and 2/3 to /light. The blocking time (sync_ms) was set to 50ms, 100ms, 150ms, and 200ms, and each setting was measured for 30 seconds. The columns to watch in the table are cpu_pct (CPU usage) and lag_p99 (p99 of event loop lag).

sync_ms	/sync-cpu p99	/light p99	cpu_pct	lag_p99
0ms	4ms	4ms	26.7%	23ms
50ms	407ms	353ms	25.5%	306ms
100ms	712ms	1524ms	25.9%	612ms
150ms	1214ms	1610ms	25.6%	764ms
200ms	1222ms	2407ms	25.7%	1224ms

cpu_pct stays almost fixed, between 25.5% and 26.7%. Quadrupling the blocking time from 50ms to 200ms barely moves CPU. lag_p99, on the other hand, is the event loop lag of the whole process, measured as a single value regardless of endpoint. The 23ms at 0ms is a floor that comes from the resolution: 20 setting; it is the baseline with no blocking. Lag rises with the blocking time: 306ms at 50ms, 612ms at 100ms, 1224ms at 200ms. CPU does not change, and only lag reflects the blocking.

At 200ms, the p99 of /light (2407ms) is higher than that of /sync-cpu (1222ms). /light has no blocking work of its own, but it sometimes waits a long time in the queue, and the larger the blocking time, the wider the spread of its p99.

Even when the overall CPU graph is flat, requests keep waiting in the queue if the event loop is congested. Looking only at overall CPU misses the congestion.

Spillover to unrelated endpoints

A Node.js process shares one event loop across all of its endpoints. While /sync-cpu is blocking the event loop, requests to /light wait in the queue too.

At 0ms, the p99 of /light is 4ms. At 50ms it jumps to 353ms. Nothing about /light itself changed. /sync-cpu occupied the event loop, and as a result the unrelated /light was kept waiting in the queue as well. The longer the blocking time, the worse the latency of /light gets.

Observing lag

Measuring event loop lag

Since Node.js v11.10.0, monitorEventLoopDelay() in node:perf_hooks gives you event loop lag.³

With resolution: 20, Node.js sets a timer that fires every 20ms, and each time it fires, it records the actual interval since the previous firing.⁴ If the event loop is blocked by other work, this timer cannot get in either. With a 300ms block, the timer does not fire during the block, and when it next fires, a single interval of a little over 300ms is recorded. With no blocking, intervals of around 20ms are recorded. These accumulate in a histogram (the distribution of how often each recorded interval occurs), and percentile(99) extracts the p99.

const { monitorEventLoopDelay } = require('node:perf_hooks');
const h = monitorEventLoopDelay({ resolution: 20 });
h.enable();

setInterval(() => {
  console.log('lag p99:', h.percentile(99) / 1e6, 'ms');
  h.reset();
}, 5000);

h.percentile(99) returns the p99 of the accumulated samples in nanoseconds, and / 1e6 converts it to milliseconds. h.reset() clears the samples and starts measurement for the next 5 seconds. Export this p99 as a metric and you can catch abnormal lag even when the CPU graph is flat.⁵

Per-core CPU

The blocking work in this experiment is a busy loop that repeats JSON.parse, and it keeps one core fully used. Overall CPU looks like around 25%, but a per-thread view with top -H or htop shows the Node.js main thread pinned at 100%, which confirms why the overall figure looked low.

When you run into the same symptom (low overall CPU, high p99), per-core CPU shows which thread the offending work is saturating.

Effect and limits of an overload guard

I implemented middleware that uses the measured lag to detect overload and return 503 (Service Unavailable, the HTTP status code that indicates the server temporarily cannot handle the request).

At startup it repeatedly sets a setTimeout with a 100ms interval. Each time the callback runs, it writes the difference between the planned 100ms and the actual elapsed ms into lagMs. If the event loop is blocked, this timer cannot get in either, so lagMs rises.

monitorEventLoopDelay can run at a 20ms interval without trouble because Node.js starts a libuv timer from the C++ side and does not call into JS for each measurement.⁶ A hand-written setTimeout loop, by contrast, creates a timer and runs a callback on the JS thread every time. The shorter the interval, the more of that overhead lands on the event loop itself, so the middleware's measurement uses 100ms.

let lagMs = 0
;(function measure() {
  const start = Date.now()
  setTimeout(() => {
    lagMs = Math.max(0, Date.now() - start - 100)
    measure()
  }, 100).unref()
})()

On each request, the middleware compares lagMs with the threshold. If lag exceeds the threshold (150ms), it returns 503 immediately with Retry-After: 1. If lag is at or below the threshold, it passes the request to the next handler.

if (lagMs > thresholdMs) {
  res.setHeader('Retry-After', '1')
  return res.status(503).json({ error: 'overloaded', lag_ms: lagMs })
}
next()

lagMs is a single global variable in the process. /sync-cpu and /light both read the same value. Even when /sync-cpu is the one pushing lag up, the request that gets the 503 once the threshold is exceeded is the next one to arrive: whichever request enters the queue first, regardless of endpoint.

I applied the same load with this guard in place (threshold 150ms).

sync_ms	/sync-cpu p99	/sync-cpu 503 rate (%)	/light p99	/light 503 rate (%)	lag_p99
0ms	3ms	0.0%	3ms	0.0%	21.7ms
50ms	208ms	95.6%	203ms	95.4%	305.7ms
100ms	212ms	98.7%	202ms	98.7%	713.0ms
150ms	307ms	98.9%	162ms	99.0%	1072.2ms
200ms	401ms	99.0%	217ms	99.0%	1653.6ms

p99 here is computed over all requests, 503s included. Since most of them are immediate 503s, this p99 is not a measure of how slow the requests that got through were. At sync=100ms, the /sync-cpu p99 goes from 712ms to 212ms, and the /light p99 from 1524ms to 202ms.

Look at the 503 rate. With a 150ms threshold, at sync=100ms, 98.7% of requests to both endpoints get a 503. Of the 18793 requests that reached /sync-cpu, only 239 got through. As long as the blocking work continues, lag piles up far beyond the threshold, and the service stays stuck rejecting most requests. The overload guard alone does not make a working service.

With the guard in place, lag_p99 did not go down, and at sync_ms of 100ms or more it went up.

sync_ms	lag_p99 (no guard)	lag_p99 (with guard, threshold 150ms)
50ms	305.9ms	305.7ms
100ms	612.4ms	713.0ms
150ms	763.9ms	1072.2ms
200ms	1223.7ms	1653.6ms

The guard turns new requests away with 503. It cannot stop blocking work that is already running on the event loop. The time /sync-cpu spends blocking the event loop continues regardless of the guard. The middleware's check and the setTimeout callback for measuring lag also run on the event loop, so while /sync-cpu is blocking, they cannot get in either.

Once lag exceeds the threshold, the guard returns 503 to every request regardless of endpoint. Even when /sync-cpu is what pushes lag up, /light gets rejected the same way.

The guard improves what requests that get through experience, but it does not resolve event loop starvation. Lag staying high shows this.

Finding and fixing the blocking work

Wherever the 503 threshold is set, lag does not go down as long as the blocking work keeps running. The overload guard limits inflow, and it leaves the work that congests the event loop untouched.

Finding it

The fundamental fix is to identify the work that is blocking the event loop and fix it. Once you have confirmed the symptom in production, bring it to a staging or development environment where you can reproduce the same load pattern. Start the process with node --prof, apply similar load, and after stopping it, process the sampling log with node --prof-process. This identifies the functions that occupy CPU time.⁷

Here is a profile taken in this test environment. It is the result of loading /sync-cpu?ms=200 with 10 concurrent connections for 20 seconds.

Statistical profiling result (27870 ticks, 2 unaccounted, 0 excluded).

[Bottom up (heavy) profile]:
   ticks parent  name
  25652   92.0%  node
  25375   98.9%    node
  23368   92.1%      JS: *blockEventLoop  routes/sync-cpu.js:10
  23368  100.0%        JS: ^<anonymous>   routes/sync-cpu.js:17

84% of all CPU ticks (the number of times the profiler, sampling at a fixed interval, observed where the CPU was executing) are concentrated in blockEventLoop. Following the callers also shows that it is called from an Express route handler. Confirming that blocking exists with lag, then narrowing down the responsible function with --prof-process, is the efficient order.

Fixing it

How to fix it depends on the kind of work. For synchronous I/O like fs.readFileSync or execSync, replacing it with the asynchronous fs.readFile or exec frees the event loop.

For work that uses CPU for a long time, the design of doing it inside a request handler on the application server needs to be reconsidered. Heavy work such as image processing or video conversion does not need to be synchronous with the response. The application server receives the request, puts a job on a queue, and immediately returns "accepted." The work itself lives outside the application server, in a separate worker process, Lambda, Cloud Functions, and so on. Because it is decoupled from the application server's event loop, the application server does not get congested however heavy the work is. For work that has to happen on the request path, such as JSON.parse on MB-sized data, moving it to a separate thread with worker_threads is an option. Before that, reconsider whether the application server should be handling that work at all.⁸

Running many Node.js application servers such as Express or Fastify as 1 vCPU containers, with a load balancer in front distributing requests, is a widely used setup. In this setup, when one container gets congested, the other containers keep accepting requests, so the single-thread constraint is absorbed operationally. The problem of one container's event loop getting congested does not go away, though. Requests assigned to that container stay queued inside it, and the same symptom shown in this article appears. Horizontal scaling solves the throughput problem; it does not resolve latency degradation inside one container.

Check whether the fix worked with lag_p99. Once the blocking work is removed, lag drops sharply.

Summary

The Node.js event loop handles every request on a single thread. When one piece of work blocks, every request in the process stops in the queue, and unrelated endpoints slow down uniformly.

An overall CPU graph does not show this congestion. On a 4 vCPU machine, one congested core shows up as only around 25% overall. Looking at CPU for this symptom will not find the cause.

What to check is lag. Export lag_p99 as a metric with monitorEventLoopDelay(), and you can detect blocking even when the CPU graph is flat. Looking at it alongside per-core CPU also shows which thread is congested.

An overload guard that returns 503 once lag exceeds a threshold improves response time, but lag itself does not go down, because the blocking work keeps running on the event loop. The guard buys time to find the root cause. Treat it as temporary protection until the blocking work is removed.

The results are here, and the code is in show-your-work. The Node.js version is 22.23.3. ↩
The core modules and JSON sections of the Node.js guide "Don't Block the Event Loop". ↩
The description of monitorEventLoopDelay() in the Node.js 22 documentation. ↩
Histogram::RecordDelta() in the Node.js 22.23.3 source. It puts the difference from the previous recording time into the histogram as is. ↩
Each block produces only one sample, so a block that happens just once in a 5-second window does not show up in p99, only in h.max. If heavy work runs only rarely, export h.max as well. A Node.js maintainer acknowledged this bias in nodejs/node#34661 and suggested reading the histogram together with Event Loop Utilization. The issue was closed with the bias left in place. ↩
CreateELDHistogram() and IntervalHistogram::OnStart() in the Node.js 22.23.3 source. The timer callback only calls RecordDelta() and does not enter JS. ↩
The profiling page of the Node.js guide. It also explains how to read the [Bottom up (heavy) profile] section of the --prof-process output. ↩
The Offloading section of the Node.js guide "Don't Block the Event Loop", and worker_threads in the Node.js 22 documentation. ↩

Buttons don't work for some users after a deploy

Kazu — Fri, 09 Oct 2026 13:03:00 +0000

Once a browser has fetched an index.html served without Cache-Control (the header that says how long a response may be cached), it keeps using its own copy without going back to the server. That period is 10% of the time since index.html was last modified.

If you deploy during that period, reopening the site still runs the previous version's page. The files that page loads later are gone from the server, so when a user uses a feature the page hasn't loaded yet, the load fails and clicking the button does nothing. An error tracker (Sentry or similar) shows this line:

TypeError: Failed to fetch dynamically imported module: <chunk URL>

It means a JS file the page tried to load at runtime with import() could not be fetched. A chunk is a separate file the build splits some of the page's features into. It is loaded for the first time when the user clicks a button or similar. The error hits only users who use a feature the previous version hadn't loaded yet. Everyone else keeps running the previous version with no error.

On a site that deploys once a week, the previous version keeps running in users' browsers for up to about 17 hours after the deploy. Browsers that open the site for the first time after the deploy, or that reload, run the new version, so checking internally may not reproduce it. Users running the previous version don't get a fix even after you deploy it.

I reproduced this with a Vite app served by nginx with its default config, and measured how long the previous version runs and what each fix does.

What is happening

For responses without Cache-Control, the HTTP caching spec¹ lets a cache guess how long it may use its copy without going back to the server, and gives 10% of the time since Last-Modified (when the file was last modified) as a typical value. Chromium uses exactly 10% of the difference between the time the response was sent (Date) and Last-Modified.² Below, the difference between Date and Last-Modified at fetch time is called the time since modification.

Files with a hash (a string the build derives from a file's contents and puts in its name) get a new name whenever their contents change. Which version's files get loaded is decided only by the file names written in index.html, whose own name never changes. Without Cache-Control, the browser decides when to fetch that index.html again.

When build artifacts are baked into an image, Last-Modified is the build time. On a site that deploys once a week, a browser that opened the site just before the next deploy uses its own copy of index.html, without going to the server, for about 17 hours, 10% of 7 days.

1           2  3            4         5
|           |  |            |         |

server serves
[=====old=====][============new=============]

browser runs
            [===========old===========][=new]

<- 7 days -><------- 16.8 hours ------>

The top row is the version the server serves. The bottom row is the version that runs when the browser that opened the site at 2 reopens it at each point in time. 1 is when the previous version was built (Last-Modified), 2 is when the user opened the site and fetched index.html (Date), 3 is the deploy, 4 is when the user reopened the site, and 5 is when 10% of the span from 1 to 2 has passed since 2. The span from 1 to 2 is compressed in the figure. If a deploy falls between 2 and 5, reopening the site still runs the previous version. After 5, index.html is fetched again and the new version runs.

Verification

I served a Vite app that loads a chunk when a button is clicked, using nginx with its default config. I opened the previous version, closed the browser, deployed by replacing the container, then relaunched the browser and opened the site again.³

Time since modification	How long the previous version ran
10 min	About 1 min
100 min	About 10 min
1000 min	About 100 min
6 days	About 14 h

Each was about 10% of the time since modification. Reopening the site in a new tab with the browser kept open gave the same result.⁴

In browsers that hadn't clicked the button on the first open, clicking it while the previous version was running produced the error. In browsers that had clicked it, no error appeared.

Reloading while the previous version was running fetched index.html again and switched to the new version.

Checking your own environment

The response headers on index.html tell you whether browsers keep using their own copy of it. The access log tells you whether any users ran the previous version and hit the error.

Response headers

Check the response headers on index.html with this command. Look at the same headers whether you serve from S3, a CDN, or a hosting service, or render the HTML on the server per request.

curl -sI https://app.example.com/ | grep -iE '^(cache-control|expires|last-modified):'

If the response has Last-Modified but neither Cache-Control nor Expires (the header that sets an expiry time), the previous version keeps running after a deploy for 10% of the time since modification.

Access log

From a browser that ran the previous version and hit the error, the new version's server receives only the following line and returns 404 (the status meaning the file doesn't exist). It is a request for the previous version's chunk, and the same browser sends no request for index.html.

"GET /assets/feature-DHyZGV0s.js HTTP/1.1" 404 153 "http://127.0.0.1:18000/assets/index-DB4b__oJ.js"

From a browser that had clicked the button under the previous version, not a single line arrives.

Fixes

There are three fixes: add no-cache to index.html, add max-age to it, or reload when loading fails.

Add no-cache to index.html

With Cache-Control: no-cache on index.html, the browser asks the server whether index.html has changed every time it opens the page.⁵ If it hasn't, the server returns 304 (the status meaning not modified) and sends no body.

In nginx, add this:

location = /index.html {
    root   /usr/share/nginx/html;
    add_header Cache-Control "no-cache";
}

The new version runs every time the site is reopened, and the window where the previous version runs disappears.⁶

The cost is one extra round trip to the server each time the page opens. Reopening when nothing has been deployed still gets a 304 every time.

Add max-age to index.html

With Cache-Control: max-age=<seconds>, the browser uses its own copy of index.html without going to the server for the number of seconds you specify, regardless of time since modification.⁷

With max-age=300 (5 minutes), the previous version runs for about 5 minutes.

The cost is that the window where the previous version runs doesn't go away. If the seconds you specify are longer than what the browser would have guessed, the window gets longer.

Reload when loading fails

When a chunk fails to load, reload the page. In Vite, a load failure is signaled with the vite:preloadError event.⁸ The code below reloads only once, so that it doesn't keep reloading if the new version also fails.

window.addEventListener('vite:preloadError', () => {
  if (!sessionStorage.getItem('reloaded-once')) {
    sessionStorage.setItem('reloaded-once', '1');
    window.location.reload();
  }
});

The window where the previous version runs doesn't change. When a user clicks the button under the previous version, the page reloads once after the chunk fails to load, and the new version runs.

The cost is that in-progress page state is lost on reload. Only users who hit the error switch to the new version.

Which to pick

Start with no-cache. It is the only one of the three that removes the window where the previous version runs. If you want fewer round trips, use max-age instead to cap the window. Reloading on failure doesn't change the window, so combine it with one of the other two and use it to switch users who hit the error to the new version.

Scope

Everything above assumes that index.html has Last-Modified but no Cache-Control, that the deploy removes the previous version's files, and that the browser is Chromium.

Browsers

Only Chromium was measured. According to their source, Firefox and Safari also use their own copy of index.html without going to the server for 10% of the time since modification, the same as Chromium.⁹

When the previous version's files stay

In setups where the previous version's files stay on the server after a deploy, the window where the previous version runs doesn't change. Clicking the button during that window returns the previous version's chunk, and no error appears.

With no error, it is harder to notice that the previous version is running. The previous version's page keeps calling the newly deployed server-side code, so that code has to stay compatible with it.

A setup that puts build artifacts in S3 and runs aws s3 sync without --delete (the option that removes files that exist only at the destination) is one of these.

When missing paths don't return 404

With try_files $uri /index.html (an nginx setting that returns index.html for paths that don't exist), the request for the previous version's chunk gets a 200 whose body is index.html. No 404 shows up in the access log.

In the access log, look for lines where a hashed file path returned 200 with the same byte count as index.html (321 in the example below).

"GET /assets/feature-DHyZGV0s.js HTTP/1.1" 200 321 "http://127.0.0.1:18024/assets/index-DB4b__oJ.js"

The message that reaches the error tracker is the same Failed to fetch dynamically imported module as with the 404.

With a Service Worker

If a Service Worker (a script that can intercept the page's requests and answer them from its own cache) serves index.html from its own cache, the same symptom appears. In that case, adding no-cache or max-age to index.html has no effect.

To check whether one is registered, run navigator.serviceWorker.getRegistrations() in the browser console.

Built with webpack

The same failure surfaces as ChunkLoadError. The message starts with Loading chunk <chunkId> failed.¹⁰

RFC 9111, section 4.2.2. ↩
The line lifetimes.freshness = (date_value - last_modified_value.value()) / 10; in Chromium 153.0.8010.12's GetFreshnessLifetimes. ↩
The browser was Chromium headless shell 153.0.8010.12 (the build without a window) driven by Playwright 1.63.0, with nginx 1.30.5 and Vite 8.3.1. The time since modification was produced by shifting the modification time of the served files into the past when starting the container (nginx returns a file's modification time as Last-Modified). The time until reopening was produced with an Age header that the previous version's server added to index.html. The browser treats the response as already that many seconds old when it receives it (RFC 9111, section 4.2.3). At a time since modification of 10 minutes, the results matched actually waiting. Each condition ran 4 times in CI, for 12 trials per data point. The reproduction setup and scripts are in show-your-work. ↩
Compared at a time since modification of 100 minutes, side by side with relaunching the browser, within the same deploy. ↩
RFC 9111, section 5.2.2.4. ↩
Using the same setup as with no fix, I opened an index.html with a time since modification of 100 minutes and compared different times until reopening. Each condition ran 4 times in CI, for 12 trials per data point. ↩
RFC 9111, section 5.2.2.1. ↩
Load error handling in the Vite docs. ↩
Firefox: nsHttpResponseHead.cpp. Safari: WebKit's CacheValidation.cpp. ↩
webpack v5.111.1 source. ↩

Only p99 is slow at low CPU usage: the cause is the CPU limit

Kazu — Tue, 06 Oct 2026 13:01:00 +0000

When a container has a CPU limit (the amount of CPU the container may use, set with --cpus in Docker and resources.limits.cpu in Kubernetes), it gets a fixed amount of CPU time for each 100ms period. Once it uses that up, the kernel stops every thread in the container until the end of the period. This happens when processing overlaps within one period, and the only requests delayed are the ones being processed when the container is stopped and the ones that arrive while it is stopped.

As long as the container is stopped in only some periods, p50 (the value at the 50% position when response times are sorted from fastest to slowest) barely changes, while p99 (the value at the 99% position) and the rest of the slow end go up. The container uses no CPU while it is stopped, so CPU usage stays low. The number of periods in which the container was stopped is counted in nr_throttled in cpu.stat (a file where the kernel writes cumulative CPU usage counters for each container).

I reproduced this with Docker and checked the conditions under which the container is stopped and the effect of raising the limit.

What is happening

A --cpus 0.5 limit is enforced as up to 50ms of CPU time in each 100ms period.¹ This 50ms is drawn down by the total CPU time used by all threads in the container.

Once the 50ms is used up, every thread in the container stops until the next period starts.² The kernel stopping a container's threads this way is called throttling. A request being processed stops partway through, and a request that arrives during the stop does not start processing until the next period.

Even when average CPU usage is low, requests arriving at random intervals sometimes overlap within one period. If three requests that each use 20ms of CPU overlap in one period, they add up to 60ms, which exceeds 50ms.

The diagram below shows A and B arriving at 0ms, C at 10ms, D at 40ms, E at 100ms, and F at 105ms. Each character is 5ms. # is time spent processing, . is time spent stopped, and - is time spent waiting before processing starts. The number of # in a column is the number of CPUs in use at that point.

ms     0   20  40  60  80  100 120 140 160 180 200
       |                   |                   |
A      ####
B      ####
C        ##................##
D              ------------###.................#
E                          ###.................#
F                           ##.................##

In the 0–100ms period, the # of A, B, and C add up to 10 at the 20ms mark, and the CPU time used reaches 50ms. A and B finish and return, but C, still being processed, is stopped until 100ms. D arrives at 40ms and does not start processing until 100ms.

In the next period, E and F join C and D, which were carried over from the previous period, and the 50ms runs out again at 115ms. C returns at 110ms, but D, E, and F are stopped until 200ms. D returns at 205ms, 165ms after it arrived.

Measurements

I ran an HTTP server that uses 20ms of CPU per request before responding (4 workers) in a container with --cpus 0.5, and sent requests at random intervals from outside the container for 60 seconds.³ I used three average arrival rates: 5, 10, and 20 per second. The throttle rate is the fraction of periods in which the container was stopped: nr_throttled divided by nr_periods (the number of elapsed periods).

Arrivals per second	CPU usage (fraction of limit)	Throttle rate	p50	p99
5	20.5%	2.8%	20.5ms	67.0ms
10	43.5%	12.2%	20.6ms	138.6ms
20	87.1%	70.1%	91.7ms	733.7ms

Even at 5 and 10 per second, where CPU usage stays below 50%, p99 reaches 67.0ms and 138.6ms. p50 stays at 20ms.

Checking your own environment

If the throttle rate rises at the times p99 goes up, the container is being stopped by its CPU limit. If the throttle rate stays at 0, the slowness has a cause other than the CPU limit.

Calculate the throttle rate from cpu.stat. From inside the container, read /sys/fs/cgroup/cpu.stat. From the host, read /sys/fs/cgroup/system.slice/docker-<id>.scope/cpu.stat.⁴

Both nr_throttled and nr_periods are cumulative since the container started, so read them twice while p99 is elevated and divide the increase in nr_throttled by the increase in nr_periods.

Fix

Raise the CPU limit until the throttle rate is close to 0. At an average of 10 arrivals per second, raising the limit to 1.0 brings the throttle rate to nearly 0, and p99 drops to the low 30ms range, the same as at 1.5 and 2.0, where the throttle rate is 0. The container uses about 0.2 cores on average, and the limit needs to be about 5 times that.

`--cpus`	Throttle rate	p99
0.5	12.2%	138.6ms
0.75	2.1%	44.5ms
1.0	0.38%	33.2ms
1.5	0%	31.1ms
2.0	0%	33.0ms

Raising the limit increases the CPU the container can use at once in periods where requests overlap. Note that at times when the host's CPU has no headroom, the container competes for that extra CPU with the other containers on the same host.

The kernel 6.17 cgroup v2 documentation on cpu.max. ↩
The kernel 6.17 documentation on CFS bandwidth control. ↩
GitHub Actions ubuntu-24.04 (4 vCPU), kernel 6.17, cgroup v2, Docker cgroup driver systemd. 294 / 609 / 1228 requests were sent, with 0 errors. Each condition ran in a separate job, so the CPU model differs between jobs. The code is in show-your-work, and the results are in the CI run. ↩
This is the location when Docker's cgroup driver is systemd. <id> is the full container ID. ↩

No shell in a distroless Pod: what kubectl debug can and can't see

Kazu — Tue, 29 Sep 2026 13:02:00 +0000

Something is off with a Pod. The app logs show nothing that looks like an error.

At this point I want to get inside and look around: open a shell with kubectl exec, check processes with ps, send a request to the app with curl localhost. The usual routine.

But the shell won't start.

$ kubectl exec app -c app -- sh
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "5559c54c81498e2d06cfaf7704ee420b7a48a48ef1531e34307aa515f53083b3": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

The last part, exec: "sh": executable file not found in $PATH, is the answer: this container has no sh. It has no ls or cat either.

The container runs from a distroless image. A distroless image contains only the app's binary and the minimum files it needs to run. It leaves out the shell and the package manager so that there are no tools for an attacker to use. As a result, even during an incident, you can't run commands inside the container to investigate.

Kubernetes has a mechanism for this situation. You can add a temporary container to a running Pod. This is called an ephemeral container, and you add one with the kubectl debug command.

You leave the app container untouched and add a separate container to the same Pod, one that has curl, ss, and tcpdump. Containers in a Pod share the network, so running curl localhost:3000 from the added container reaches the app directly.

The official docs cover the basic usage of kubectl debug, so this article covers what you can see once you're inside.

Once you're inside, you quickly run into cases where you can't get the information you want.

You try to read the app's config file, and nothing exists at that path
You check the CPU limit in /sys/fs/cgroup/cpu.max, and instead of the app's 50000 100000 you get your own unlimited max 100000
There's a way around to see the app's files, but without SYS_PTRACE you get Permission denied

Even within the same Pod, how far you can reach into the app is determined by Linux namespaces, and whether you can read what you reach is determined by permissions. This article doesn't walk through tracking down the cause of an incident. It checks, one item at a time against real output, what you can and can't see once inside along these two axes, and shows a workaround for each thing you can't see.

Test setup

I ran everything on a kind v0.33.0 cluster on a GitHub Actions x86_64 runner (ubuntu-24.04, kernel 6.17). Kubernetes is v1.37.0 and the container runtime is containerd v2.3.4. The target app is a small HTTP server written in Go, built on gcr.io/distroless/static-debian12:nonroot. For the debug image I used nicolaka/netshoot:v0.16. All output in this article is pasted as-is from this environment.

kind is a tool that runs a Kubernetes cluster using Docker containers as nodes. I used a cluster with a single node (ded-control-plane).

I put one Pod, app, on this cluster and added containers to it with kubectl debug to investigate.

Node ded-control-plane (kind)
└── Pod app
    ├── app            distroless HTTP server (:3000, uid 65532)
    ├── sidecar        HTTP server from the same image (:4000)
    ├── dbg-notarget   added with kubectl debug (no --target)
    ├── dbg-default    added with kubectl debug (--target=app)
    └── dbg-sysadmin   added with kubectl debug (--target=app --profile=sysadmin)

The app container is the one under investigation. The distroless :nonroot image sets the runtime user to uid 65532 (nonroot), and this Pod doesn't override it with runAsUser, so app runs as uid 65532. Two of its settings matter in later sections.

The CPU limit is 500m (half a core)
The ConfigMap's app.json is mounted at /config

sidecar is a regular container defined in the Pod from the start, running the same image as app on a different port. It's there to compare what containers in a Pod share.

The three containers whose names start with dbg- are ephemeral containers added later with kubectl debug. They differ only in the options passed.

--target=app makes the added container see the same set of processes as the app container. With it, app's processes show up in ps.

--profile selects the permissions the added container gets. If you don't specify it, you get general, which adds one capability for inspecting other processes (SYS_PTRACE). dbg-notarget and dbg-default use this profile. Specifying sysadmin gives you a privileged container, meaning one with nearly every permission. A later section shows measured differences between the profiles.

The reproduction code is here.

https://github.com/shinagawa-web/show-your-work/tree/main/distroless-ephemeral-debug

Background: container isolation is set per namespace

Linux can partition what a process sees (the network, process IDs, the filesystem, and so on), each one separately. Each of these partitions is called a namespace. A container is built from a combination of these namespaces.

The key point is that each namespace is independent. You can't describe a container as simply "isolated". It's normal for a container to share the network with its neighbor while having a separate filesystem. This explains why, from a container added with kubectl debug, curl localhost reaches the app while the app's config file is not visible. Both happen at the same time.

You can check how they're actually split from the node. Each process has one link per namespace under /proc/<pid>/ns/. Reading /proc/<pid>/ns/net with readlink, for example, returns a string like net:[4026532670]. The number in brackets identifies the namespace, and processes with the same number are in the same namespace.

To do this you need each container's PID. On the node ded-control-plane, I used crictl (a command that talks to the container runtime directly) and took each container's process PID on the node from .info.pid in crictl inspect <container ID>. For each PID I ran readlink on every link under /proc/<pid>/ns/ and put the results on one line per container. That gives the table below, trimmed to the rows and columns this article needs. I dropped the ipc, uts, user, and time columns because this article doesn't use them.

container      node_pid  net                    pid                    mnt                    cgroup
node-init      1         net:[4026532365]       pid:[4026532362]       mnt:[4026532359]       cgroup:[4026532364]
app            4082      net:[4026532670]       pid:[4026532736]       mnt:[4026532735]       cgroup:[4026532737]
sidecar        4118      net:[4026532670]       pid:[4026532739]       mnt:[4026532738]       cgroup:[4026532740]
dbg-notarget   4193      net:[4026532670]       pid:[4026532742]       mnt:[4026532741]       cgroup:[4026532743]
dbg-default    4247      net:[4026532670]       pid:[4026532736]       mnt:[4026532744]       cgroup:[4026532745]
dbg-sysadmin   4863      net:[4026532670]       pid:[4026532736]       mnt:[4026532754]       cgroup:[4026532364]

node_pid is the PID on the node from crictl, and the columns to its right are the readlink results. node-init is PID 1 on the node, included for comparison.

net has the same number for every container. For pid, only dbg-default and dbg-sysadmin have the same number as app. mnt is different for every container. For cgroup, only dbg-sysadmin has the same number as the node.

The next table summarizes these results per namespace and adds what each namespace separates and what a debug container can see.

namespace	What it separates	Between containers in a Pod	Ephemeral container	Result
network	IP addresses, ports, sockets	Shared	Shared	`curl localhost` and `ss` show the app's traffic
PID	Process IDs and which processes are visible	Separate (can be shared with `shareProcessNamespace`)	Shared with the target when `--target` is set, separate otherwise	With `--target`, the target's processes show up in `ps`
mount	How the filesystem looks	Separate	Separate	The other container's files aren't directly visible
cgroup	How CPU and memory limits look	Separate	Separate (only sysadmin shares it with the node)	`/sys/fs/cgroup` shows your own values (except with sysadmin)

This table covers how far you can reach into the app. Whether you can read what you reach depends on permissions, which I cover in "Permissions: SYS_PTRACE and profiles".

Adding an ephemeral container

For the debug image I use nicolaka/netshoot. It bundles network troubleshooting tools such as curl, ss, tcpdump, and dig, along with basic commands like ps.

To get an interactive shell, run this.

$ kubectl debug -it app --image=nicolaka/netshoot:v0.16 --target=app -c dbg-it -- bash
Targeting container "app". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.

Two warnings appear before the shell opens.

The first says that --target only works if the container runtime supports it. In this article I confirmed it works on containerd v2.3.4. I don't cover what things look like on a runtime without support.

The second says that every command you type in this session, and its output, is recorded in the container logs. For example, if you print /proc/1/environ as in "Environment and arguments: read from /proc/1" below, the app's environment variables end up in these logs, secrets included.

`--target` changes what `ps` shows

In the table in the background section, the PID row was the only one that changed with --target. Let's check this with ps.

For this test I wanted to run commands in the same container repeatedly and compare the output. So instead of entering with -it, I kept each container running with sleep infinity and ran commands with kubectl exec -c <container name>. I added dbg-notarget without --target and dbg-default with --target=app as follows. --image-pull-policy=Never is there because the image was preloaded onto the kind node.

$ kubectl debug app --image=nicolaka/netshoot:v0.16 --image-pull-policy=Never -c dbg-notarget -- sleep infinity
$ kubectl debug app --image=nicolaka/netshoot:v0.16 --image-pull-policy=Never -c dbg-default --target=app -- sleep infinity
Targeting container "app". If you don't see processes from this container it may be because the container runtime doesn't support this feature.

I ran ps auxf in each.

$ kubectl exec app -c dbg-notarget -- ps auxf
PID   USER     TIME  COMMAND
    1 root      0:00 sleep infinity
   14 root      0:00 ps auxf

$ kubectl exec app -c dbg-default -- ps auxf
PID   USER     TIME  COMMAND
    1 65532     0:00 /server -http=:3000
   29 root      0:00 sleep infinity
   42 root      0:00 ps auxf

Without --target, all you see is your own sleep infinity and the ps you just ran. The app's process is nowhere to be found.

With --target=app, the app's /server shows up as PID 1, and your own sleep infinity becomes PID 29. The 65532 in the USER column is the uid of the distroless nonroot user.

The app starts as PID 1 inside its container, so it also appears as PID 1 from a container that joins its PID namespace with --target. That's why later sections write /proc/1/.... However, if the app is launched through an init process like tini, or if the Pod has shareProcessNamespace enabled, the app may not be PID 1. Check with ps first.

dbg-notarget, which has no --target, also has the capability for inspecting other processes (SYS_PTRACE). But it can't see the app's process at all, so there's nothing to use it on. Every step below that uses /proc/1/root or /proc/1/environ assumes you entered with --target.

What you can see directly

From here on, I check the background table against real output. First come the things an ephemeral container reaches directly. The network namespace is shared across the whole Pod, and the PID namespace is shared with app through --target. Unless stated otherwise, the results are from inside dbg-default, which was added with --target=app.

dbg-default has SYS_PTRACE, so everything read in this section was readable except /proc/1/stack. Some of these become unreadable without SYS_PTRACE. I cover that in "Permissions: SYS_PTRACE and profiles".

Network: use it directly

The network row in the table said "Shared". From dbg-default, you see the same network as the app.

$ ss -tanp
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      4096               *:3000            *:*    users:(("server",pid=1,fd=4))
LISTEN 0      4096               *:4000            *:*

:3000 is the port the app (server, pid 1) listens on. :4000 is the sidecar's port. You can see that something is listening on it, but not which process.

This also matches the table. The network is shared across the whole Pod, so the sidecar's socket is visible. The PID namespace, on the other hand, is shared only with app, so the sidecar's process isn't visible. ss finds a socket's owner by looking through the processes under /proc, so it can't show the name of a process that isn't visible.

localhost reaches the app as well.

$ curl -sv -m 3 localhost:3000/health
...
< HTTP/1.1 200 OK
< Date: Sun, 27 Sep 2026 02:28:19 GMT
< Content-Length: 3
< Content-Type: text/plain; charset=utf-8
< 
{ [3 bytes data]
ok
* Connection #0 to host localhost:3000 left intact

The response is 200 OK with the body ok, so the app answers requests from localhost. The ... part is the connection setup, which I left out.

You can capture packets too. I ran tcpdump while sending curl requests in the background, and stopped after the first 2 packets, which show the SYN and SYN-ACK of one connection.

$ (for i in 1 2 3; do sleep 1; curl -s -m 2 localhost:3000/health >/dev/null; done) & tcpdump -i any -nn -c 2 port 3000; rc=$?; wait; exit $rc
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
02:28:20.415650 lo    In  IP6 ::1.43214 > ::1.3000: Flags [S], seq 838274280, win 65476, options [mss 65476,sackOK,TS val 235319763 ecr 0,nop,wscale 10], length 0
02:28:20.415666 lo    In  IP6 ::1.3000 > ::1.43214: Flags [S.], seq 4234760723, ack 838274281, win 65464, options [mss 65476,sackOK,TS val 235319763 ecr 235319763,nop,wscale 10], length 0
2 packets captured
20 packets received by filter
0 packets dropped by kernel

Both captured packets are IPv6 traffic over lo (loopback). The first is a connection request (Flags [S], SYN) from port 43214 on ::1 to the app's :3000. The second is the app's reply (Flags [S.], SYN-ACK). Traffic to the app can be captured from inside the debug container.

Name resolution works too. I looked up the name of the Service app against the nameserver listed in dbg-default's own /etc/resolv.conf.

$ dig @$(awk '/^nameserver/{print $2; exit}' /etc/resolv.conf) app.default.svc.cluster.local +short
10.96.248.99

The returned 10.96.248.99 is the ClusterIP of the Service app. The app's environment variable APP_SERVICE_HOST holds the same value.

Process state: read from /proc/1

As shown in "--target changes what ps shows", the app appears as PID 1 when you enter with --target=app. Its process state is readable under /proc/1/.

$ grep -E '^(Name|State|Uid|Threads|VmRSS):' /proc/1/status
Name:   server
State:  S (sleeping)
Uid:    65532   65532   65532   65532
VmRSS:      7256 kB
Threads:    4

State is S (sleeping), meaning the process is blocked waiting for something. Threads shows 4 threads, and VmRSS, the physical memory actually in use, is 7256 kB. Uid is 65532 in every field because the process runs as the nonroot user.

The list of open files is in /proc/1/fd.

$ ls -l /proc/1/fd
total 0
lrwx------    1 65532    65532           64 Sep 27 02:28 0 -> /dev/null
l-wx------    1 65532    65532           64 Sep 27 02:28 1 -> pipe:[45581]
l-wx------    1 65532    65532           64 Sep 27 02:28 2 -> pipe:[45582]
lr-x------    1 65532    65532           64 Sep 27 02:28 3 -> /sys/fs/cgroup/cpu.max
lrwx------    1 65532    65532           64 Sep 27 02:28 4 -> socket:[45610]
lrwx------    1 65532    65532           64 Sep 27 02:28 5 -> anon_inode:[eventpoll]
lrwx------    1 65532    65532           64 Sep 27 02:28 6 -> anon_inode:[eventfd]

fd 4 is a socket, and it corresponds to fd=4 (:3000) in the earlier ss -tanp output.

For slow operations such as receiving from the network or reading from disk, a program makes a system call and blocks inside the kernel (the core of the OS) until the call completes. /proc/1/wchan (short for wait channel) holds the name of the kernel function where the process is currently blocked, so it tells you what the process is waiting for.

$ cat /proc/1/wchan; echo
ep_poll

In this run it was ep_poll. When I entered the same Pod with --profile=sysadmin, it returned futex_do_wait. The value depends on when you read it, so read it a few times instead of judging from a single read.

ep_poll: waiting for data to arrive on one of its connections. This is the idle state of waiting for requests
futex_do_wait: blocked while synchronizing with another thread of the same program. In Go apps, idle threads are often in this state

For more detail on what the process is waiting for, /proc/1/stack prints the call stack inside the kernel. However, it's not readable from dbg-default.

$ cat /proc/1/stack
cat: read error: Permission denied

Among the profiles I tried, only the privileged dbg-sysadmin could read it.

$ cat /proc/1/stack
[<0>] futex_do_wait+0x3a/0x70
[<0>] __futex_wait+0x99/0x100
[<0>] futex_wait+0x72/0x120
[<0>] do_futex+0xc9/0x190
[<0>] __x64_sys_futex+0x125/0x1f0
[<0>] x64_sys_call+0x1d1d/0x20d0
[<0>] do_syscall_64+0x7b/0xb40
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

This is the call path inside the kernel, read from bottom to top. The bottom is the entry point where the app entered the kernel, and the top is where it's blocked right now. In this output, the app is waiting in futex (a system call for synchronizing threads) and is blocked in futex_do_wait at the top. Numbers like +0x3a/0x70 are offsets within the function, and you can usually ignore them.

The top entry, futex_do_wait, is the same as the wchan value read from the same dbg-sysadmin. stack also shows the path that led there. Still, if all you need is what the process is waiting for, wchan is usually enough.

Environment and arguments: read from /proc/1

The app's environment variables at startup are in /proc/1/environ. The entries are separated by NUL characters, so replace them with newlines to read them.

$ tr '\0' '\n' < /proc/1/environ
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=app
SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
APP_MODE=production
APP_UPSTREAM={"upstream": "http://backend.default.svc:8080", "timeout_ms": 250}
...

After the ..., the output continues with Service addresses and other variables that Kubernetes injects automatically.

The startup arguments are in /proc/1/cmdline.

$ tr '\0' ' ' < /proc/1/cmdline; echo
/server -http=:3000

The only argument is -http=:3000, the same command line that ps showed.

What you can't see directly

Next come the things the table marks "Separate", which you can't reach just by entering: mount and cgroup. Both have a workaround through /proc/1/root.

Files: read from /proc/1/root

In the introduction I wrote that when you try to read the app's config file, nothing exists at that path. Here is that case. The app mounts a ConfigMap at /config, but reading the same path from dbg-default finds nothing.

$ cat /config/app.json
cat: can't open '/config/app.json': No such file or directory

The mount row in the table said "Separate". The / in dbg-default is the netshoot image's filesystem, which is separate from the app's.

This is where /proc/1/root comes in. /proc/<pid>/root is a link that points to "/ as seen by that process", and following it takes you straight into the filesystem the app sees. You can read the contents of an image that has no shell and no ls with the tools on your side.

$ ls /proc/1/root/
bin
boot
config
dev
etc
home
lib
proc
...
root
run
sbin
scratch
server
sys
tmp
usr
var

config is where the ConfigMap is mounted, scratch is where the Pod spec mounts an emptyDir, and server is the app's binary. The ConfigMap's contents are readable too.

$ cat /proc/1/root/config/app.json
{"upstream": "http://backend.default.svc:8080", "timeout_ms": 250}

The config the app reads is now readable from the debug container.

If you suspect name resolution, you can read the resolv.conf the app actually uses in the same way.

$ cat /proc/1/root/etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local urp3p5u2pksefpdlab2z34ww2g.gx.internal.cloudapp.net
nameserver 10.96.0.10
options ndots:5

10.96.0.10 in nameserver is where the app sends its queries. search lists the domains that are appended to short names in order. ndots:5 means that for a name with fewer than 5 dots, the search domains are appended and tried first. When a name doesn't resolve, this tells you which names the app actually queries. The last search entry, …cloudapp.net, isn't a cluster domain. It comes from the host side's configuration.

cgroup: read from /proc/1/root

A cgroup is the mechanism that applies CPU and memory limits per container. The limit values are readable from files under /sys/fs/cgroup. The app's CPU limit is 500m (half a core), so cpu.max should contain 50000 100000 (up to 50,000 microseconds out of every 100,000 microseconds).

From dbg-default, I checked where my own cgroup is, that /sys/fs/cgroup lists cgroup files, and the value of cpu.max, in one go. The result looks like this.

$ cat /proc/self/cgroup; ls /sys/fs/cgroup | head -n 8; cat /sys/fs/cgroup/cpu.max
0::/
cgroup.controllers
cgroup.events
cgroup.freeze
cgroup.kill
cgroup.max.depth
cgroup.max.descendants
cgroup.pressure
cgroup.procs
max 100000

max 100000 means "no limit", and it's not the app's value. This is the case from the introduction where you get your own value. What you see is the debug container's own cgroup, which has no limit.

The cgroup row in the table said "Separate". dbg-default has a cgroup namespace of its own, and its own cgroup appears at the top of /sys/fs/cgroup. The first line, 0::/, means "my cgroup is at the top of the hierarchy as I see it".

To read the app's values, go through /proc/1/root as in "Files: read from /proc/1/root". Reading /sys/fs/cgroup as the app sees it gives the app's own values.

$ cat /proc/1/root/sys/fs/cgroup/cpu.max
50000 100000

The output is 50000 100000, matching the app's 500m limit. This path goes through /proc/1/root, so whether you can read it depends on having SYS_PTRACE. It was readable with general (no --profile) and sysadmin, and returned Permission denied with legacy, baseline, and netadmin. I explain why in "Permissions: SYS_PTRACE and profiles".

The exception is sysadmin. dbg-sysadmin shares the cgroup namespace with the node, so the app's cgroup path shown in /proc/1/cgroup exists at the same path under its own /sys/fs/cgroup.

$ ls -d /sys/fs/cgroup$(cut -d: -f3 /proc/1/cgroup)
/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-pod719135d1_ee24_4f46_948a_440e1c6f4f77.slice/cri-containerd-698b44cb65ab91fba808118cba48f23ab4761ddd8a3e05977a04c4323bcef648.scope

This reaches the app's cgroup directory directly, without going through /proc/1/root.

If you have permission to get onto the node, you can also read the app's cgroup directly from the node side. I cover that later in "Reading cgroups from the node".

Permissions: SYS_PTRACE and profiles

The reads of /proc/1/root and /proc/1/environ so far worked because dbg-default has SYS_PTRACE. The table in the background section only covers how far you can reach into the app. Whether you can read what you reach depends on the permissions in this section.

SYS_PTRACE: without it, you can't read the contents

SYS_PTRACE is what decides between reading the contents of /proc/1 and getting Permission denied.

When you read /proc/1/root, /proc/1/environ, /proc/1/maps, the link targets in /proc/1/fd, and similar files, the kernel checks whether the reading process is allowed to inspect the contents of the other process. dbg-default runs as root (uid 0 / gid 0). The app runs as uid 65532, and its gid is also 65532 (you can see this in the owner columns, 65532 65532, of the earlier ls -l /proc/1/fd). When your uid and gid differ from the target's like this, inspecting it requires SYS_PTRACE.

This isn't about file permissions. You're root, and you even have the capability to bypass file permissions (DAC_OVERRIDE), but that doesn't get you past this check.

To confirm this, I also entered with --profile=legacy, a profile without SYS_PTRACE, ran the same commands as with default, and compared the results. This container is dbg-legacy, added to the same Pod app with only --profile changed. legacy is deprecated, and kubectl prints a warning when you add it saying that it's scheduled for removal in v1.39. baseline, which also lacks SYS_PTRACE, gave the same results as legacy.

I compared 9 items that read the contents of the app's process. They are the process column of ss -tanp, the link targets in /proc/1/fd, /proc/1/maps, /proc/1/environ, the listing of /proc/1/root/, and reads of app.json, resolv.conf, cpu.max, and cpu.stat through /proc/1/root/.

All 9 items failed with legacy, and all 9 passed with default.

Here are two examples of the failures after entering app with legacy.

$ cat /proc/1/root/config/app.json
cat: can't open '/proc/1/root/config/app.json': Permission denied

$ ls -l /proc/1/fd
total 0
ls: /proc/1/fd/0: cannot read link: Permission denied
lrwx------    1 65532    65532           64 Sep 27 02:28 0
...

Both are Permission denied. ls -l /proc/1/fd prints the list but can't read the link targets.

The users:(...) column disappears from ss -tanp.

$ ss -tanp
State     Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
LISTEN    0      4096               *:3000             *:*   
LISTEN    0      4096               *:4000             *:*   
...

The :3000 line has no users:(("server",pid=1,fd=4)) either, so you can't tell which process owns the socket.

wchan doesn't produce an error, which makes this case easy to miss. Reading it after entering app with legacy returned 0 instead of futex_do_wait or ep_poll.

$ cat /proc/1/wchan; echo
0

If wchan is 0, first check whether you have SYS_PTRACE.

On the other hand, /proc/1/status, /proc/1/cmdline, and the listing of /proc/1/fd itself were readable even with legacy.

SYS_PTRACE is the capability for inspecting another process, and it's useless if that process isn't visible. --target makes the target visible, and SYS_PTRACE grants permission to inspect it. The commands so far work only with both in place.

uid and gid: when both match, reads worked without SYS_PTRACE

There's another way past this kernel check. In this test, when both the uid and the gid of my process matched the target's, reads worked without SYS_PTRACE. However, every target I tried drops all capabilities with capabilities.drop: ["ALL"]. The kernel check also has a condition on the target's capabilities, and I didn't test targets that keep any capabilities.

To confirm this, I prepared two Pods in addition to app, with only runAsUser and runAsGroup changed. app-uid0 runs the app as uid 0 / gid 65532, and app-uid0-gid0 runs it as uid 0 / gid 0.

Reading each app's /proc/<pid>/status from the node gave the following. The first block is app-uid0, and the second is app-uid0-gid0.

$ grep -E '^(Uid|Gid|Groups|CapInh|CapPrm|CapEff|CapBnd|CapAmb|NoNewPrivs|Seccomp):' /proc/7049/status
Uid:    0   0   0   0
Gid:    65532   65532   65532   65532
...
CapEff: 0000000000000000
...

$ grep -E '^(Uid|Gid|Groups|CapInh|CapPrm|CapEff|CapBnd|CapAmb|NoNewPrivs|Seccomp):' /proc/7042/status
Uid:    0   0   0   0
Gid:    0   0   0   0
...
CapEff: 0000000000000000
...

app-uid0 has uid 0 and gid 65532, and app-uid0-gid0 has uid 0 and gid 0. Both have CapEff set to all zeros, so both run with every capability dropped.

I entered each one with legacy and ran the same commands. The results for app are included for comparison.

Target app	uid / gid	Result with legacy
`app`	65532 / 65532	All 9 items failed
`app-uid0`	0 / 65532	All 9 items failed
`app-uid0-gid0`	0 / 0	All passed

Matching only the uid (app-uid0) isn't enough. The reads pass only when the gid matches as well (app-uid0-gid0).

wchan follows the same condition. Entering app-uid0-gid0 with legacy returned futex_do_wait rather than 0. So the 0 seen in "SYS_PTRACE: without it, you can't read the contents" can be read as the value the kernel returns when this check fails.

The distroless nonroot image uses 65532 for both uid and gid. As long as you enter as root, both the uid and the gid differ from the app's, so the quickest route is to enter with a profile that includes SYS_PTRACE.

Profiles: only general and sysadmin can read /proc/1/root

The capabilities the debug container gets are determined by --profile, and the profile determines which commands fail.

What differs between profiles is the securityContext, the part of the spec that sets things like a container's capabilities. I also added legacy, general, baseline, and netadmin to the same Pod app as dbg-default and dbg-sysadmin, changing only --profile, and ran the same commands to compare them. restricted requires an image that runs as non-root, and it didn't start with netshoot, which runs as root, so I left it out of the comparison. Here are the results.

profile	securityContext	Capabilities	`/proc/1/root`	`iptables`
general (same as no `--profile`)	`capabilities.add: [SYS_PTRACE]`	Base + `SYS_PTRACE`	Readable	Fails
legacy / baseline	None	Base only	Not readable	Fails
netadmin	`capabilities.add: [NET_ADMIN, NET_RAW]`	Base + `NET_ADMIN`	Not readable	Works
sysadmin	`privileged: true`	Nearly all	Readable	Works

"Base" in the table is the set of capabilities the container runtime (containerd) grants when the securityContext specifies nothing.

Without NET_ADMIN, iptables fails with the misleading error Permission denied (you must be root), even when it runs as root. tcpdump worked with every profile in the table, because NET_RAW, which packet capture needs, is in the base set from the start. netadmin adds NET_ADMIN and NET_RAW, but NET_RAW is already there, so the only real addition is NET_ADMIN. That's why the Capabilities column shows "Base + NET_ADMIN".

Reading cgroups from the node

As shown in "cgroup: read from /proc/1/root", an ephemeral container can read the app's cgroup values through /proc/1/root/sys/fs/cgroup/..., and with sysadmin you can follow them directly from /sys/fs/cgroup. Looking from the node is the way out when neither of these is available. Here are two such cases.

Pod Security settings or similar policies on the app's Kubernetes namespace allow neither SYS_PTRACE nor privileged containers
You want to look across multiple Pods on the node

Even in the first case, starting the node debug Pod in a different Kubernetes namespace that allows privileged Pods should let you get onto the node. I didn't test that far in this article, though.

Enter the node with kubectl debug node

kubectl debug node/<node name> starts a debug Pod on the specified node and attaches you to it.

$ kubectl debug node/ded-control-plane -it --profile=sysadmin --image=nicolaka/netshoot:v0.16 -- bash
Creating debugging pod node-debugger-ded-control-plane-sjpv8 with container debugger on node ded-control-plane.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.

This Pod's spec shows that it shares the PID, network, and IPC namespaces with the node, runs privileged, and mounts the node's / at /host.

$ kubectl get pod/node-debugger-ded-control-plane-sjpv8 -o json | jq '{hostPID: .spec.hostPID, hostNetwork: .spec.hostNetwork, hostIPC: .spec.hostIPC, sc: .spec.containers[0].securityContext, mounts: .spec.containers[0].volumeMounts, volumes: .spec.volumes}'
{
  "hostPID": true,
  "hostNetwork": true,
  "hostIPC": true,
  "sc": {
    "privileged": true
  },
  "mounts": [
    {
      "mountPath": "/host",
      "name": "host-root"
    },
...
  "volumes": [
    {
      "hostPath": {
        "path": "/",
        "type": ""
      },
      "name": "host-root"
    },
...

It's privileged, and on top of that it has the node's PID, network, and IPC namespaces and the node's filesystem. That makes it a Pod with very strong permissions. Every process on the node is visible, and so is the node's entire /. When you use it on a production node, keep in mind that this is the level of access you're working with.

Let's look under /host.

$ ls --color=never /host
LICENSES  bin       boot      dev       etc       home      kind      lib       lib64     media     mnt       opt       proc      root      run       sbin      srv       sys       tmp       usr       var

This is the node's /.

Trace the cgroup from the app's process

The PID namespace is shared with the node, so you can find the app's process on the node with ps.

$ ps -eo pid,user,args | grep '[/]server -http='
 4082 65532    /server -http=:3000
 4118 65532    /server -http=:4000

:3000 is the app and :4000 is the sidecar, both running as uid 65532. Take the PID of the :3000 process.

$ P=$(pgrep -u 65532 -f '/server -http=:3000')
$ echo "P=$P"
P=4082

4082 is the same as app's PID on the node in the table from the background section. Reading /proc/<pid>/cgroup for this PID gives the path from the root of the node's cgroup hierarchy.

$ cat /proc/$P/cgroup
0::/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-pod719135d1_ee24_4f46_948a_440e1c6f4f77.slice/cri-containerd-698b44cb65ab91fba808118cba48f23ab4761ddd8a3e05977a04c4323bcef648.scope

The path is the part after 0::. Extract it.

$ CG=$(cut -d: -f3 /proc/$P/cgroup)
$ echo "CG=$CG"
CG=/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-pod719135d1_ee24_4f46_948a_440e1c6f4f77.slice/cri-containerd-698b44cb65ab91fba808118cba48f23ab4761ddd8a3e05977a04c4323bcef648.scope

Append this path to /host/sys/fs/cgroup, which corresponds to the node's /sys/fs/cgroup.

$ cat /host/sys/fs/cgroup$CG/cpu.max
50000 100000

50000 100000 in cpu.max is a limit of up to 50,000 microseconds of CPU time out of every 100,000 microseconds (100 ms). It matches the 500m limit set in "Test setup".

Whether the app actually hits this limit shows up in cpu.stat in the same directory.

$ cat /host/sys/fs/cgroup$CG/cpu.stat
usage_usec 56270
user_usec 34699
system_usec 21570
nice_usec 0
core_sched.force_idle_usec 0
nr_periods 22
nr_throttled 0
throttled_usec 0
nr_bursts 0
burst_usec 0

In cpu.stat, nr_periods is the number of 100 ms cpu.max periods in which the app tried to use the CPU. nr_throttled is the number of those periods in which it hit the limit and was throttled. In this run it was 0 out of 22, so the app isn't hitting the limit.

Limitations and caveats

Added containers can't be removed from the Pod

When you add a container with kubectl debug, its definition is appended to the Pod's spec. The process inside stops using resources once it exits, but the definition stays, even for containers that have already terminated. Here's the list after adding dbg-exit, which exits immediately because it runs -- true.

$ kubectl get pod app -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'
dbg-notarget dbg-default dbg-it dbg-legacy dbg-general dbg-baseline dbg-restricted dbg-netadmin dbg-sysadmin dbg-exit

Every container I added for this article's tests is on the list. An attempt to remove one is rejected by the API server.

$ kubectl patch pod app --subresource=ephemeralcontainers --type=json -p '[{"op":"remove","path":"/spec/ephemeralContainers/0"}]'
The Pod "app" is invalid: spec.ephemeralContainers: Forbidden: existing ephemeral containers "dbg-notarget" may not be removed

The record goes away only when the Pod is recreated. Until then the names stay taken as well. An attempt to add another container named dbg-exit was rejected with existing ephemeral containers "dbg-exit" may not be changed. If you experiment casually on a production Pod, these records stay there.

You can't use `--target` on a stopped container

An app that crashes right after starting and keeps restarting is in a state called CrashLoopBackOff. For testing, I prepared a Pod crash that exits right after it starts. It mounts the same ConfigMap as app at /config and gets the same environment variables (APP_MODE, APP_UPSTREAM).

$ kubectl get pod crash
NAME    READY   STATUS             RESTARTS      AGE
crash   0/1     CrashLoopBackOff   4 (81s ago)   2m50s

It's in CrashLoopBackOff and has already restarted 4 times. To find out why it crashed, start with the logs of the previous container.

$ kubectl logs crash -c app --previous
2026/09/27 02:31:58 exiting on purpose

exiting on purpose is the message from the deliberate exit I built in for this test.

If you add a debug container to this Pod with --target=app, it doesn't start (its state is CreateContainerError). The process it's supposed to share a PID namespace with isn't running.

The workaround that fits this constraint is the --copy-to option of kubectl debug. It creates a copy of the Pod and lets you replace a container in the copy and run it.

First, I replaced only the image with netshoot using --set-image.

$ kubectl debug crash --copy-to=crash-copy --set-image=app=nicolaka/netshoot:v0.16
$ kubectl get pod crash-copy -o jsonpath='{range .spec.containers[*]}{.name} {.image} {.args}{"\n"}{end}'
app nicolaka/netshoot:v0.16 ["-exit"]

The image is now netshoot, but the original Pod's argument -exit is still there.

$ kubectl get pod crash-copy
NAME         READY   STATUS              RESTARTS     AGE
crash-copy   0/1     RunContainerError   2 (4s ago)   20s

The copy didn't start either and ended up in RunContainerError. The reason is in the last termination state.

$ kubectl get pod crash-copy -o jsonpath='{.status.containerStatuses[0].lastState}{"\n"}'
{"terminated":{..."message":"failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: \"-exit\": executable file not found in $PATH","reason":"StartError",...}}

The leftover argument -exit is treated as the command to run. It doesn't exist in netshoot, so the start fails.

So I specified the container to replace with --container=app, and passed both an image with --image and a command after --.

$ kubectl debug crash --copy-to=crash-copy2 --container=app --image=nicolaka/netshoot:v0.16 --image-pull-policy=Never -- sleep infinity
$ kubectl get pod crash-copy2 -o jsonpath='{range .spec.containers[*]}{.name} {.image} {.command} {.args}{"\n"}{end}'
app nicolaka/netshoot:v0.16 ["sleep","infinity"]

This time the command is sleep infinity and the arguments are empty.

$ kubectl get pod crash-copy2
NAME          READY   STATUS    RESTARTS   AGE
crash-copy2   1/1     Running   0          1s

The copy is Running and stays up.

In the copy's app container, the image and command are replaced, but the environment variables and volume mounts are carried over unchanged from the original Pod. So you can check the config file and environment variables the app would read at startup, under the same conditions.

$ kubectl exec crash-copy2 -c app -- cat /config/app.json
{"upstream": "http://backend.default.svc:8080", "timeout_ms": 250}

$ kubectl exec crash-copy2 -c app -- printenv APP_MODE APP_UPSTREAM
production
{"upstream": "http://backend.default.svc:8080", "timeout_ms": 250}

Both match what the original app receives. However, the copy doesn't run the app itself, so you can't see the process at the moment it crashes. What you can check is the set of conditions the app receives at startup.

Wrap-up

A distroless container has no shell, so you can't get in with kubectl exec. If you add an ephemeral container with kubectl debug, though, you can bring your own tools and investigate.

The network is shared with the Pod, so curl localhost, ss, tcpdump, and dig work directly
If you enter with --target, the app's process is visible, and you can read its state and environment variables from /proc/1
The app's filesystem is separate and not directly visible. You can read it through /proc/1/root
/sys/fs/cgroup shows the debug container's own values, except with sysadmin. The app's values are readable through /proc/1/root. If an ephemeral container can't reach them, read them from the node

Whether you can read the contents of what you reach depends on two things.

Inspecting another process requires SYS_PTRACE. You get it by entering with general (no --profile) or sysadmin
If the target drops all capabilities (every target in this test did), reads worked without SYS_PTRACE when your uid and gid both matched the target's

When you use this, watch out for two things.

Added containers can't be removed from the Pod
You can't use --target on a stopped container. For a Pod that keeps crashing, make a copy with --copy-to and investigate the copy

How far you can reach into the app depends on which namespaces you share with it. Whether you can read what you reach depends mainly on SYS_PTRACE. If the target drops all capabilities, matching uid and gid also let you read it. If you keep these two questions apart, you'll know which one to suspect when you get stuck.

Kubernetes: iptables is empty after migrating to Cilium — what to look at next

Kazu — Tue, 15 Sep 2026 13:03:00 +0000

A Kubernetes Service stops responding. The first move is always the same: get on the node and run iptables-save | grep KUBE-SERVICES to see which Pod a given Service address maps to.

kube-proxy wrote those rules. It's a standard Kubernetes component that runs on every node and keeps Service forwarding rules registered in iptables. Looking up where a Service routes means reading those rules.

Cilium is a Kubernetes CNI plugin — the piece that wires up Pod networking so Pods can talk to each other. Flannel and Calico fill the same role; you pick one per cluster.

Cilium has a mode that goes beyond Pod networking and also handles Service forwarding, the job kube-proxy used to own. That mode is called kube-proxy replacement. With kubeProxyReplacement=true, Cilium runs on the assumption that kube-proxy is absent. Cilium itself does not remove kube-proxy; on an existing cluster you have to remove it separately. The test environment in this article is built with kind's kubeProxyMode: none, so kube-proxy was never present.

On a node in that Cilium environment, run the same iptables-save | grep KUBE-SERVICES from before. Nothing comes back. The same setup on a kube-proxy cluster returns 13 lines; here it returns 0. KUBE-SERVICES is the chain kube-proxy creates — the entry point for packets destined to a ClusterIP. It doesn't exist.

Broaden the grep to KUBE- and you get a few lines back, but they're KUBE-FIREWALL and KUBE-KUBELET-CANARY, chains created by kubelet that have nothing to do with Service forwarding. The nat table itself has some rules; it isn't empty. Only the Service forwarding rules are missing.

You try a different table, a different node. KUBE-SERVICES is nowhere, and yet the application is up and Services are reachable.

Nothing is broken. The information just isn't in iptables.

This article covers three things: where Cilium stores the forwarding rules that used to live in iptables; why tcpdump on a ClusterIP produces nothing and conntrack -L returns no matches; and what to check, in what order, when connectivity fails.

The test setup runs two kind clusters side by side — one with kube-proxy, one with Cilium — and runs the same commands against the same Service on both. All output shown is from real machines.

All tests run on kind v0.27.0, node image kindest/node:v1.32.2, Cilium 1.17.3, cilium-cli v0.20.0, kernel 6.8. Both clusters are single-node; multi-node-only behaviors are out of scope. ClusterIPs change each time a cluster is recreated, so IPs may differ between sections.

The reproduction repo is at https://github.com/shinagawa-web/kubernetes-cilium-kube-proxy-replacement-debug-guide

Where forwarding rules live, and when they get written

kube-proxy

Rules were registered in netfilter

A Service's ClusterIP doesn't exist on any machine. When a Pod sends a packet to it, the destination gets rewritten to a real Pod IP somewhere in the forwarding path. kube-proxy was what registered those rewrite rules in netfilter. netfilter is the Linux kernel's packet-processing framework; iptables is the command for reading and writing its rules.

That's why iptables-save worked. The forwarding rule — which Pod a ClusterIP packet goes to — was stored as a text rule registered in the kernel, so dumping everything was enough.

Here's what the packet path looks like when a Pod connects to a ClusterIP:

client Pod
   |  sends packet destined for 10.96.111.7:80 (ClusterIP)
   v
network stack
   |
   v
netfilter nat table
   KUBE-SERVICES → KUBE-SVC-xxxx → KUBE-SEP-xxxx
   |  rewrites destination to 10.244.0.6:80 (Pod IP) here
   v
Pod

tcpdump could capture the pre-DNAT packet

In this path, there's a stretch before the rewrite where the packet still carries the ClusterIP as its destination. Running tcpdump on the node with the ClusterIP as the filter caught those packets. Capturing before and after the rewrite showed exactly which Pod IP the destination changed to.

iptables-save for the rules, tcpdump for the live packets. Those two tools covered Service forwarding.

Cilium

Cluster setup

To enable kube-proxy replacement, you disable kube-proxy at cluster creation and install Cilium as the CNI. The test repo does this with kind. If you already have Cilium running, skip ahead to "No kube-proxy means no chains."

First, the kind cluster config disables both kube-proxy and the default CNI:

# cluster/kind-cilium.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.32.2
networking:
  disableDefaultCNI: true
  kubeProxyMode: none

kubeProxyMode: none keeps kube-proxy off the nodes. disableDefaultCNI: true removes kindnet so Cilium can take the CNI slot.

After creating the cluster, install Cilium via Helm. kubeProxyReplacement=true starts Cilium in the mode where it owns Service forwarding as well.

helm install cilium cilium/cilium \
  --version 1.17.3 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost="${API_SERVER_IP}" \
  --set k8sServicePort="${API_SERVER_PORT}"

k8sServiceHost and k8sServicePort are the address Cilium uses to reach the Kubernetes API server. Without kube-proxy, you have to supply these explicitly. The full setup script is in scripts/setup-cilium.sh.

No kube-proxy means no chains

With this setup, kube-proxy doesn't exist. No process registers rules, so KUBE-SERVICES is never created. That's why the initial grep returned 0 lines.

The forwarding table moves to eBPF maps

The destination mapping moves to eBPF. eBPF lets you load programs into the kernel and run them at predefined hook points — on packet receive, on system call entry — without modifying the kernel itself or rebooting. The attached program can read and modify the data passing through that point.

eBPF programs are stateless across invocations — local variables don't persist between calls. State lives in maps instead: a key-value store the kernel maintains. Programs read and write there. Cilium writes the ClusterIP-to-Pod-IP mapping into these maps; on each connection, the attached program looks up the map to decide where to forward.

The rewrite happens at connect()

If only the storage location changed, you'd just learn a new command. But there's one more difference: the rewrite happens earlier.

The same connection in a Cilium environment no longer goes through netfilter at all, and the rewrite point shifts:

client Pod
   |  calls connect() with 10.96.89.117:80 (ClusterIP) as the destination
   v
eBPF program attached to connect()
   |  rewrites destination to 10.0.0.183:80 (Pod IP)
   v
Pod

Cilium attaches an eBPF program to connect(), the system call where an application tells the kernel what it wants to connect to. At this point, not a single byte of data has been sent. The destination address gets replaced with the Pod IP right here.

The rewrite is done before any packet is constructed, so the resulting packet carries the Pod IP as its destination — no ClusterIP packet is ever created. That's why tcpdump on a network interface never shows the ClusterIP.

This applies to connections from within the cluster. Packets arriving from outside the cluster on a NodePort are handled differently, as covered in a later section.

Two kinds of "no output"

At this point there are two separate "nothing shows up" situations with different causes. iptables-save returning 0 lines is because the forwarding rules moved from iptables to eBPF maps. tcpdump returning 0 packets when filtering on a ClusterIP is because there are no packets with that ClusterIP as a destination in that path. The causes are different, so the replacement commands are different. The next section handles the first one.

Reading the forwarding table

The test environment has a single ClusterIP Service named demo, backed by a Deployment with 2 replicas. Here's how to read the forwarding table for that Service in each environment.

kube-proxy

Take another look at those 13 lines from iptables-save | grep KUBE-SERVICES.

:KUBE-SERVICES - [0:0]
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
:KUBE-SERVICES - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES -d 10.96.111.7/32 -p tcp -m comment --comment "default/demo cluster IP" -m tcp --dport 80 -j KUBE-SVC-73JNOU5FOFXIWZLE
-A KUBE-SERVICES -d 10.96.182.1/32 -p tcp -m comment --comment "default/demo-nodeport cluster IP" -m tcp --dport 80 -j KUBE-SVC-IECF2FXKD7A6IGZV
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS

The first 6 lines are chain definitions and hook points — no forwarding destination information. Lines 7 onward list each Service's entry. -d is the ClusterIP; -m comment --comment contains default/<ServiceName> cluster IP. Find the Service you want by that comment. demo is line 8 — "default/demo cluster IP" — and its ClusterIP is 10.96.111.7.

The -j KUBE-SVC-73JNOU5FOFXIWZLE at the end of that line is the jump to the backend-selection chain. Grep for that chain name to get the backend rules:

iptables-save -t nat | grep KUBE-SVC-73JNOU5FOFXIWZLE

-A KUBE-SVC-73JNOU5FOFXIWZLE ! -s 10.244.0.0/16 -d 10.96.111.7/32 -p tcp -m comment --comment "default/demo cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.6:80" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-4BM33MODP4BKIE3G
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.7:80" -j KUBE-SEP-C7KHQYF2QRUY23U4

The KUBE-SVC chain selects the backend. The first line — ! -s 10.244.0.0/16 — handles masquerading for traffic coming from outside the Pod CIDR; it's not part of backend selection. Lines 2 and 3 are the actual selection: the comment shows default/demo -> <Pod IP>:80 and --probability sets the distribution ratio. The -j KUBE-SEP-xxx at the end is the jump to the chain that executes the DNAT:

iptables-save -t nat | grep -E "KUBE-SEP-4BM33MODP4BKIE3G|KUBE-SEP-C7KHQYF2QRUY23U4"

-A KUBE-SEP-4BM33MODP4BKIE3G -p tcp -m comment --comment "default/demo" -m tcp -j DNAT --to-destination 10.244.0.6:80
-A KUBE-SEP-C7KHQYF2QRUY23U4 -p tcp -m comment --comment "default/demo" -m tcp -j DNAT --to-destination 10.244.0.7:80

--to-destination is the backend Pod IP and port.

Summary of how to read the forwarding table in a kube-proxy environment:

ClusterIP: the -A KUBE-SERVICES -d line (confirm the Service name in the comment)
Backend Pod IP: --to-destination in the corresponding KUBE-SEP chain
Weight: --probability in the KUBE-SVC chain

Cilium

First, get the ClusterIP from kubectl get svc:

kubectl get svc demo

NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
demo   ClusterIP   10.96.24.124    <none>        80/TCP    4s

ClusterIP is 10.96.24.124. cilium-dbg service list -o json shows the frontend (ClusterIP), Service name, and backend Pod IPs all at once:

kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | [
    (.spec.id | tostring),
    (.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
    .spec.flags.type,
    (.spec.flags.namespace + "/" + .spec.flags.name),
    (.spec["backend-addresses"] // [] | map(.ip + ":" + (.port|tostring)) | join(" "))
  ] | @tsv' | column -t

1   10.96.0.1:443      ClusterIP  default/kubernetes       172.18.0.2:6443
5   10.96.0.10:53      ClusterIP  kube-system/kube-dns     10.0.0.88:53     10.0.0.192:53
12  10.96.24.124:80    ClusterIP  default/demo             10.0.0.40:80     10.0.0.8:80
2   0.0.0.0:30080      NodePort   default/demo-nodeport    10.0.0.113:80    10.0.0.195:80
...

The 10.96.24.124 from kubectl get svc matches ID 12. The last column shows the backends: 10.0.0.40:80 and 10.0.0.8:80. The three-hop walk through KUBE-SERVICES → KUBE-SVC → KUBE-SEP in kube-proxy fits in a single line here.

In a multi-node cluster, kubectl -n kube-system exec ds/cilium execs into whichever Pod in the DaemonSet gets scheduled. In a multi-node cluster you only see the local state of that agent. The same applies to hubble observe flows — traffic on other nodes won't appear. If you know which node your target Pod is on, check kubectl get pod -n kube-system -o wide | grep cilium for that node's cilium Pod name and exec into it explicitly with kubectl -n kube-system exec <PodName>.

Packet tracing

Comparing how far tcpdump gets you when chasing ClusterIP traffic in each environment.

kube-proxy

Running tcpdump on the node and filtering on the ClusterIP captured packets. DNAT happened in netfilter, so there was a stretch before the rewrite where packets actually carried the ClusterIP as their destination. That "bracket the DNAT with two tcpdump captures" technique only worked because of that structure.

kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "host 10.96.104.4"

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
11:38:19.980617 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [S], seq 3530419686, win 64240, options [mss 1460,sackOK,TS val 4270529724 ecr 0,nop,wscale 10], length 0
11:38:19.980665 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [S.], seq 2901899611, ack 3530419687, win 65160, options [mss 1460,sackOK,TS val 4170938121 ecr 4270529724,nop,wscale 10], length 0
11:38:19.980675 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1, win 63, options [nop,nop,TS val 4270529724 ecr 4170938121], length 0
11:38:19.980726 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [P.], seq 1:76, ack 1, win 63, options [nop,nop,TS val 4270529724 ecr 4170938121], length 75: HTTP: GET / HTTP/1.1
11:38:19.980741 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [.], ack 76, win 64, options [nop,nop,TS val 4170938121 ecr 4270529724], length 0
11:38:19.983523 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [P.], seq 1:1545, ack 76, win 64, options [nop,nop,TS val 4170938124 ecr 4270529724], length 1544: HTTP: HTTP/1.1 200 OK
11:38:19.983538 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1545, win 67, options [nop,nop,TS val 4170938124 ecr 4270529724], length 0
11:38:19.983641 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], seq 76, ack 1545, win 67, options [nop,nop,TS val 4270529727 ecr 4170938124], length 0
11:38:19.983883 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [F.], seq 1545, ack 77, win 64, options [nop,nop,TS val 4170938124 ecr 4270529727], length 0
11:38:19.983898 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1546, win 67, options [nop,nop,TS val 4270529727 ecr 4170938124], length 0

10 packets captured
10 packets received by filter
0 packets dropped by kernel

Packets to the ClusterIP (10.96.104.4) are captured. SYN through FIN, all there. DNAT happens in netfilter after this, so the packets with the ClusterIP destination exist in this stretch.

conntrack -L also shows the DNAT in a single entry:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- conntrack -L | grep "10.96.104.4"

tcp  6 118 TIME_WAIT src=10.244.0.5 dst=10.96.104.4 sport=54462 dport=80 src=10.244.0.6 dst=10.244.0.5 sport=80 dport=54462 [ASSURED] mark=0 use=1

dst=10.96.104.4 is the ClusterIP (outbound direction); src=10.244.0.6 is the backend Pod IP (return direction). Before and after the DNAT, on one line.

Cilium

Same experiment on the Cilium cluster. Run tcpdump while curling ClusterIP 10.96.89.117 from the client Pod:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "host 10.96.89.117"

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
0 packets captured
0 packets received by filter
0 packets dropped by kernel

The curl succeeds. Connectivity is there. The empty capture is not a filter mistake — there are simply no packets with that ClusterIP as their destination.

connect() rewrites the destination to a Pod IP before any packet is created, so whatever hits the network already has the Pod IP. The ClusterIP never reaches the network layer.

conntrack -L filtered on the ClusterIP also returns nothing. The conntrack table contains only Pod-IP entries, but since no packets with the ClusterIP pass through netfilter, there's nothing to match.

kubectl debug node/<node> -it --image=nicolaka/netshoot -- conntrack -L | grep "10.96.89.117"

No output.

Seeing the ClusterIP translation with Hubble

tcpdump can't show the ClusterIP. So how do you trace ClusterIP traffic in a Cilium environment? Hubble is Cilium's observability component — it surfaces eBPF events as flow records. The test repo's scripts/setup-cilium.sh installs with Hubble Relay enabled. Curl the ClusterIP from the client Pod and then run hubble observe:

kubectl -n kube-system exec ds/cilium -- hubble observe --last 30 | grep -E "client|demo"

--last 30 outputs the 30 most recent flows.

Sep  7 07:23:31.106: default/client (ID:52940) <> 10.96.89.117:80 (world) pre-xlate-fwd TRACED (TCP)
Sep  7 07:23:31.106: default/client (ID:52940) <> default/demo-864f5f87b9-rvr95:80 (ID:10947) post-xlate-fwd TRANSLATED (TCP)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: SYN)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Sep  7 07:23:31.110: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Sep  7 07:23:31.110: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Sep  7 07:23:31.110: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, FIN)

Focus on the first two lines. pre-xlate-fwd TRACED is the pre-translation flow, destination still recorded as 10.96.89.117:80 — the ClusterIP from kubectl get svc demo. post-xlate-fwd TRANSLATED is after, destination now the actual Pod demo-864f5f87b9-rvr95:80.

This confirms why tcpdump couldn't see the ClusterIP: the translation is done before the packet reaches the network. Hubble records both sides of the translation as eBPF events, so you can read "which ClusterIP became which Pod" directly.

NodePort is visible to tcpdump

Try NodePort on the same cluster and the result is different. Send a curl from outside the cluster to the NodePort (172.18.0.2:30080) while running tcpdump for that port on the node:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "port 30080"

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
07:23:24.736193 eth0  In  IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [S], seq 3175489113, win 64240, options [mss 1460,sackOK,TS val 194961069 ecr 0,nop,wscale 10], length 0
07:23:24.736306 eth0  Out IP 172.18.0.2.30080 > 172.18.0.1.52306: Flags [S.], seq 2271513508, ack 3175489114, win 64308, options [mss 1410,sackOK,TS val 855246694 ecr 194961069,nop,wscale 10], length 0
07:23:24.736333 eth0  In  IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [.], ack 1, win 63, options [nop,nop,TS val 194961069 ecr 855246694], length 0
07:23:24.736421 eth0  In  IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [P.], seq 1:80, ack 1, win 63, options [nop,nop,TS val 194961069 ecr 855246694], length 79
...
10 packets captured
10 packets received by filter
0 packets dropped by kernel

10 packets captured. The difference is the path. NodePort packets arrive from outside the cluster as real packets, so they're not subject to socket LB. Processing happens at the tc layer (TCX), after the NIC, which is why tcpdump sees them.

cilium-dbg status --verbose shows which layers are active:

kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg status --verbose | grep -E "Attach Mode|Socket LB|XDP Acceleration"

Attach Mode:            TCX
  Socket LB:              Enabled
  XDP Acceleration:       Disabled

Socket LB: Enabled means Cilium rewrites destinations at the socket layer (the connect() hook). ClusterIP traffic gets translated to a Pod IP there, before any packet is created — which is why tcpdump never sees it.

XDP Acceleration: Disabled means NIC-driver-level (XDP) processing is off. XDP requires a compatible NIC, so it's disabled in kind by default. Instead, as Attach Mode: TCX shows, BPF programs are attached at the tc (Traffic Control) layer. NodePort traffic is real packets from outside the cluster, not subject to socket LB, so it gets processed at that tc layer and passes through the NIC — making it visible to tcpdump.

Packet tracing summary

	kube-proxy	Cilium
tcpdump on ClusterIP	captures (packet exists before DNAT)	nothing (translated at connect())
conntrack on ClusterIP	captures (DNAT before/after on one line)	nothing (doesn't go through netfilter)
tcpdump on NodePort	captures	captures (processed at tc layer)
translation visibility	tcpdump before/after DNAT	Hubble pre-xlate-fwd / post-xlate-fwd

E2E debugging flow

Starting from "Service is unreachable," walk through the investigation steps for a Cilium environment. By the end of this section, you'll have a clear path from symptom to cause.

The test repo's manifests/broken/ has two failure patterns: svc-wrong-selector.yaml (a Service with a selector that matches no Pods) and netpol-deny.yaml (a NetworkPolicy that blocks the client). The full debug script is in tests/debug-cilium.sh.

Case A — selector mismatch

Apply manifests/broken/svc-wrong-selector.yaml. The demo-broken Service has selector app: demo-v2, but every Pod in the cluster has the label app: demo. Nothing matches.

Check the ClusterIP first:

kubectl get svc demo-broken

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
demo-broken   ClusterIP   10.96.171.11    <none>        80/TCP    4s

Send a curl from the client Pod:

kubectl exec -n default client -- \
  curl -s --max-time 5 -o /dev/null -w '%{http_code} (%{time_total}s)' http://10.96.171.11/

000 (0.000063s)

%{http_code} returning 000 means no HTTP response. %{time_total} shows it returned immediately. When a Service has zero backends, Cilium's eBPF program rejects the connection right away — same speed as kube-proxy's REJECT --reject-with icmp-port-unreachable.

Step 1 — check Endpoints

Same command as before Cilium:

kubectl get endpoints demo-broken

NAME          ENDPOINTS   AGE
demo-broken   <none>      8s

<none> means no Pods are registered as backends. Compare against a working Service:

kubectl get endpoints demo

NAME   ENDPOINTS                     AGE
demo   10.0.0.117:80,10.0.0.119:80   34s

kubectl get endpoints works exactly the same as in a kube-proxy environment (from 1.33, kubectl get endpointslices is preferred).

Step 2 — check Cilium's internal state

The JSON output includes the Service name:

kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | [
    (.spec.id | tostring),
    (.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
    .spec.flags.type,
    (.spec.flags.namespace + "/" + .spec.flags.name),
    (.spec["backend-addresses"] | length | tostring)
  ] | @tsv' | column -t

12  10.96.171.11:80   ClusterIP  default/demo-broken  0
8   10.96.249.121:80  ClusterIP  default/demo         2

The last column is backend count. demo (ID 8) has 2; demo-broken (ID 12) has 0. A backend count of 0 indicates endpoint shortage. Cilium registers the Service but has nowhere to forward connections.

In a kube-proxy environment, the same situation shows up in the filter table with a comment:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- iptables-save -t filter | grep "10.96.171.11"

-A KUBE-SERVICES -d 10.96.171.11/32 -p tcp -m comment --comment "default/demo-broken has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable

has no endpoints is right there in the comment. Run that same command on the Cilium cluster and you get nothing. For endpoint-shortage cases (as opposed to NetworkPolicy), iptables stops explaining itself. Use cilium-dbg service list -o json | jq backend count instead. 0 means no endpoints.

Root cause

The most common reason Endpoints is empty is a selector mismatch between the Service and its Pods. Compare them directly:

kubectl get svc demo-broken -o jsonpath='{.spec.selector}'

{"app":"demo-v2"}

kubectl get pod -l app=demo -o jsonpath='{range .items[*]}{.metadata.name}  {.metadata.labels}{"\n"}{end}'

demo-864f5f87b9-dccbz  {"app":"demo","pod-template-hash":"864f5f87b9"}
demo-864f5f87b9-zzngq  {"app":"demo","pod-template-hash":"864f5f87b9"}

The selector points to app: demo-v2; the Pods carry app: demo.

Case B — NetworkPolicy deny

Apply manifests/broken/netpol-deny.yaml. The deny-client-to-demo NetworkPolicy allows Ingress to Pods with app: demo only from Pods labeled app: not-client. The client Pod doesn't match, so all connections to demo are denied.

kubectl exec -n default client -- \
  curl -s --max-time 5 -o /dev/null -w '%{http_code} (%{time_total}s)' http://10.96.249.121/

000 (5.001378s)

Same symptoms as Case A — but %{time_total} is 5 seconds. Case A (no backends) returned in 0.000063 seconds. Here, --max-time was exhausted. Packets are reaching the destination; nothing is sending a response back. Endpoints tell a different story.

Step 1 — check Endpoints

kubectl get endpoints demo

NAME   ENDPOINTS                     AGE
demo   10.0.0.117:80,10.0.0.119:80   34s

Endpoints are healthy. That's the branch point from Case A. Empty Endpoints → check selector match and Pod readiness. Endpoints present → keep going.

Step 2 — confirm Cilium sees active backends

kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | select(.spec["frontend-address"].ip == "10.96.249.121") | [
    (.spec.id | tostring),
    (.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
    .spec.flags.type,
    (.spec.flags.namespace + "/" + .spec.flags.name),
    (.spec["backend-addresses"] | length | tostring)
  ] | @tsv' | column -t

8  10.96.249.121:80  ClusterIP  default/demo  2

Backend count is 2; Cilium knows about them. Verify the eBPF map itself agrees with what the agent reports:

kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list | grep "10.96.249.121"

10.96.249.121:80/TCP (0)    0.0.0.0:0 (8) (0) [ClusterIP, non-routable]
10.96.249.121:80/TCP (1)    10.0.0.52:80/TCP (8) (1)
10.96.249.121:80/TCP (2)    10.0.0.167:80/TCP (8) (2)

service list is the agent's view of what it configured. bpf lb list reads the eBPF maps directly. They usually agree; when they diverge, the agent reports healthy backends the datapath never received. Here they match — the forwarding table is correct. Something else is dropping the packets.

Step 3 — check Hubble for packet drops

Cilium enforces NetworkPolicy directly in eBPF, so nothing shows up in iptables. Use Hubble:

kubectl exec -n default client -- curl -s --max-time 5 http://10.96.249.121/ > /dev/null
kubectl -n kube-system exec ds/cilium -- hubble observe --verdict DROPPED --last 20 | grep -E "client|demo"

Sep  9 08:07:46.753: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Sep  9 08:07:46.753: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) Policy denied DROPPED (TCP Flags: SYN)
Sep  9 08:07:47.777: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Sep  9 08:07:47.777: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) Policy denied DROPPED (TCP Flags: SYN)

Policy denied DROPPED. A NetworkPolicy is blocking the connection. policy-verdict:none INGRESS DENIED is the same verdict from a different angle. Either of these in hubble observe --verdict DROPPED points to NetworkPolicy as the cause.

Root cause

kubectl get networkpolicy deny-client-to-demo -o jsonpath='{.spec.ingress}'

[{"from":[{"podSelector":{"matchLabels":{"app":"not-client"}}}]}]

Ingress is only allowed from Pods labeled app: not-client. The client Pod doesn't qualify.

Debug flow summary

The two cases point to this sequence:

Run kubectl get endpoints (or kubectl get endpointslices from 1.33). <none> → check selector match and Pod readiness.
If Endpoints exist, run cilium-dbg service list -o json | jq and check the backend count. 0 → Cilium doesn't see the backends.
If backends exist but there's no connectivity, run hubble observe --verdict DROPPED. Policy denied → NetworkPolicy is the cause.

Tool mapping from kube-proxy to Cilium:

Old tool (kube-proxy)	New tool (Cilium)	What it shows
`iptables-save -t nat \	grep KUBE-SERVICES`	`cilium-dbg service list -o json \
{% raw %}`iptables-save -t filter \	grep `	backend count from `cilium-dbg service list -o json \
{% raw %}`conntrack -L`	`cilium-dbg bpf ct list global`	Connection tracking table
`tcpdump`	`hubble observe`	Per-flow packet visibility (see "Packet tracing")
(no equivalent)	`hubble observe --verdict DROPPED`	NetworkPolicy denials

Wrap-up

iptables-save returning 0 lines was not a broken cluster. kube-proxy doesn't exist, so nothing writes forwarding rules there — they moved to eBPF maps.

To read the forwarding table, use cilium-dbg service list -o json | jq. One command gets you the ClusterIP, Service name, and backend Pod IPs. The three-hop KUBE-SERVICES → KUBE-SVC → KUBE-SEP walk collapses to one line.

Parts of the troubleshooting workflow carry over unchanged. kubectl get endpoints works fine in a Cilium environment. After that, check cilium-dbg service list -o json | jq for backend count, and if backends are present but connectivity still fails, check hubble observe --verdict DROPPED for NetworkPolicy denials. The has no endpoints comment that kube-proxy wrote in the filter table is gone, but a backend count of 0 means the same thing.

Appendix — iptables evaluation cost grows with endpoint count

Measured while scaling demo replicas from 1 to 16. The kube-proxy column shows the number of KUBE-SVC chain rules; the Cilium column shows LB map backend slots.

replicas   kube-proxy SVC rules   Cilium backend slots
1          2                      1
2          3                      2
4          5                      4
8          9                      8
16         17                     16

Both grow linearly with endpoint count. The difference is lookup cost. The table shows equal growth in entry count; the asymmetry is in how those entries are evaluated per new connection.

kube-proxy evaluates KUBE-SVC chain rules top to bottom to pick a backend. Each rule's --probability check either hits or falls through to the next. When the last endpoint is selected, every rule before it was evaluated first. With 16 replicas, up to 16 evaluations happen. This only applies to the first packet of a new connection — subsequent packets are handled by conntrack and skip the rules entirely. The impact shows up in environments with high new-connection rates and many Services.

-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.10:80" -m statistic --mode random --probability 0.06250000000 -j KUBE-SEP-...
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.11:80" -m statistic --mode random --probability 0.06666666688 -j KUBE-SEP-...
...
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.9:80"  -j KUBE-SEP-...

Cilium computes random() % N to get a slot number and does one map lookup to find the backend. Slot 0 is the service master entry; slots 1+ are backends. With 16 Pods, it's still one lookup. This slot format is cilium-dbg bpf lb list output — it reads the eBPF maps directly, not the agent's in-memory state.

kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list | grep "10.96.210.178"

10.96.210.178:80/TCP (0)    0.0.0.0:0 (8) (0) [ClusterIP, non-routable]
10.96.210.178:80/TCP (1)    10.0.0.207:80/TCP (8) (1)
10.96.210.178:80/TCP (2)    10.0.0.48:80/TCP (8) (2)
...
10.96.210.178:80/TCP (16)   10.0.0.187:80/TCP (8) (16)

Migrating from ingress-nginx to Gateway API: a three-category approach to annotations

Kazu — Tue, 08 Sep 2026 13:02:00 +0000

The Kubernetes project announced the retirement of ingress-nginx. ingress-nginx retires in March 2026. No more patches, no more updates. No successor was named — just a recommendation to migrate to Gateway API.

The reason is straightforward: not enough maintainers. For years, effectively one or two people had kept the project running in their spare time. After the retirement announcement, there were plans to develop a successor controller (InGate) in collaboration with the Gateway API community, but no contributors came forward, and InGate was retired as well. No successor was named because none could be named.

Existing deployments won't break overnight. Helm charts and container images remain usable. But security patches stop. When new vulnerabilities are found, they'll go unfixed. The system keeps running while the risk accumulates slowly.

ingress-nginx had a kind of self-evidence: "if you need HTTP in Kubernetes, use this." Gateway API dissolves that.

This article covers how to choose a Gateway API implementation and how to migrate ingress-nginx annotations by sorting them into three categories — with before/after manifests throughout.

What "migrate to Gateway API" actually means

Gateway API is a Kubernetes specification. It is not software.

In the Ingress era, HTTP routing configuration looked like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service
            port:
              number: 8080

Apply this YAML and it worked. Assuming ingress-nginx was installed in the cluster — but since "using Kubernetes Ingress" and "using ingress-nginx" were nearly synonymous, that distinction rarely came up. Spec and implementation mapped almost one-to-one.

Gateway API is different. Writing an HTTPRoute does nothing unless you have separate software installed that reads those resources and actually routes requests.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-route
spec:
  parentRefs:
  - name: my-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: my-service
      port: 8080

Apply this YAML without a Gateway controller installed in the cluster, and nothing happens. Gateway API is a spec defining what resources exist — it's not a process that handles traffic.

"Migrating to Gateway API" meant picking an implementation and installing it. The question of which software was actually running — obvious in the Ingress world — is now something you have to decide yourself.

Too many implementations

"Pick one" sounds simple, but the options are many. In the Envoy-based family alone: Envoy Gateway, kgateway, Istio, Contour, Cilium. Outside Envoy: Traefik, Kong, NGINX Gateway Fabric. Cloud providers each offer their own. And all of them claim "Gateway API conformant."

It's tempting to think: "if they're all conformant, they'll behave the same within the standard, and I can switch later." That doesn't hold.

Three reasons "conformant" isn't uniform

First, conformance levels vary. Gateway API features are divided into three tiers: Core, Extended, and Implementation-specific. Core is the minimum every implementation must support. Extended is "recommended but optional." Implementation-specific is the open space for vendor extensions. Claiming "Gateway API conformant" only requires Core support. Extended and above vary across implementations.

Second, the features you actually need in production tend to land in vendor extensions. Auth, rate limiting, circuit breaking — these commonly belong to Extended or Implementation-specific. Envoy Gateway uses a SecurityPolicy CRD; Istio uses AuthorizationPolicy. The moment you write one of those resources, you're locked into that implementation.

Third, there's no default. When ingress-nginx was around, there was a clear first choice. Now there isn't. Every team is being forced to make that choice simultaneously, without a settled answer.

The first pick carries weight. The comparison below shows why.

Comparing the major implementations

	Envoy Gateway	Istio	Traefik	NGINX Gateway Fabric
Data plane	Envoy	Envoy	Traefik	nginx
Service mesh support	Partial	Full	None	None
Migration ease from ingress-nginx	Good	Moderate	Good	Excellent
Extended feature coverage	Excellent	Excellent	Good	Good
Development momentum	Excellent	Excellent	Good	Moderate

Istio is the only one with full service mesh support — east-west traffic control and sidecar injection included. The other three have no mesh capabilities. NGINX Gateway Fabric scores highest on migration ease because it shares the same nginx data plane. Istio scores lower because installing the entire service mesh stack is a prerequisite. The difference in Extended feature coverage becomes concrete once you read the Category B before/after examples. For development momentum, Envoy Gateway (CNCF incubating) and Istio (CNCF graduated) lead, while NGINX Gateway Fabric has a lower release cadence.

No recommendation here — the right answer depends on whether you already have a service mesh, how you weigh the annotation rewrite cost, and whether your team has Envoy operational experience. But before picking an implementation, there's work to do: inventory your current ingress-nginx annotations.

Sorting annotations into three categories

The before/after manifests and validation scripts used in this article are at shinagawa-web/ingress-nginx-to-gateway-api. before/ holds the ingress-nginx manifests, after/category-a/ the standard Gateway API equivalents, and after/category-b-envoy-gateway/ and after/category-b-istio/ the implementation-specific ones.

Envoy Gateway and Istio were chosen for the Category B manifests because they have broad coverage of features that can't be expressed in the standard spec — external auth, rate limiting, and so on. Traefik and NGINX Gateway Fabric can handle basic routing just as well, but their vendor extensions are less mature. Putting Envoy-based (Envoy Gateway) next to service-mesh-capable (Istio) also makes the architectural difference visible. Links to the relevant files appear at the end of each category section so you can run them locally as you read.

The goal is to inventory every annotation you have and sort each into one of three categories. The approach differs fundamentally by category. Note that Category C is evaluated line by line within the annotation, not annotation by annotation. A single configuration-snippet can contain lines that fall into Category A and lines that require Category B.

Category A: migrates directly to standard Gateway API

rewrite

A rewrite is the process of rewriting the URL path before the request reaches the backend. If a client hits /api/hello and the app should receive /hello, that's a rewrite. In ingress-nginx, you'd use regex capture groups.

# before/02-rewrite.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rewrite
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: echo-v1
            port:
              number: 8080

The pattern /api(/|$)(.*) captures the second group ($2) and forwards it as the path. A request to /api/hello puts hello into $2, delivering /hello to the backend.

Gateway API expresses the same thing with the URLRewrite filter in HTTPRoute. (URLRewrite is an Extended-tier feature — not Core. Verify your chosen implementation supports it; the four implementations in this article's comparison table all do.)

# after/category-a/02-rewrite.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: rewrite
spec:
  parentRefs:
  - name: eg
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /
    backendRefs:
    - name: echo-v1
      port: 8080

Read it as: replace the /api prefix with /. No regex — prefix replacement is explicit.

canary

Canary deployment means routing a small percentage of traffic to a new version, then gradually increasing it rather than doing an immediate cutover. In ingress-nginx, this required two Ingress resources: a primary route to v1 and a canary route to v2, with the canary's weight set in an annotation.

# before/03-canary.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-primary
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo-v1
            port:
              number: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo-v2
            port:
              number: 8080

Two Ingress resources define the same host: app.example.com and path /. Normally that's a conflict, but ingress-nginx treats any Ingress with canary: "true" as a weighted companion to the other. canary-weight: "20" sends 20% of traffic to echo-v2.

One routing decision split across two resources is the problem. Reading canary-primary alone gives you no visibility into the actual traffic split.

Gateway API fits it into a single HTTPRoute.

# after/category-a/03-canary.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: canary
spec:
  parentRefs:
  - name: eg
  rules:
  - backendRefs:
    - name: echo-v1
      port: 8080
      weight: 80
    - name: echo-v2
      port: 8080
      weight: 20

List v1 and v2 in backendRefs with weight ratios. 80:20 sends 80% to v1, 20% to v2. One resource, full picture.

The conversions in this category can be automated with the Ingress2Gateway CLI (1.0 GA). Worth using to reduce manual work.

before/02-rewrite.yaml → after/category-a/02-rewrite.yaml, before/03-canary.yaml → after/category-a/03-canary.yaml

Category B: requires implementation-specific extensions

external auth

External auth delegates authentication decisions to a separate service rather than the application itself. On each incoming request, the proxy asks the auth service whether to allow it. Only when the auth service responds OK does the request proceed to the app. Because the auth logic lives outside the app, the same auth service can cover multiple apps.

In ingress-nginx, two annotation lines did it.

# before/04-auth.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: auth
  annotations:
    nginx.ingress.kubernetes.io/auth-url: http://auth-service.default.svc.cluster.local/auth
    nginx.ingress.kubernetes.io/auth-response-headers: X-Auth-User
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /protected
        pathType: Prefix
        backend:
          service:
            name: echo-v1
            port:
              number: 8080

auth-url is the auth service endpoint. auth-response-headers lists headers from the auth service response to forward to the backend. When the auth service returns 200, the specified headers are added to the upstream request. Two annotation lines cover delegation and response propagation.

The sample repository includes a mock auth service in app/auth-service.yaml — a Python server that returns 200 for Authorization: Bearer valid-token and 401 for everything else.

With Envoy Gateway, HTTPRoute alone has no place to write auth configuration. You create a separate SecurityPolicy CRD — an Envoy Gateway-specific resource — and bind it to the target HTTPRoute.

# after/category-b-envoy-gateway/04-auth.yaml (excerpt)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: auth
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: protected
  extAuth:
    http:
      backendRefs:
      - group: ""
        kind: Service
        name: auth-service
        port: 80
      headersToBackend:
      - x-auth-user

targetRefs specifies which HTTPRoute the policy applies to — here, the HTTPRoute named protected. extAuth.http.backendRefs points to the auth service.

The intent is clearer than in ingress-nginx. Annotations were scoped to their Ingress resource, so tracking which route had which auth policy meant reading every Ingress. SecurityPolicy stands as an independent resource with its target made explicit via targetRefs.

To forward X-Auth-User from the auth service response to the backend, headersToBackend must be configured explicitly. Unlike ingress-nginx's auth-response-headers, Envoy Gateway does not forward auth response headers automatically — only the headers listed in headersToBackend are propagated.

Migrating to Istio changes the architecture. AuthorizationPolicy evaluates request attributes locally — headers, JWT claims, source principals — rather than calling out to an external service.

# after/category-b-istio/04-auth.yaml (excerpt)
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: auth
spec:
  selector:
    matchLabels:
      app: echo
      version: v1
  action: DENY
  rules:
  - when:
    - key: request.headers[authorization]
      notValues:
      - "Bearer valid-token"

action: DENY means "deny requests that match the condition." Requests where request.headers[authorization] is not Bearer valid-token are denied — only valid-token requests get through. Note that selector.matchLabels scopes the policy to the workload, not to the /protected path — the DENY rule applies to all traffic reaching the matched pods. To restrict it to a specific path, add to.operation.paths to the rule. Where Envoy Gateway's extAuth asks an external service for an OK/NG decision, Istio's AuthorizationPolicy evaluates the policy inside the cluster. The mechanism is fundamentally different from ingress-nginx's auth-url callback model, so migrating to Istio means rethinking the auth design.

AuthorizationPolicy evaluates request attributes locally. For logic that requires an external decision — like checking permissions against a database — Istio provides AuthorizationPolicy with action: CUSTOM plus an extensionProviders entry in meshConfig. This is the proper external auth delegation path in Istio, comparable in ergonomics to Envoy Gateway's extAuth. The EnvoyFilter route can also be used but writes Envoy internals directly.

Category A works the same way regardless of which implementation you choose. Category B changes fundamentally depending on which one you picked. This is the "operational features drift into vendor extensions" reality the comparison table pointed to: you can't form a migration plan for this category until you've chosen an implementation.

before/04-auth.yaml → after/category-b-envoy-gateway/04-auth.yaml / after/category-b-istio/04-auth.yaml

rate limit

ingress-nginx's rate limit was self-contained in annotations, but there's a trap: queueing behavior changes after migration.

# before/05-rate-limit.yaml (excerpt)
annotations:
  nginx.ingress.kubernetes.io/limit-rps: "2"
  nginx.ingress.kubernetes.io/limit-burst-multiplier: "1"

limit-rps: "2" allows at most 2 requests per second per IP. Requests over the limit don't get rejected immediately — they queue up to the limit set by limit-burst-multiplier. With limit-burst-multiplier: "1", the queue cap is limit-rps × 1 = 2. Once the queue fills, subsequent requests get a 503 immediately.

Envoy Gateway uses BackendTrafficPolicy. Like SecurityPolicy for external auth, it's a separate resource from HTTPRoute, with targetRefs pointing at the target.

# after/category-b-envoy-gateway/05-rate-limit.yaml (excerpt)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: rate-limit
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: rate-limited
  rateLimit:
    type: Local
    local:
      rules:
      - limit:
          requests: 2
          unit: Second

requests: 2, unit: Second expresses 2 requests per second, but there's no field corresponding to limit-burst-multiplier. The queueing behavior can't be reproduced.

Istio configures the token bucket directly via EnvoyFilter. There's no abstraction layer like BackendTrafficPolicy — you're writing Envoy internals directly.

# after/category-b-istio/05-rate-limit.yaml (excerpt)
token_bucket:
  max_tokens: 2
  tokens_per_fill: 2
  fill_interval: 1s

tokens_per_fill: 2, fill_interval: 1s replenishes 2 tokens per second (= 2 req/s). max_tokens: 2 is the burst cap — the maximum number of tokens the bucket can hold. This corresponds to ingress-nginx's limit-rps: "2", limit-burst-multiplier: "1". The algorithm differs from nginx's queuing model, but the behavior — 2 requests per second, burst cap of 2 — is equivalent. Whether reproducing limit-burst-multiplier is worth trading against implementation choice is a judgment call that varies by team.

before/05-rate-limit.yaml → after/category-b-envoy-gateway/05-rate-limit.yaml / after/category-b-istio/05-rate-limit.yaml

Category C: no equivalent

configuration-snippet

Probably the most useful annotation ingress-nginx had — and the most problematic.

nginx config has server blocks and location blocks. configuration-snippet lets you inject nginx directives directly into the location block, overriding or extending what ingress-nginx generates. It was effectively an escape hatch for anything the standard annotations couldn't express.

# before/06-configuration-snippet.yaml (excerpt)
annotations:
  nginx.ingress.kubernetes.io/configuration-snippet: |
    more_set_headers "X-Custom-Header: my-value";
    more_set_headers "X-Request-ID: $request_id";

more_set_headers is a directive from the headers-more module bundled with ingress-nginx, used to add or override response headers. "X-Custom-Header: my-value" adds a static header. "X-Request-ID: $request_id" attaches nginx's built-in variable $request_id — a unique ID auto-generated per request — as a response header.

Gateway API has no direct equivalent. But saying "no equivalent" oversimplifies it. Break down the annotation line by line, and the picture splits.

Adding a static header like X-Custom-Header migrates to HTTPRoute's ResponseHeaderModifier filter. That's standard spec — Category A.

# after/category-a/06-response-header.yaml (excerpt)
filters:
- type: ResponseHeaderModifier
  responseHeaderModifier:
    add:
    - name: X-Custom-Header
      value: my-value

Dynamic values like $request_id are a different story. Nothing in the Gateway API standard covers referencing nginx built-in variables. With Envoy Gateway, you can reproduce it by injecting a Lua script via the EnvoyPatchPolicy CRD — Category B.

# after/category-b-envoy-gateway/06-request-id.yaml (excerpt)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: request-id-lua
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
  - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
    name: "default/eg/http"
    operation:
      op: add
      path: "/default_filter_chain/filters/0/typed_config/http_filters/0"
      value:
        name: envoy.filters.http.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          default_source_code:
            inline_string: |
              function envoy_on_request(request_handle)
                local req_id = request_handle:headers():get("x-request-id")
                if req_id then
                  request_handle:streamInfo():dynamicMetadata():set("lua_ns", "req_id", req_id)
                end
              end
              function envoy_on_response(response_handle)
                local meta = response_handle:streamInfo():dynamicMetadata():get("lua_ns")
                if meta and meta["req_id"] then
                  response_handle:headers():add("x-request-id", meta["req_id"])
                end
              end

Envoy generates an x-request-id header automatically on incoming requests. The Lua script reads it during request processing, stores it in metadata, then attaches it as a response header during response processing. nginx's $request_id becomes Envoy's request ID, but the intent — a unique per-request ID on the response header — is the same.

The same Lua logic works in Istio via EnvoyFilter, but the apply context and metadata namespace differ.

# after/category-b-istio/06-request-id.yaml (excerpt)
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: request-id-response-header
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.lua
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          default_source_code:
            inline_string: |
              function envoy_on_request(request_handle)
                local req_id = request_handle:headers():get("x-request-id")
                if req_id then
                  request_handle:streamInfo():dynamicMetadata():set("lua_metadata", "req_id", req_id)
                end
              end
              function envoy_on_response(response_handle)
                local meta = response_handle:streamInfo():dynamicMetadata():get("lua_metadata")
                if meta and meta["req_id"] then
                  response_handle:headers():add("x-request-id", meta["req_id"])
                end
              end

The Envoy Gateway version uses EnvoyPatchPolicy with JSONPatch to insert the Lua filter at the Listener level, with state stored under metadata namespace lua_ns. The Istio version uses EnvoyFilter with applyTo: HTTP_FILTER and match.context: GATEWAY, with state stored under lua_metadata. The Lua logic is the same, but the configuration layer differs.

What truly has no equivalent in Category C is nginx-specific functionality that EnvoyPatchPolicy can't reach — directives that depend deeply on nginx internals, or snippets that assume concepts Envoy doesn't have. In practice, most configuration-snippet contents, when broken down line by line, land in Category A or Category B. The annotation as a whole isn't Category C — each line has to be read and evaluated individually.

before/06-configuration-snippet.yaml → after/category-a/06-response-header.yaml (X-Custom-Header) / after/category-b-envoy-gateway/06-request-id.yaml (X-Request-ID, Envoy Gateway) / after/category-b-istio/06-request-id.yaml (X-Request-ID, Istio)

Standardization didn't make migration easier

Gateway API gave us a unified spec, so "the interface is now shared." But more implementations means more decisions. The Core tier is consistent across implementations. But operational features drift into vendor extensions, so locking in to an implementation comes after, not before.

The frustration with ingress-nginx — "settings are scattered across annotations, hard to see the structure" — was a real problem. Gateway API was designed to address it. But from the migration side, new decisions piled on: which software to pick, which annotations map to the standard and which need vendor extensions and which get dropped.

"Migrate to Gateway API" wasn't a work order. It compressed two separate jobs into one phrase: choose an implementation, then classify each annotation into one of three categories and handle each accordingly.

Second	Requests the upstream finished	200s returned by nginx	504s returned by nginx
9	167	167	0
10	167	167	0
11	162	162	0
12	166	166	0
13	168	52	139
14	164	0	200
15	165	0	200
16	167	0	200

Second	Requests the upstream finished	200s returned by nginx	504s returned by nginx
9	167	167	0
10	167	167	0
11	162	162	0
12	166	166	0
13	168	52	139
14	164	0	200
15	165	0	200
16	167	0	200

DEV Community: Kazu

Slow 200s in nginx, fast in the application server's logs

What is happening

The accept queue

SYNs are dropped when the queue is full

Retransmissions get through

Verification

Setup

Time to connect

Spotting it in your own logs

Access log

Application server log

Fix

How this varies by environment

With a shortened connect timeout

When the stop is not 3 seconds

Kernels older than 6.5

With a proxy in front of the application server

HPA does not keep the replicas you set with kubectl scale

Reproduction

Narrowing it down

What is happening

What a sync does

From the scale-up to the revert

How long the kubectl scale count lasts

Remedies

Raise minReplicas

Disable scale-down, then scale up

Which to choose

301 redirect change doesn't take effect for some users

Impact

Reproduction

Why v3 keeps showing

Fix

Use a 302

Add max-age to the 301

Which to choose

Effect of the fixes

With a 302

With max-age on the 301

Notes

Affected browsers

Users who still land on v3 after changing to 302

Setups that give the redirect a lifetime

Setups not tested

K8s upgrade removes cgroup paths and changes how you read the values

Three reads that change

Throttle time

Memory usage

OOM count

Why the container's oom_kill reads 0

Finding the Pod's slice

Reading the slice's memory.events

Docker exit code 137: telling the causes apart from the logs

Which log points to which cause

Using the table in production

The container has restarted

OOMKilled is false

What exit code 137 indicates

What was not checked

Appendix: each way of stopping and what it left

--memory 128m, app allocates 256MiB

Another container with no limit uses up the host's memory

docker stop -t 2

docker stop -t 2, with a restart policy

docker kill

docker kill, with a restart policy

kill -9 from the host to the app

The app returns 137 itself

--memory 128m, with a restart policy

nginx 504 Gateway Timeout with zero 429s from limit_req

Impact

What limit_req looks at

Conditions under which the upstream falls behind within the limit

Setup

Results

Reading the conditions

Why no 200s come back

Mechanism

In the logs

Raise `minReplicas`

`OOMKilled` is false

`--memory 128m`, app allocates 256MiB

`docker stop -t 2`

`docker stop -t 2`, with a restart policy

`docker kill`

`docker kill`, with a restart policy

`kill -9` from the host to the app

`--memory 128m`, with a restart policy

`--target` changes what `ps` shows

You can't use `--target` on a stopped container

Second	Requests the upstream finished	200s returned by nginx	504s returned by nginx
9	167	167	0
10	167	167	0
11	162	162	0
12	166	166	0
13	168	52	139
14	164	0	200
15	165	0	200
16	167	0	200