DEV Community: Kazu

High p99 latency in Node.js at low CPU usage: a blocked event loop

Kazu — Mon, 12 Oct 2026 13:04:00 +0000

The latency of a Node.js service suddenly gets worse. p99 (the 99th percentile of response time: when 1000 requests are sorted from fastest to slowest, the value of the 990th) jumps. CPU looks low when you check overall usage. Memory is normal too. Nothing points to a cause. Endpoints that respond immediately, like a health check, are slow in the same way.

Node.js runs its event loop on a single thread. When one piece of work blocks, every other request waits in the queue. This congestion rarely shows up on an overall CPU graph.

I reproduced event loop starvation on GitHub Actions (ubuntu-24.04, 4 vCPU) and show, with the code I ran, which metric catches the degradation that an overall CPU graph tends to miss. In the same environment I also measured how much an overload guard (load shedding: returning 503 immediately once event loop delay exceeds a threshold) improves p99, and what it does not improve.

shinagawa-web/show-your-work: nodejs-eventloop-starvation-p99

How Node.js handles requests

In an architecture that assigns a thread to each request, more concurrent connections mean more threads, and the cost of context switches (the overhead incurred each time the CPU switches from one thread to another) grows. Node.js avoids this by handling every request on an event loop that runs on one thread.

The event loop works through a to-do list (a queue) in order on a single thread. Each incoming request is added to the end of the queue, and the event loop takes items from the front and runs them. Work that has to wait for I/O (reading a file, network communication) leaves the queue while it waits and comes back when the I/O completes. In the meantime the loop can pick up the next item, so it can serve many connections without adding threads.

If one piece of work keeps hold of the thread, though, the rest of the queue has to wait until it finishes. This is event loop blocking. Reading a file with fs.readFileSync inside a request handler, running a shell command with execSync, and calling JSON.parse on a response several MB in size are all examples.

How far this blocking backs up the queue is called lag. setTimeout(fn, 100) is a reservation to call fn after 100ms, but while the event loop is blocked, the call to fn is held back too. If fn should have been called after 100ms and is called after 450ms, the lag is 350ms.

Why overall CPU looks flat

A CPU graph shows usage for the whole system. Because Node.js runs on a single thread, a blocking operation that consumes CPU uses only one core's worth. On a server with 4 vCPU (virtual CPUs, the unit in which cloud environments divide physical cores and allocate them), one core at 100% shows up as only 25% overall.

I checked this by measurement, with two endpoints. /sync-cpu occupies one core with a busy loop (a loop that keeps hold of the CPU until its completion condition is met) that repeats JSON.parse until a deadline, blocking the event loop for a specified time. /light returns a response immediately with no computation. Ten concurrent connections sent a mix of requests, 1/3 to /sync-cpu and 2/3 to /light. The blocking time (sync_ms) was set to 50ms, 100ms, 150ms, and 200ms, and each setting was measured for 30 seconds. The environment was GitHub Actions (ubuntu-24.04, 4 vCPU). The columns to watch in the table are cpu_pct (CPU usage) and lag_p99 (p99 of event loop lag).

sync_ms	/sync-cpu p99	/light p99	cpu_pct	lag_p99
0ms	4ms	4ms	26.7%	23ms
50ms	407ms	353ms	25.5%	306ms
100ms	712ms	1524ms	25.9%	612ms
150ms	1214ms	1610ms	25.6%	764ms
200ms	1222ms	2407ms	25.7%	1224ms

Results (CI run #37862107644)

cpu_pct stays almost fixed, between 25.5% and 26.7%. Quadrupling the blocking time from 50ms to 200ms barely moves CPU. lag_p99, on the other hand, is the event loop lag of the whole process, measured as a single value regardless of endpoint. The 23ms at 0ms is a floor that comes from the resolution: 20 setting; it is the baseline with no blocking. Lag rises with the blocking time: 306ms at 50ms, 612ms at 100ms, 1224ms at 200ms. CPU does not change, and only lag reflects the blocking.

At 200ms, the p99 of /light (2407ms) is higher than that of /sync-cpu (1222ms). /light has no blocking work of its own, but it sometimes waits a long time in the queue, and the larger the blocking time, the wider the spread of its p99.

Even when the overall CPU graph is flat, requests keep waiting in the queue if the event loop is congested. Looking only at overall CPU misses the congestion.

Spillover to unrelated endpoints

A Node.js process shares one event loop across all of its endpoints. While /sync-cpu is blocking the event loop, requests to /light wait in the queue too.

At 0ms, the p99 of /light is 4ms. At 50ms it jumps to 353ms. Nothing about /light itself changed. /sync-cpu occupied the event loop, and as a result the unrelated /light was kept waiting in the queue as well. The longer the blocking time, the worse the latency of /light gets.

Observing lag

Measuring event loop lag

Since Node.js v11.10.0, monitorEventLoopDelay() in node:perf_hooks gives you event loop lag.

With resolution: 20, Node.js repeatedly sets a timer that says "call me back in 20ms." Each time it is called back, it records how many ms it actually took against the planned 20ms. If the event loop is blocked by other work, this timer cannot get in either. With a 300ms block, a timer meant for 20ms is called only after 320ms. The difference, 300ms, is the lag for that sample. The samples accumulate in a histogram (a distribution recording how often each lag value occurs), and percentile(99) extracts the p99.

const { monitorEventLoopDelay } = require('node:perf_hooks');
const h = monitorEventLoopDelay({ resolution: 20 });
h.enable();

setInterval(() => {
  console.log('lag p99:', h.percentile(99) / 1e6, 'ms');
  h.reset();
}, 5000);

h.percentile(99) returns the p99 of the accumulated samples in nanoseconds, and / 1e6 converts it to milliseconds. h.reset() clears the samples and starts measurement for the next 5 seconds. Export this as a metric and you can catch abnormal lag even when the CPU graph is flat.

Per-core CPU

The blocking work in this experiment is a busy loop that repeats JSON.parse, and it keeps one core fully used. Overall CPU looks like around 25%, but a per-thread view with top -H or htop shows the Node.js main thread pinned at 100%, which confirms why the overall figure looked low.

When you run into the same symptom (low overall CPU, high p99), per-core CPU shows which thread the offending work is saturating.

Effect and limits of an overload guard

I implemented middleware that uses the measured lag to detect overload and return 503 (Service Unavailable, the HTTP status code that indicates the server temporarily cannot handle the request).

At startup it repeatedly sets a setTimeout with a 100ms interval. Each time the callback runs, it writes the difference between the planned 100ms and the actual elapsed ms into lagMs. If the event loop is blocked, this timer cannot get in either, so lagMs rises.

monitorEventLoopDelay can run at a 20ms interval without trouble because it is implemented in the C++ layer of Node.js (libuv), which keeps its impact on the JS thread to a minimum. A hand-written setTimeout loop, by contrast, creates a timer and runs a callback on the JS thread every time. The shorter the interval, the more of that overhead lands on the event loop itself, so the middleware's measurement uses 100ms.

let lagMs = 0
;(function measure() {
  const start = Date.now()
  setTimeout(() => {
    lagMs = Math.max(0, Date.now() - start - 100)
    measure()
  }, 100).unref()
})()

On each request, the middleware compares lagMs with the threshold. If lag exceeds the threshold (150ms), it returns 503 immediately with Retry-After: 1. If lag is at or below the threshold, it passes the request to the next handler.

if (lagMs > thresholdMs) {
  res.setHeader('Retry-After', '1')
  return res.status(503).json({ error: 'overloaded', lag_ms: lagMs })
}
next()

lagMs is a single global variable in the process. /sync-cpu and /light both read the same value. Even when /sync-cpu is the one pushing lag up, the request that gets the 503 once the threshold is exceeded is the next one to arrive: whichever request enters the queue first, regardless of endpoint.

I applied the same load with this guard in place (threshold 150ms, same run).

sync_ms	/sync-cpu p99	/sync-cpu 503 rate (%)	/light p99	/light 503 rate (%)	lag_p99
0ms	3ms	0.0%	3ms	0.0%	21.7ms
50ms	208ms	95.6%	203ms	95.4%	305.7ms
100ms	212ms	98.7%	202ms	98.7%	713.0ms
150ms	307ms	98.9%	162ms	99.0%	1072.2ms
200ms	401ms	99.0%	217ms	99.0%	1653.6ms

p99 here is computed over all requests, 503s included. Since most of them are immediate 503s, this p99 is not a measure of how slow the requests that got through were. At sync=100ms, the /sync-cpu p99 goes from 712ms to 212ms, and the /light p99 from 1524ms to 202ms.

Look at the 503 rate. With a 150ms threshold, at sync=100ms, 98.7% of requests to both endpoints get a 503. Of the 18793 requests that reached /sync-cpu, only 239 got through. As long as the blocking work continues, lag piles up far beyond the threshold, and the service stays stuck rejecting most requests. The overload guard alone does not make a working service.

With the guard in place, lag_p99 did not go down, and at sync_ms of 100ms or more it went up.

sync_ms	lag_p99 (no guard)	lag_p99 (with guard, threshold 150ms)
50ms	305.9ms	305.7ms
100ms	612.4ms	713.0ms
150ms	763.9ms	1072.2ms
200ms	1223.7ms	1653.6ms

The guard turns new requests away with 503. It cannot stop blocking work that is already running on the event loop. The time /sync-cpu spends blocking the event loop continues regardless of the guard. The middleware's check and the setTimeout callback for measuring lag also run on the event loop, so while /sync-cpu is blocking, they cannot get in either.

Once lag exceeds the threshold, the guard returns 503 to every request regardless of endpoint. Even when /sync-cpu is what pushes lag up, /light gets rejected the same way.

The guard improves what requests that get through experience, but it does not resolve event loop starvation. Lag staying high shows this.

Finding and fixing the blocking work

Wherever the 503 threshold is set, lag does not go down as long as the blocking work keeps running. The overload guard limits inflow, and it leaves the work that congests the event loop untouched.

Finding it

The fundamental fix is to identify the work that is blocking the event loop and fix it. Once you have confirmed the symptom in production, bring it to a staging or development environment where you can reproduce the same load pattern. Start the process with node --prof, apply similar load, and after stopping it, process the sampling log with node --prof-process. This identifies the functions that occupy CPU time.

Here is a profile taken in this test environment. It is the result of loading /sync-cpu?ms=200 with 10 concurrent connections for 20 seconds (same run).

Statistical profiling result (27870 ticks, 2 unaccounted, 0 excluded).

[Bottom up (heavy) profile]:
   ticks parent  name
  25652   92.0%  node
  25375   98.9%    node
  23368   92.1%      JS: *blockEventLoop  routes/sync-cpu.js:10
  23368  100.0%        JS: ^<anonymous>   routes/sync-cpu.js:17

84% of all CPU ticks (the number of times the profiler, sampling at a fixed interval, observed where the CPU was executing) are concentrated in blockEventLoop. Following the callers also shows that it is called from an Express route handler. Confirming that blocking exists with lag, then narrowing down the responsible function with --prof-process, is the efficient order.

Fixing it

How to fix it depends on the kind of work. For synchronous I/O like fs.readFileSync or execSync, replacing it with the asynchronous fs.readFile or exec frees the event loop.

For work that uses CPU for a long time, the design of doing it inside a request handler on the application server needs to be reconsidered. Heavy work such as image processing or video conversion does not need to be synchronous with the response. The application server receives the request, puts a job on a queue, and immediately returns "accepted." The work itself lives outside the application server, in a separate worker process, Lambda, Cloud Functions, and so on. Because it is decoupled from the application server's event loop, the application server does not get congested however heavy the work is. For work that has to happen on the request path, such as JSON.parse on MB-sized data, moving it to a separate thread with worker_threads is an option. Before that, reconsider whether the application server should be handling that work at all.

Running many Node.js application servers such as Express or Fastify as 1 vCPU containers, with a load balancer in front distributing requests, is a widely used setup. In this setup, when one container gets congested, the other containers keep accepting requests, so the single-thread constraint is absorbed operationally. The problem of one container's event loop getting congested does not go away, though. Requests assigned to that container stay queued inside it, and the same symptom shown in this article appears. Horizontal scaling solves the throughput problem; it does not resolve latency degradation inside one container.

Check whether the fix worked with lag_p99. Once the blocking work is removed, lag drops sharply.

Summary

The Node.js event loop handles every request on a single thread. When one piece of work blocks, every request in the process stops in the queue, and unrelated endpoints slow down uniformly.

An overall CPU graph does not show this congestion. On a 4 vCPU machine, one congested core shows up as only around 25% overall. Looking at CPU for this symptom will not find the cause.

What to check is lag. Export lag_p99 as a metric with monitorEventLoopDelay(), and you can detect blocking even when the CPU graph is flat. Looking at it alongside per-core CPU also shows which thread is congested.

An overload guard that returns 503 once lag exceeds a threshold improves response time, but lag itself does not go down, because the blocking work keeps running on the event loop. The guard buys time to find the root cause. Treat it as temporary protection until the blocking work is removed.

Buttons don't work for some users after a deploy

Kazu — Fri, 09 Oct 2026 13:03:00 +0000

After a deploy, the service that collects errors from users' browsers (Sentry or similar) starts showing this:

TypeError: Failed to fetch dynamically imported module: https://app.example.com/assets/feature-DHyZGV0s.js

The browser tried to load a JS file with import() at runtime and could not fetch it. feature-DHyZGV0s.js in the URL is a chunk: part of the app's functionality, split out into its own file by the build. DHyZGV0s is a hash the build derived from the file's contents.

Around the same time, some users report that "the button does nothing." Someone on the team opens the same page and clicks the same button, and it works. Nobody can reproduce it locally, and the errors keep coming in, not only right after the deploy but for hours afterward.

Impact

Users who lose a feature this way have no way to fix it themselves. They don't know what would make it work again, so all they can do is file a report. On the receiving end, the team spends time answering reports that don't reproduce internally and chasing their cause. This repeats with every deploy. In the browsers that throw the error, the deployed change is not running. A deploy meant to fix a bug does not reach those users. The side serving the app does not control which version runs in which user's browser. After every deploy, some users stay off the fixed version for as long as the previous version keeps running.

Reproduction

I built a minimal app with Vite. The build produces three files:

index.html: the HTML the browser fetches first when it opens the site
Entry JS: loaded by index.html, renders the page
Chunk: loaded with import() when the button is clicked

I built it twice with slightly different contents: a previous version, served first, and a new version that replaces it in the deploy. Since the contents differ, the hashes in the JS file names differ too. index.html is the only file without a hash.

Each version's files were baked into an nginx container image. To check whether this happens without any special configuration, nginx ran with the default config that ships in the image. A deploy means replacing the previous version's container with the new version's container. The new image does not include the previous version's JS, so after the deploy those files are gone from the server. The previous version started being served 10 minutes before the browser opened the site.

The browser is Chromium driven by Playwright. Each trial uses a fresh profile (the on-disk storage for history and other data that survives closing the browser), and runs these steps:

Open the site while it serves the previous version. Two cases: click the button here to fetch the chunk, or don't
Close the browser
Deploy the new version. This finishes within a few seconds of opening the site
Once a set time has passed since the first open, relaunch the browser with the same profile, open the site again, and click the button

Only two things are recorded: which version ran after reopening (the app displays its own version on the page), and whether the chunk loaded.

I tried reopening 30, 54, and 66 seconds after the first open.¹

Without clicking the button first

Each case ran 18 times in CI, with identical results every time.

First open to reopen	Version on reopen	Result of clicking the button
30.1 to 30.3 s	Previous	Error
54.1 to 54.2 s	Previous	Error
66.1 to 66.2 s	New	New version's feature shown

When the previous version ran, clicking the button failed to load the chunk with the following error, and nothing appeared on the page.

TypeError: Failed to fetch dynamically imported module: http://127.0.0.1:18000/assets/feature-DHyZGV0s.js

The site was opened after the deploy, yet the previous version ran, tried to load its own chunk, and failed. This is the same error as at the top, and the same "button does nothing" behavior.

With the button clicked first

On the first open, I clicked the button so the previous version's chunk was loaded, then closed the browser. Each case ran 9 times in CI, with identical results every time.

First open to reopen	Version on reopen	Result of clicking the button
30.1 to 30.2 s	Previous	Previous version's feature shown
54.1 to 54.2 s	Previous	Previous version's feature shown
66.1 to 66.4 s	New	New version's feature shown

The version on reopen was the same as without the click. The only difference was what happened when the button was clicked while the previous version was running: the already-loaded chunk ran as is, and not a single error occurred.

In both cases, for a browser that opened the site 10 minutes after the previous version started being served, reopening 54 seconds after the first open still ran the previous version, and reopening at 66 seconds ran the new one. The error appears only when the user had not clicked that button under the previous version. Only part of the users running the previous version show up in the error collector. The rest never appear there.

Narrowing it down

Access log

Look at the nginx access log around the time of the error. For a browser that ran the previous version and hit the error, the new version's server logged only this one line:

"GET /assets/feature-DHyZGV0s.js HTTP/1.1" 404 153 "http://127.0.0.1:18000/assets/index-DB4b__oJ.js"

The previous version's chunk is being requested by the same version's entry JS (index-DB4b__oJ.js). The new version's server doesn't have that file, so it returns 404.

After the deploy, the previous version's page is running. I checked the possible causes in order, starting from the server side.

Old and new mixed during the deploy

In a rolling deploy, some requests reach previous-version servers while the replacement is in progress. The reproduction replaces a single container, and that finishes within a few seconds of the first open. The request above reached the new version's server about 50 seconds after that. The previous version's page is running after the replacement has finished, so this is ruled out.

Server-side cache

Check whether the server is returning the previous version's index.html, including any proxy or cache in between if the setup has one. Look at the new version's access log per browser. From browsers that ran the new version, requests for index.html and the entry JS arrived, and both returned 200.

"GET / HTTP/1.1" 200 321 "-"
"GET /assets/index-D879X9_S.js HTTP/1.1" 200 2573 "http://127.0.0.1:18000/"
"GET /assets/feature-CpSIXz8z.js HTTP/1.1" 200 89 "http://127.0.0.1:18000/assets/index-D879X9_S.js"

From browsers that ran the previous version, no request arrived for index.html or for the entry JS. The chunk request was the only line. The server isn't returning the previous index.html; it isn't being asked for index.html at all. Ruled out.

Tabs left open

A tab left open since before the deploy can also request the previous version's chunk. In the reproduction, the browser was closed after the first open and relaunched after the deploy. The page was loaded after the deploy, so it is not a tab left open. Ruled out. To check this in your own environment, attach the page load time (performance.timeOrigin) when reporting errors and compare it with the deploy time.

Browser HTTP cache

What remains is the browser using an index.html it already had. Whether a page load came from the server or from the browser's own copy shows up in transferSize in Navigation Timing, which the browser records for each page load. Run this in the browser DevTools console to see the value:

performance.getEntriesByType('navigation')[0].transferSize

JS files have the same transferSize in Resource Timing, which you can list with:

performance.getEntriesByType('resource').map(r => [r.name, r.transferSize])

The value is positive when bytes came over the network and 0 when the response came from the browser's HTTP cache.² When the previous version ran, both index.html and the entry JS were 0. When the new version ran, index.html was 621 (larger than the byte count in the access log because it includes headers) and the entry JS was 2873. The previous version's page was rendered from the index.html and JS in the browser's HTTP cache. In the case where the button had been clicked first, the chunk was 0 too, and not a single request reached the new version's server.

The headers on the index.html received at first open had no Cache-Control, which specifies how long a response may be cached. nginx's default config doesn't add one. Instead, they had the time the response was sent (Date) and the time the file was last modified (Last-Modified).

Date: Tue, 29 Sep 2026 08:26:55 GMT
Last-Modified: Tue, 29 Sep 2026 08:16:53 GMT

Last-Modified is about 10 minutes before Date, matching the reproduction's setup of starting to serve the previous version 10 minutes before opening the site.

The browser is using an index.html with no specified cache lifetime without asking the server. But when reopening 66 seconds after the first open, it does go back to the server for index.html.

Relationship to time since modification

Next, I measured what determines how long the browser uses index.html without asking the server. Three things were varied:

How long index.html had gone unmodified at the moment it was fetched: the difference between Date and Last-Modified at fetch time, called time since modification below. "Opened 10 minutes after the previous version started being served" in the reproduction is a time since modification of 10 minutes. The 30, 54, and 66 second reopens correspond to 5%, 9%, and 11% of that
The time from the first open to the reopen
How the page was reopened: relaunching the browser and opening the site, or reloading the open page

Everything else matched the reproduction, and the button was not clicked on the first open. Each condition ran 4 times in CI, for 12 trials per data point.

Time since modification

I set the time since modification to 10 minutes, 100 minutes, 1000 minutes, and 6 days, and reopened at 5%, 9%, 11%, and 20% of each.³

index.html modified to first open	How long the previous version ran
10 min	About 1 min
100 min	About 10 min
1000 min (about 17 h)	About 100 min
6 days	About 14 h

Each was about 10% of the time from modification to first open. With 100 minutes, for example, reopening after 9 minutes ran the previous version, and after 11 minutes the new one. With 6 days, reopening after 13 hours still ran the previous version, and clicking the button produced the error.

Reload

With a time since modification of 100 minutes, I relaunched the browser, opened the site, and reloaded the page before clicking the button. In all 24 trials where the previous version was running when the site opened, the reload switched to the new version, and clicking the button showed the new version's feature. The reload fetched index.html from the server again.

What is happening

For responses without Cache-Control, the HTTP caching spec lets a cache guess how long it may use the response without checking, and gives 10% of the time since Last-Modified as a typical value.⁴ Chromium uses exactly this, with no upper limit.⁵

When build artifacts are baked into an image, Last-Modified is the build time. On a site that deploys once a week, a browser that opened the site just before the next deploy uses index.html and the entry JS without checking for about 17 hours, 10% of 7 days.

If a deploy happens in that window, reopening the site still runs the previous version's page. Its chunks are gone from the server, so using a feature that hasn't been loaded yet returns 404. After about 17 hours, index.html is fetched again and the new version takes over.

Users who opened the site earlier in the week had a shorter time since modification, so the previous version lingers for less time after the deploy. With weekly deploys, the errors last at most about 17 hours. Browsers that open the site for the first time after the deploy, or that reload, get the new version, so checking internally may not reproduce it.

What decides it

Which version runs in the browser is decided not by when you deployed, but by when the HTML the browser holds gets fetched again.

Hashed files change name whenever their contents change. Which version's files get loaded is decided entirely by the HTML, whose name never changes. If you don't specify a cache lifetime for that HTML, the browser decides when to fetch it again. When a deployed version reaches users is then out of the serving side's hands.

Fixes

Set a cache lifetime for the HTML

Add no-cache to index.html

With Cache-Control: no-cache on index.html, the browser checks with the server every time it opens the page whether index.html has changed.⁶ If it hasn't, the server returns 304 and sends no body. Opening the site after a deploy gets the new version's index.html.

In the reproduction's nginx, I added this:

location = /index.html {
    root   /usr/share/nginx/html;
    add_header Cache-Control "no-cache";
}

The cost is one extra round trip to the server each time the page opens. This fits when you want deployed versions to reach users right away.

Add max-age to index.html

With Cache-Control: max-age=<seconds>, the browser uses index.html without checking for that many seconds, regardless of time since modification.⁷ The window in which the previous version runs doesn't go away. You decide its length yourself instead of letting the browser guess.

If the seconds you specify are longer than what the browser would have guessed, the previous version actually runs longer. This fits when you want to avoid a round trip on every open but still want a ceiling on how long the previous version can run.

Reload when loading fails

Catch the chunk load failure and reload the page. A reload fetches index.html again, so it switches to the new version. In Vite, a load failure is signaled with the vite:preloadError event.⁸ In the reproduction, I reloaded only once, like this:

window.addEventListener('vite:preloadError', () => {
  if (!sessionStorage.getItem('reloaded-once')) {
    sessionStorage.setItem('reloaded-once', '1');
    window.location.reload();
  }
});

The cost is that in-progress page state is lost on reload, and you need a guard so that it doesn't reload repeatedly if the new version also fails. The code above never clears its flag, so if the same tab fails again after the next deploy, it won't reload. The previous version still runs, and this does nothing for users who are running it without hitting an error. Don't use it alone. Combine it with no-cache or max-age, and use it to move users who still hit the error onto the new version.

Effect of the fixes

Using the same setup as the reproduction, I opened an index.html with a time since modification of 100 minutes, varied the time until reopening, and compared against no fix. Each condition ran 4 times in CI, for 12 trials per data point.

no-cache

Reopen after	No fix	no-cache
5 min	Previous (error)	New
9 min	Previous (error)	New
11 min	New	New
20 min	New	New

The new version ran at every point measured, and the window where the previous version runs disappeared. When reopening without a deploy, there was one request for index.html each time, which returned 304 with no body.

max-age

I added max-age=300 (5 minutes).

Setup	How long the previous version ran
No fix	About 10 min (previous at 9 min, new at 11 min)
max-age=300	About 5 min (previous at 4 min, new at 6 min)

The window where the previous version runs shrank from about 10 minutes to about 5. Reopening at 9 and 11 minutes also ran the new version. With a time since modification of 1000 minutes, the result was the same: previous at 4 minutes, new at 6. The window is the length you specify, regardless of time since modification.

Reload on failure

Reopen after	No fix	Reload on failure
5 min	Previous (error)	Previous, new after reload
9 min	Previous (error)	Previous, new after reload
11 min	New	New
20 min	New	New

The previous version ran for about 10 minutes, the same as with no fix. Clicking the button under the previous version failed to load the chunk, triggered one reload, and then showed the new version's feature.

Scope

These results assume that index.html has Last-Modified but no Cache-Control, that hashed files are removed by the deploy, and that the browser is Chromium. Change any of these and the length of the previous-version window or the way errors show up changes. Of the variations below, only two were measured: the setup that keeps the previous version's JS, and the setup that doesn't return 404. The rest are read from source code and how the mechanism works.

Browsers

Only Chromium headless shell 153.0.8010.12 was measured.

According to their source, Firefox and Safari also use index.html without checking for 10% of the time since Last-Modified, the same as Chromium.⁹ With weekly deploys, the maximum is about 17 hours in every browser.

Firefox is the only one that differs from Chromium: it caps this window at one week. The cap kicks in only when the time since modification exceeds 70 days, so if you deploy more often than every 70 days, there is no difference.

In the reproduction, the browser was closed and relaunched for each reopen. Opening the site in a new tab with the browser still running was not measured.

When the previous version's JS is kept

I included the previous version's JS in the new image as well and measured with a time since modification of 100 minutes. The previous version ran for about 10 minutes, the same as with no fix, and clicking the button during that time returned the previous version's chunk with 200 and showed the previous version's feature. Not a single error occurred.

With no errors, it is harder to notice that the previous version is running. The previous version's page keeps calling the newly deployed server-side code, so that code has to stay compatible with it. A setup that puts build artifacts in S3 and runs aws s3 sync without --delete is in this state, because the previous version's files are never removed.

When missing paths don't return 404

In the main results, the request for the previous version's chunk got a 404. With nginx configured to return index.html for missing paths (try_files $uri /index.html), the same request gets a 200 whose body is index.html. Searching the access log for 404 finds nothing. Instead, look for lines where a hashed file path returned 200 with 321 bytes, the same size as index.html.

"GET /assets/feature-DHyZGV0s.js HTTP/1.1" 200 321 "http://127.0.0.1:18024/assets/index-DB4b__oJ.js"

The message that reached the error collector was the same Failed to fetch dynamically imported module as with the 404. The browser console also showed this error:

Failed to load module script: Expected a JavaScript-or-Wasm module script but the server responded with a MIME type of "text/html". Strict MIME type checking is enforced for module scripts per HTML spec.

Other ways of serving

Whether you serve from S3, a CDN, or a hosting service, or render the HTML on the server per request, you can tell whether this applies by looking at the response headers for index.html.

curl -sI https://app.example.com/ | grep -iE '^(cache-control|expires|last-modified|age):'

If there is neither Cache-Control nor Expires, and only Last-Modified, the same thing as in this article will happen. If there is an Age header, the response came from a CDN in between that was holding it.

When file timestamps are pinned

If the build pins file modification times to a fixed past time (SOURCE_DATE_EPOCH and similar), Last-Modified is that time, not the build time. The time since modification is then not the deploy interval but however long it has been since that fixed time. Chromium has no upper limit, so the further back the fixed time, the longer index.html is used without checking, potentially for years.

With a Service Worker

If a Service Worker serves index.html from its own cache, the same symptom appears. In that case, adding no-cache or max-age to index.html has no effect. To check whether one is registered, run navigator.serviceWorker.getRegistrations() in the console.

Built with webpack

The same failure surfaces as ChunkLoadError. The message starts with Loading chunk <chunkId> failed.¹⁰

Appendix: how the times were produced

The time since modification was produced without waiting. nginx returns a file's modification time as Last-Modified, so when starting the container, I shifted the modification time of the files on the server into the past.

The time until reopening was actually waited out in the reproduction, and produced without waiting in the conditions that varied the time since modification. When the previous version's server adds an Age: <seconds> header to index.html, the browser treats the response as already that many seconds old at the moment it receives it.¹¹ The reopen happened right after the fetch, and the Age value plus the few seconds that actually passed was recorded as the time until reopening.

I confirmed that this method doesn't change the results by comparing it with actually waiting, at a time since modification of 10 minutes. With both methods, the switch to the new version happened once the time until reopening exceeded 10% of the time since modification.

Changing when the new version is deployed (right after the first open, halfway to the reopen, or right before the reopen) gave the same results. This was confirmed by actually waiting, at a time since modification of 10 minutes.

The setup was Chromium headless shell 153.0.8010.12 (the build without a window), Playwright 1.63.0, nginx 1.30.5, and Vite 8.3.1. The reproduction setup and scripts are in show-your-work. ↩
The definition of transferSize in W3C Resource Timing. ↩
9% of 6 days is over 13 hours. None of the times in this section were actually waited out; the appendix at the end explains how they were produced. ↩
RFC 9111, section 4.2.2. ↩
The line lifetimes.freshness = (date_value - last_modified_value.value()) / 10; in Chromium 153.0.8010.12's GetFreshnessLifetimes. ↩
RFC 9111, section 5.2.2.4. ↩
RFC 9111, section 5.2.2.1. ↩
Load error handling in the Vite docs. A Vite maintainer has commented that this failure is not specific to Vite, also occurs with webpack, and has no proposed fix on the bundler side (vitejs/vite#11804). ↩
Firefox: nsHttpResponseHead.cpp. Safari: WebKit's CacheValidation.cpp. ↩
webpack v5.111.1 source. ↩
RFC 9111, section 4.2.3. ↩

High p99 latency at low CPU usage is caused by CPU throttling

Kazu — Tue, 06 Oct 2026 13:01:00 +0000

Every metric on the dashboard is within its normal range. CPU usage is 44%, memory has headroom, and the error rate is zero. p50 (the value in the exact middle when response times are sorted from shortest to longest) is 21ms, the same as usual.

In that state, p99 (the value at the boundary with the slowest 1% when response times are sorted) goes up and stays up. It is normally 31ms, and now it is 139ms.

The slow requests are not concentrated in any particular time of day. A single request to the same endpoint returns quickly.

At the time this was found, it was not clear under what conditions requests became slow.

I reproduced this state with Docker and measured why only p99 gets slow.

Impact

The first impact is slower responses for users. At least 1 in 100 requests waits 139ms or more for a response that would normally come back in 31ms. Since every metric on the dashboard is within its normal range, no alert fires, and nobody notices that slow responses keep happening.

The second impact is cost. Raising the CPU limit to bring p99 back means fewer containers fit on one host, so more hosts are needed. And because usage is 44%, usage cannot tell you how far to raise the limit. Too little, and requests stay slow. Too much, and the extra becomes idle host capacity.

Reproduction

I reproduced the state above with Docker.¹

The server under test is an HTTP server that spends 20ms of CPU time per request and returns an empty 200. Four workers each accept and handle connections. There is no queue inside the application; while all workers are busy, connections wait in the kernel's accept queue. This is so the queue length can be read from outside the server with ss. The container has a CPU limit set with --cpus 0.5.

Load was applied from outside the container for 60 seconds. Each request opened its own connection, at an average of 10 per second, with random intervals between arrivals (Poisson arrivals). This models many users sending requests independently.

At 20ms per request and 10 requests per second, CPU usage works out to 40% of the limit. Latency was measured on the load generator side, from send to receive.

Item	Value
CPU usage (60-second average)	43.5%
p50	20.6ms
p99	138.6ms
Errors	0 of 609

This matches the dashboard described at the start.²

Narrowing it down

I narrowed down the source of the delay by ruling out candidates one at a time.

The accept queue

A slow request was either delayed while waiting for a worker to become free, or delayed after processing started. If there are not enough workers, connections queue in the listen socket's accept queue. Its length can be read from Recv-Q in ss -lnt.

Over the 60 seconds, Recv-Q peaked at 3 and averaged 0.018. At the moments the 7 slow requests at or above p99 were sent, it was 0 to 2. Requests were not queuing for busy workers. This rules out a worker shortage.

Downstream calls

A delay after processing starts means either waiting on something downstream (a DB or an external call) or not getting CPU. If it is waiting downstream, long entries show up in APM spans or DB-side records at the same time as the slow requests. The server under test makes no downstream calls, so this check is listed only as a step to perform on a real system, and I moved on to the CPU side.

CPU usage

The dashboard shows 44% CPU usage, so CPU looks sufficient. But that is a 60-second average, and consumption concentrated in short bursts gets buried in the average. So I collected usage again as 1-second averages.

Even at 1-second averages, usage peaked at 73.5% of the limit, and 0 of the 60 seconds were at 90% or above. Shrinking the aggregation interval to 1 second still does not show any moment where CPU runs short. Instead of making the average finer, I looked at the records of hitting the limit directly.

`cpu.stat`

When a container hits its CPU limit, it is stopped (throttled). The number of times it was stopped accumulates as nr_throttled in the container cgroup's cpu.stat.³

Over the 60 seconds, nr_throttled increased by 65. The limit is enforced per 100ms period, and there were 531 periods in the 60 seconds (nr_periods in cpu.stat). In 12% of them, roughly once per second, the container hit the limit and was stopped. If nr_throttled is not increasing, suspect CPU contention with other processes on the host.

Usage is only 44% averaged over 60 seconds. The records so far do not explain why it still hits the limit.⁴

Conditions for hitting the limit

--cpus 0.5 is a limit that allows using up to half a core. The kernel enforces it as up to 50ms of CPU time per 100ms period.⁵ Even with 44% usage averaged over 60 seconds, arrivals are random, so the CPU time used in each period varies.

Two possible reasons for hitting the limit: usage is high to begin with, or the 4 workers use CPU at the same time. Starting from the reproduction setup, I changed one condition at a time, ran each for 60 seconds, and compared the fraction of periods that hit the limit (nr_throttled / nr_periods, called the throttle rate below).

Usage

With 4 workers, I changed arrivals to 5 and 20 per second.

Arrivals per second	CPU usage (60-second average)	Throttle rate
5	20.5%	2.8%
10 (same as reproduction)	43.5%	12.2%
20	87.1%	70.1%

The throttle rate rises with usage. But even at 20% usage it does not reach zero; the limit is hit about 3 times per 100 periods. Usage alone does not determine it.

Number of workers

With arrivals at 10 per second, I reduced the workers to 1.

Workers	Arrivals per second	CPU usage (60-second average)	Throttle rate
4 (same as reproduction)	10	43.5%	12.2%
1	10	42.9%	13.1%

The throttle rate barely changed.

The kernel checks the total CPU time used within the period, not how many workers used it. Even when one worker handles requests in sequence, 3 arrivals in one period use 60ms, which exceeds 50ms.

CPU time per period

From these two results, whether the limit is hit appears to depend on how much CPU time is needed within a period. I split the 60 seconds of the reproduction (4 workers) into 100ms periods aligned to the times the kernel switches periods, and counted the CPU time needed in each. The CPU time needed is the sum of processing for requests arriving in that period and processing that spilled over unfinished from the previous period.

Periods needing less than 40ms never hit the limit. Periods needing 40 to 50ms hit it in 4 of 78, periods needing 50 to 60ms in 16 of 17, and periods needing 60ms or more in all 45.

Restricting to periods with no spillover from the previous period, the split follows the number of arrivals. Periods with 0 to 2 arrivals hit the limit in 0 of 410, and periods with 3 or more arrivals in 29 of 34. In the periods with 3 or more arrivals that did not hit the limit, the last arrival came 93ms or later after the period started, and its processing spilled into the next period.

Of the 65 periods that hit the limit, 36 carried spillover from the previous period. Even with low average usage, random arrivals sometimes overlap, and processing that overlaps and does not finish is carried into the next period.

Both the overlap and the carryover happen on a 100ms scale, so they do not show up as usage in either the 60-second or the 1-second average. Even at 100ms granularity, what you see depends on where the window falls.

When usage is cut into 100ms windows aligned to period boundaries, all 65 periods that hit the limit used 45ms or more. In the 1-worker run, the 100ms windows used to collect usage were offset by 74ms from the period boundaries. Each window straddled two periods, and of the 70 periods that hit the limit, only 26 showed 45ms or more.

What is happening

To see how much a request in a throttled period is delayed, I followed a single request from arrival to response.

Waiting before accept

When a connection arrives, the kernel completes the TCP handshake. Even while the container is stopped by the limit, the connection is established and placed in the accept queue.

A queued connection is accepted by a free worker, which starts processing it. This request needs 20ms of CPU time. But while the container is stopped, the free workers are stopped too and cannot accept. Even at the head of the queue, the connection waits until the next period starts.

The period's 50ms

The container is allotted a budget of 50ms of CPU time per 100ms period. This 50ms is drawn down by the total CPU time used by all workers in the container. If N workers run at the same time on separate cores, it drops by N ms per 1ms of elapsed time.

The rate of drawdown depends on the number of workers, but the total CPU time demanded in that period does not. The total decides whether the budget runs out. If 3 requests arrive in one period, the demand is 60ms, which exceeds 50ms. With 4 workers, 3 run at the same time and use it up about 17ms into the period. With 1 worker, requests are processed one at a time and it is used up at 50ms. Either way it runs out.

Whether a given request is stopped depends on whether the period's 50ms runs out before the request finishes its own 20ms. What matters is how much the requests that arrived earlier in the same period have already used. If it does not run out, the request returns in a little over 20ms.

The whole container stops

When the 50ms is used up, all threads in the container are stopped.⁶ Requests being processed stop mid-processing, and connections not yet accepted wait unaccepted.

At this point nr_throttled in cpu.stat goes up by 1. The stopped time accumulates in throttled_usec in the same file, but what gets added is not the time for one container stop. It is the time per stopped CPU.⁷ If threads running on 3 CPUs are stopped for 30ms, throttled_usec goes up by 90ms.

So dividing throttled_usec by nr_throttled does not give the length of one stop. In the 60-second reproduction, this division gave 75.0ms, but the length of each stop measured individually with bpftrace averaged 39.3ms. Summing the stopped time per stopped CPU gives 4960ms, which nearly matches the 4873ms increase in throttled_usec.

While stopped, no request in the container makes progress, whatever stage it is in. Its response is delayed by the time remaining in the period.

What the number of workers changes is when the budget runs out and how long the stop lasts from there. In the example of 3 arrivals in one period, 4 workers stop for about 83ms and 1 worker for 50ms. However, as shown below in "Tracing each request", every slow request at or above p99 was stopped twice with either number of workers.

When the next period starts, the 50ms is restored. Waiting connections are accepted, and requests that were mid-processing continue. If the budget runs out again before they finish, they stop again.

Tracing each request

In the reproduction setup, I traced each request to the thread that handled it. I matched the load generator's source port to the peer port of the connection the server accepted, and used bpftrace to record which threads were running at the moment of throttling. The length of each stop was measured as the time from the kernel's throttle_cfs_rq to unthrottle_cfs_rq.⁸

A single stop lasted at most 84.4ms, with a median of 34.5ms, and all 65 were released when the next period started.

All 7 slow requests at or above p99 were throttled twice before returning a response. Two stops on top of 20ms of processing give 139 to 192ms. The stages at which the two stops occurred were as follows.

Stage stopped	4 workers (same as reproduction)	1 worker
Once before accept, once during processing	6 of 7	6 of 7
Twice during processing	1 of 7	0
Twice before accept	0	1 of 7

With 4 workers, the 6 requests stopped before accept waited 32 to 69ms to be accepted. Every stop during processing happened while that request's thread was running.

When these 7 were sent, Recv-Q was 0 to 2. That is the same value as when the worker shortage was ruled out in "The accept queue", so the workers were not full. Even with a short queue, if the accepting side is stopped, the connection waits at the head of the queue.

Mapping to the symptoms

Only some requests are slow because identical requests to the same endpoint split by arrival timing: some finish before the period's 50ms runs out, and the rest are stopped when it does.

The slow requests are not concentrated in any time of day because the only thing deciding the split is arrival timing; the time of day itself plays no part.

A single request is fast because no other worker is running and the period's 50ms has not been drawn down. 20ms always fits within the period's 50ms.

CPU usage stays at 44% because how much of the period's 50ms was used shows up in usage, but the time spent unable to proceed after it ran out does not.

What decides it

For a resource with a fixed amount available per period, whether it runs out is not decided by average usage alone, but by how much is used within one period. Here the period is 100ms, the available amount is 50ms, and the amount used within a period was the sum of arrivals overlapping in that period and processing spilled over from the previous period. To reduce how often it runs out, change how it is used within the period, or change how much is available per period.

When a resource is handed out in short periods and observed as averages over longer intervals, the averages do not show that the allotment ran out. Here, against the 100ms period, the dashboard used 60-second averages, and even re-collected data used 1-second averages; neither showed the periods that ran out.

So a low average is not evidence of headroom. Even if the aggregation interval is shrunk to the allotment period, windows not aligned to period boundaries show only some of the periods that ran out. The reliable check is to count the number of times it ran out. For a CPU limit, that count is nr_throttled in cpu.stat.

Remedies

The cause was the combination of the CPU limit and arrivals overlapping within one period, and it did not show up in usage. There are two places to act: change how much is available per period (raise the CPU limit), and make it possible to count how many times it ran out.

Raise the CPU limit

This increases the amount available per period. Unless it is raised enough to cover the arrivals that overlap within one period, throttling remains.

This fits when the host has spare capacity and p99 takes priority over cost. It is an option to consider when the application cannot be changed.

Add `nr_throttled` and `nr_periods` to monitoring

This does not fix p99. The next time the same symptom appears, the monitoring screen shows that the limit is being hit, without re-collecting usage as 60-second and then 1-second averages.

Both are cumulative counters, so taking the difference at your current collection interval gives the number of times the limit was hit and the number of periods in that interval.

There are two costs. The location of cpu.stat differs by environment, so collection needs per-environment configuration. And as shown by the 2.8% throttle rate at 20% usage, the throttle rate is not necessarily zero in normal operation, so choosing an alert threshold takes work.

Effect of the remedy

Keeping the reproduction setup (4 workers, 10 per second), I changed only the CPU limit to 0.5 / 0.75 / 1.0 / 1.5 / 2.0 and ran each for 60 seconds. 0.5 is the reproduction condition. The limit number is how many cores' worth may be used. 2.0 is 2 cores' worth, half of the host's 4 cores.

CPU limit	CPU usage (fraction of limit)	Throttle rate	p50	p99
0.5	43.5%	12.2%	20.6ms	138.6ms
0.75	28.1%	2.1%	20.4ms	44.5ms
1.0	21.1%	0.38%	20.4ms	33.2ms
1.5	14.5%	0%	20.5ms	31.1ms
2.0	10.5%	0%	20.4ms	33.0ms

The higher the limit, the lower the throttle rate. At 1.0 it was 0.38% (2 of 522 periods), and at 1.5 and above it was 0.

p99 is 31 to 33ms at 1.0 and above, and raising the limit further barely changes it. The "usual p99" of 31ms from the start falls in this range.

The CPU the container actually used averaged 0.21 to 0.22 cores in every condition. The limit of 1.0 needed to nearly eliminate throttling is about 5 times that.

The throttle rate went from 12.2% to 0.38% when the limit was raised from 0.5 to 1.0. With nr_throttled and nr_periods in monitoring, the effect of raising the limit can also be confirmed by the change in this value.

Notes

The CPU limit appears in cpu.max under Docker, systemd, and Kubernetes (k3s) alike, and cpu.stat can be read in all of them. A half-core limit was 50000 100000 in every case. What differs between environments is where the limit is set and the path to cpu.stat.⁹

Environment	Where the limit is set	Location of `cpu.stat`	From inside the container
Docker (systemd cgroup driver)	`--cpus`	`/sys/fs/cgroup/system.slice/docker-<id>.scope/`	Readable at `/sys/fs/cgroup/cpu.stat`
systemd service	`CPUQuota=`	`/sys/fs/cgroup/system.slice/<name>.service/`	N/A
Kubernetes (k3s)	Pod `resources.limits.cpu`	`/sys/fs/cgroup/kubepods.slice/kubepods-<qos>.slice/kubepods-<qos>-pod<uid>.slice/cri-containerd-<cid>.scope/`	Readable at `/sys/fs/cgroup/cpu.stat`

Docker ran on a GitHub Actions runner (ubuntu-24.04, 4 vCPU, cgroup v2). ↩
The reproduction results are here, and the code is in show-your-work. Each condition ran in a separate CI job, and the runner CPU model differed between jobs (a mix of AMD EPYC and Intel Xeon). Comparisons between conditions in this article therefore span different CPU models. ↩
The kernel 6.8 cgroup v2 docs on cpu.stat. ↩
On kernel 5.3 and earlier, a bug where the unused part of the quota handed out to each CPU expires also produces a high throttle rate at low CPU usage. The fix went into 5.4 (de53fd7). The measurements in this article were taken on a newer kernel. ↩
The description of --cpus in Docker's resource constraints docs, and the kernel 6.8 cgroup v2 docs on cpu.max. ↩
The kernel 6.8 documentation on CFS bandwidth control. ↩
In kernel 6.8, unthrottle_cfs_rq() adds up the time each per-CPU run queue was stopped, each time it releases one. ↩
The kernel 6.8 functions throttle_cfs_rq() and unthrottle_cfs_rq(). ↩
The CPU limit settings in systemd (CPUQuota=) and Kubernetes (Resource Management for Pods and Containers). ↩

No shell in a distroless Pod: what kubectl debug can and can't see

Kazu — Tue, 29 Sep 2026 13:02:00 +0000

Something is off with a Pod. The app logs show nothing that looks like an error.

At this point I want to get inside and look around: open a shell with kubectl exec, check processes with ps, send a request to the app with curl localhost. The usual routine.

But the shell won't start.

$ kubectl exec app -c app -- sh
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "5559c54c81498e2d06cfaf7704ee420b7a48a48ef1531e34307aa515f53083b3": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

The last part, exec: "sh": executable file not found in $PATH, is the answer: this container has no sh. It has no ls or cat either.

The container runs from a distroless image. A distroless image contains only the app's binary and the minimum files it needs to run. It leaves out the shell and the package manager so that there are no tools for an attacker to use. As a result, even during an incident, you can't run commands inside the container to investigate.

Kubernetes has a mechanism for this situation. You can add a temporary container to a running Pod. This is called an ephemeral container, and you add one with the kubectl debug command.

You leave the app container untouched and add a separate container to the same Pod, one that has curl, ss, and tcpdump. Containers in a Pod share the network, so running curl localhost:3000 from the added container reaches the app directly.

The official docs cover the basic usage of kubectl debug, so this article covers what you can see once you're inside.

Once you're inside, you quickly run into cases where you can't get the information you want.

You try to read the app's config file, and nothing exists at that path
You check the CPU limit in /sys/fs/cgroup/cpu.max, and instead of the app's 50000 100000 you get your own unlimited max 100000
There's a way around to see the app's files, but without SYS_PTRACE you get Permission denied

Even within the same Pod, how far you can reach into the app is determined by Linux namespaces, and whether you can read what you reach is determined by permissions. This article doesn't walk through tracking down the cause of an incident. It checks, one item at a time against real output, what you can and can't see once inside along these two axes, and shows a workaround for each thing you can't see.

Test setup

I ran everything on a kind v0.33.0 cluster on a GitHub Actions x86_64 runner (ubuntu-24.04, kernel 6.17). Kubernetes is v1.37.0 and the container runtime is containerd v2.3.4. The target app is a small HTTP server written in Go, built on gcr.io/distroless/static-debian12:nonroot. For the debug image I used nicolaka/netshoot:v0.16. All output in this article is pasted as-is from this environment.

kind is a tool that runs a Kubernetes cluster using Docker containers as nodes. I used a cluster with a single node (ded-control-plane).

I put one Pod, app, on this cluster and added containers to it with kubectl debug to investigate.

Node ded-control-plane (kind)
└── Pod app
    ├── app            distroless HTTP server (:3000, uid 65532)
    ├── sidecar        HTTP server from the same image (:4000)
    ├── dbg-notarget   added with kubectl debug (no --target)
    ├── dbg-default    added with kubectl debug (--target=app)
    └── dbg-sysadmin   added with kubectl debug (--target=app --profile=sysadmin)

The app container is the one under investigation. The distroless :nonroot image sets the runtime user to uid 65532 (nonroot), and this Pod doesn't override it with runAsUser, so app runs as uid 65532. Two of its settings matter in later sections.

The CPU limit is 500m (half a core)
The ConfigMap's app.json is mounted at /config

sidecar is a regular container defined in the Pod from the start, running the same image as app on a different port. It's there to compare what containers in a Pod share.

The three containers whose names start with dbg- are ephemeral containers added later with kubectl debug. They differ only in the options passed.

--target=app makes the added container see the same set of processes as the app container. With it, app's processes show up in ps.

--profile selects the permissions the added container gets. If you don't specify it, you get general, which adds one capability for inspecting other processes (SYS_PTRACE). dbg-notarget and dbg-default use this profile. Specifying sysadmin gives you a privileged container, meaning one with nearly every permission. A later section shows measured differences between the profiles.

The reproduction code is here.

https://github.com/shinagawa-web/show-your-work/tree/main/distroless-ephemeral-debug

Background: container isolation is set per namespace

Linux can partition what a process sees (the network, process IDs, the filesystem, and so on), each one separately. Each of these partitions is called a namespace. A container is built from a combination of these namespaces.

The key point is that each namespace is independent. You can't describe a container as simply "isolated". It's normal for a container to share the network with its neighbor while having a separate filesystem. This explains why, from a container added with kubectl debug, curl localhost reaches the app while the app's config file is not visible. Both happen at the same time.

You can check how they're actually split from the node. Each process has one link per namespace under /proc/<pid>/ns/. Reading /proc/<pid>/ns/net with readlink, for example, returns a string like net:[4026532670]. The number in brackets identifies the namespace, and processes with the same number are in the same namespace.

To do this you need each container's PID. On the node ded-control-plane, I used crictl (a command that talks to the container runtime directly) and took each container's process PID on the node from .info.pid in crictl inspect <container ID>. For each PID I ran readlink on every link under /proc/<pid>/ns/ and put the results on one line per container. That gives the table below, trimmed to the rows and columns this article needs. I dropped the ipc, uts, user, and time columns because this article doesn't use them.

container      node_pid  net                    pid                    mnt                    cgroup
node-init      1         net:[4026532365]       pid:[4026532362]       mnt:[4026532359]       cgroup:[4026532364]
app            4082      net:[4026532670]       pid:[4026532736]       mnt:[4026532735]       cgroup:[4026532737]
sidecar        4118      net:[4026532670]       pid:[4026532739]       mnt:[4026532738]       cgroup:[4026532740]
dbg-notarget   4193      net:[4026532670]       pid:[4026532742]       mnt:[4026532741]       cgroup:[4026532743]
dbg-default    4247      net:[4026532670]       pid:[4026532736]       mnt:[4026532744]       cgroup:[4026532745]
dbg-sysadmin   4863      net:[4026532670]       pid:[4026532736]       mnt:[4026532754]       cgroup:[4026532364]

node_pid is the PID on the node from crictl, and the columns to its right are the readlink results. node-init is PID 1 on the node, included for comparison.

net has the same number for every container. For pid, only dbg-default and dbg-sysadmin have the same number as app. mnt is different for every container. For cgroup, only dbg-sysadmin has the same number as the node.

The next table summarizes these results per namespace and adds what each namespace separates and what a debug container can see.

namespace	What it separates	Between containers in a Pod	Ephemeral container	Result
network	IP addresses, ports, sockets	Shared	Shared	`curl localhost` and `ss` show the app's traffic
PID	Process IDs and which processes are visible	Separate (can be shared with `shareProcessNamespace`)	Shared with the target when `--target` is set, separate otherwise	With `--target`, the target's processes show up in `ps`
mount	How the filesystem looks	Separate	Separate	The other container's files aren't directly visible
cgroup	How CPU and memory limits look	Separate	Separate (only sysadmin shares it with the node)	`/sys/fs/cgroup` shows your own values (except with sysadmin)

This table covers how far you can reach into the app. Whether you can read what you reach depends on permissions, which I cover in "Permissions: SYS_PTRACE and profiles".

Adding an ephemeral container

For the debug image I use nicolaka/netshoot. It bundles network troubleshooting tools such as curl, ss, tcpdump, and dig, along with basic commands like ps.

To get an interactive shell, run this.

$ kubectl debug -it app --image=nicolaka/netshoot:v0.16 --target=app -c dbg-it -- bash
Targeting container "app". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.

Two warnings appear before the shell opens.

The first says that --target only works if the container runtime supports it. In this article I confirmed it works on containerd v2.3.4. I don't cover what things look like on a runtime without support.

The second says that every command you type in this session, and its output, is recorded in the container logs. For example, if you print /proc/1/environ as in "Environment and arguments: read from /proc/1" below, the app's environment variables end up in these logs, secrets included.

`--target` changes what `ps` shows

In the table in the background section, the PID row was the only one that changed with --target. Let's check this with ps.

For this test I wanted to run commands in the same container repeatedly and compare the output. So instead of entering with -it, I kept each container running with sleep infinity and ran commands with kubectl exec -c <container name>. I added dbg-notarget without --target and dbg-default with --target=app as follows. --image-pull-policy=Never is there because the image was preloaded onto the kind node.

$ kubectl debug app --image=nicolaka/netshoot:v0.16 --image-pull-policy=Never -c dbg-notarget -- sleep infinity
$ kubectl debug app --image=nicolaka/netshoot:v0.16 --image-pull-policy=Never -c dbg-default --target=app -- sleep infinity
Targeting container "app". If you don't see processes from this container it may be because the container runtime doesn't support this feature.

I ran ps auxf in each.

$ kubectl exec app -c dbg-notarget -- ps auxf
PID   USER     TIME  COMMAND
    1 root      0:00 sleep infinity
   14 root      0:00 ps auxf

$ kubectl exec app -c dbg-default -- ps auxf
PID   USER     TIME  COMMAND
    1 65532     0:00 /server -http=:3000
   29 root      0:00 sleep infinity
   42 root      0:00 ps auxf

Without --target, all you see is your own sleep infinity and the ps you just ran. The app's process is nowhere to be found.

With --target=app, the app's /server shows up as PID 1, and your own sleep infinity becomes PID 29. The 65532 in the USER column is the uid of the distroless nonroot user.

The app starts as PID 1 inside its container, so it also appears as PID 1 from a container that joins its PID namespace with --target. That's why later sections write /proc/1/.... However, if the app is launched through an init process like tini, or if the Pod has shareProcessNamespace enabled, the app may not be PID 1. Check with ps first.

dbg-notarget, which has no --target, also has the capability for inspecting other processes (SYS_PTRACE). But it can't see the app's process at all, so there's nothing to use it on. Every step below that uses /proc/1/root or /proc/1/environ assumes you entered with --target.

What you can see directly

From here on, I check the background table against real output. First come the things an ephemeral container reaches directly. The network namespace is shared across the whole Pod, and the PID namespace is shared with app through --target. Unless stated otherwise, the results are from inside dbg-default, which was added with --target=app.

dbg-default has SYS_PTRACE, so everything read in this section was readable except /proc/1/stack. Some of these become unreadable without SYS_PTRACE. I cover that in "Permissions: SYS_PTRACE and profiles".

Network: use it directly

The network row in the table said "Shared". From dbg-default, you see the same network as the app.

$ ss -tanp
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      4096               *:3000            *:*    users:(("server",pid=1,fd=4))
LISTEN 0      4096               *:4000            *:*

:3000 is the port the app (server, pid 1) listens on. :4000 is the sidecar's port. You can see that something is listening on it, but not which process.

This also matches the table. The network is shared across the whole Pod, so the sidecar's socket is visible. The PID namespace, on the other hand, is shared only with app, so the sidecar's process isn't visible. ss finds a socket's owner by looking through the processes under /proc, so it can't show the name of a process that isn't visible.

localhost reaches the app as well.

$ curl -sv -m 3 localhost:3000/health
...
< HTTP/1.1 200 OK
< Date: Sun, 27 Sep 2026 02:28:19 GMT
< Content-Length: 3
< Content-Type: text/plain; charset=utf-8
< 
{ [3 bytes data]
ok
* Connection #0 to host localhost:3000 left intact

The response is 200 OK with the body ok, so the app answers requests from localhost. The ... part is the connection setup, which I left out.

You can capture packets too. I ran tcpdump while sending curl requests in the background, and stopped after the first 2 packets, which show the SYN and SYN-ACK of one connection.

$ (for i in 1 2 3; do sleep 1; curl -s -m 2 localhost:3000/health >/dev/null; done) & tcpdump -i any -nn -c 2 port 3000; rc=$?; wait; exit $rc
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
02:28:20.415650 lo    In  IP6 ::1.43214 > ::1.3000: Flags [S], seq 838274280, win 65476, options [mss 65476,sackOK,TS val 235319763 ecr 0,nop,wscale 10], length 0
02:28:20.415666 lo    In  IP6 ::1.3000 > ::1.43214: Flags [S.], seq 4234760723, ack 838274281, win 65464, options [mss 65476,sackOK,TS val 235319763 ecr 235319763,nop,wscale 10], length 0
2 packets captured
20 packets received by filter
0 packets dropped by kernel

Both captured packets are IPv6 traffic over lo (loopback). The first is a connection request (Flags [S], SYN) from port 43214 on ::1 to the app's :3000. The second is the app's reply (Flags [S.], SYN-ACK). Traffic to the app can be captured from inside the debug container.

Name resolution works too. I looked up the name of the Service app against the nameserver listed in dbg-default's own /etc/resolv.conf.

$ dig @$(awk '/^nameserver/{print $2; exit}' /etc/resolv.conf) app.default.svc.cluster.local +short
10.96.248.99

The returned 10.96.248.99 is the ClusterIP of the Service app. The app's environment variable APP_SERVICE_HOST holds the same value.

Process state: read from /proc/1

As shown in "--target changes what ps shows", the app appears as PID 1 when you enter with --target=app. Its process state is readable under /proc/1/.

$ grep -E '^(Name|State|Uid|Threads|VmRSS):' /proc/1/status
Name:   server
State:  S (sleeping)
Uid:    65532   65532   65532   65532
VmRSS:      7256 kB
Threads:    4

State is S (sleeping), meaning the process is blocked waiting for something. Threads shows 4 threads, and VmRSS, the physical memory actually in use, is 7256 kB. Uid is 65532 in every field because the process runs as the nonroot user.

The list of open files is in /proc/1/fd.

$ ls -l /proc/1/fd
total 0
lrwx------    1 65532    65532           64 Sep 27 02:28 0 -> /dev/null
l-wx------    1 65532    65532           64 Sep 27 02:28 1 -> pipe:[45581]
l-wx------    1 65532    65532           64 Sep 27 02:28 2 -> pipe:[45582]
lr-x------    1 65532    65532           64 Sep 27 02:28 3 -> /sys/fs/cgroup/cpu.max
lrwx------    1 65532    65532           64 Sep 27 02:28 4 -> socket:[45610]
lrwx------    1 65532    65532           64 Sep 27 02:28 5 -> anon_inode:[eventpoll]
lrwx------    1 65532    65532           64 Sep 27 02:28 6 -> anon_inode:[eventfd]

fd 4 is a socket, and it corresponds to fd=4 (:3000) in the earlier ss -tanp output.

For slow operations such as receiving from the network or reading from disk, a program makes a system call and blocks inside the kernel (the core of the OS) until the call completes. /proc/1/wchan (short for wait channel) holds the name of the kernel function where the process is currently blocked, so it tells you what the process is waiting for.

$ cat /proc/1/wchan; echo
ep_poll

In this run it was ep_poll. When I entered the same Pod with --profile=sysadmin, it returned futex_do_wait. The value depends on when you read it, so read it a few times instead of judging from a single read.

ep_poll: waiting for data to arrive on one of its connections. This is the idle state of waiting for requests
futex_do_wait: blocked while synchronizing with another thread of the same program. In Go apps, idle threads are often in this state

For more detail on what the process is waiting for, /proc/1/stack prints the call stack inside the kernel. However, it's not readable from dbg-default.

$ cat /proc/1/stack
cat: read error: Permission denied

Among the profiles I tried, only the privileged dbg-sysadmin could read it.

$ cat /proc/1/stack
[<0>] futex_do_wait+0x3a/0x70
[<0>] __futex_wait+0x99/0x100
[<0>] futex_wait+0x72/0x120
[<0>] do_futex+0xc9/0x190
[<0>] __x64_sys_futex+0x125/0x1f0
[<0>] x64_sys_call+0x1d1d/0x20d0
[<0>] do_syscall_64+0x7b/0xb40
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

This is the call path inside the kernel, read from bottom to top. The bottom is the entry point where the app entered the kernel, and the top is where it's blocked right now. In this output, the app is waiting in futex (a system call for synchronizing threads) and is blocked in futex_do_wait at the top. Numbers like +0x3a/0x70 are offsets within the function, and you can usually ignore them.

The top entry, futex_do_wait, is the same as the wchan value read from the same dbg-sysadmin. stack also shows the path that led there. Still, if all you need is what the process is waiting for, wchan is usually enough.

Environment and arguments: read from /proc/1

The app's environment variables at startup are in /proc/1/environ. The entries are separated by NUL characters, so replace them with newlines to read them.

$ tr '\0' '\n' < /proc/1/environ
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=app
SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
APP_MODE=production
APP_UPSTREAM={"upstream": "http://backend.default.svc:8080", "timeout_ms": 250}
...

After the ..., the output continues with Service addresses and other variables that Kubernetes injects automatically.

The startup arguments are in /proc/1/cmdline.

$ tr '\0' ' ' < /proc/1/cmdline; echo
/server -http=:3000

The only argument is -http=:3000, the same command line that ps showed.

What you can't see directly

Next come the things the table marks "Separate", which you can't reach just by entering: mount and cgroup. Both have a workaround through /proc/1/root.

Files: read from /proc/1/root

In the introduction I wrote that when you try to read the app's config file, nothing exists at that path. Here is that case. The app mounts a ConfigMap at /config, but reading the same path from dbg-default finds nothing.

$ cat /config/app.json
cat: can't open '/config/app.json': No such file or directory

The mount row in the table said "Separate". The / in dbg-default is the netshoot image's filesystem, which is separate from the app's.

This is where /proc/1/root comes in. /proc/<pid>/root is a link that points to "/ as seen by that process", and following it takes you straight into the filesystem the app sees. You can read the contents of an image that has no shell and no ls with the tools on your side.

$ ls /proc/1/root/
bin
boot
config
dev
etc
home
lib
proc
...
root
run
sbin
scratch
server
sys
tmp
usr
var

config is where the ConfigMap is mounted, scratch is where the Pod spec mounts an emptyDir, and server is the app's binary. The ConfigMap's contents are readable too.

$ cat /proc/1/root/config/app.json
{"upstream": "http://backend.default.svc:8080", "timeout_ms": 250}

The config the app reads is now readable from the debug container.

If you suspect name resolution, you can read the resolv.conf the app actually uses in the same way.

$ cat /proc/1/root/etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local urp3p5u2pksefpdlab2z34ww2g.gx.internal.cloudapp.net
nameserver 10.96.0.10
options ndots:5

10.96.0.10 in nameserver is where the app sends its queries. search lists the domains that are appended to short names in order. ndots:5 means that for a name with fewer than 5 dots, the search domains are appended and tried first. When a name doesn't resolve, this tells you which names the app actually queries. The last search entry, …cloudapp.net, isn't a cluster domain. It comes from the host side's configuration.

cgroup: read from /proc/1/root

A cgroup is the mechanism that applies CPU and memory limits per container. The limit values are readable from files under /sys/fs/cgroup. The app's CPU limit is 500m (half a core), so cpu.max should contain 50000 100000 (up to 50,000 microseconds out of every 100,000 microseconds).

From dbg-default, I checked where my own cgroup is, that /sys/fs/cgroup lists cgroup files, and the value of cpu.max, in one go. The result looks like this.

$ cat /proc/self/cgroup; ls /sys/fs/cgroup | head -n 8; cat /sys/fs/cgroup/cpu.max
0::/
cgroup.controllers
cgroup.events
cgroup.freeze
cgroup.kill
cgroup.max.depth
cgroup.max.descendants
cgroup.pressure
cgroup.procs
max 100000

max 100000 means "no limit", and it's not the app's value. This is the case from the introduction where you get your own value. What you see is the debug container's own cgroup, which has no limit.

The cgroup row in the table said "Separate". dbg-default has a cgroup namespace of its own, and its own cgroup appears at the top of /sys/fs/cgroup. The first line, 0::/, means "my cgroup is at the top of the hierarchy as I see it".

To read the app's values, go through /proc/1/root as in "Files: read from /proc/1/root". Reading /sys/fs/cgroup as the app sees it gives the app's own values.

$ cat /proc/1/root/sys/fs/cgroup/cpu.max
50000 100000

The output is 50000 100000, matching the app's 500m limit. This path goes through /proc/1/root, so whether you can read it depends on having SYS_PTRACE. It was readable with general (no --profile) and sysadmin, and returned Permission denied with legacy, baseline, and netadmin. I explain why in "Permissions: SYS_PTRACE and profiles".

The exception is sysadmin. dbg-sysadmin shares the cgroup namespace with the node, so the app's cgroup path shown in /proc/1/cgroup exists at the same path under its own /sys/fs/cgroup.

$ ls -d /sys/fs/cgroup$(cut -d: -f3 /proc/1/cgroup)
/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-pod719135d1_ee24_4f46_948a_440e1c6f4f77.slice/cri-containerd-698b44cb65ab91fba808118cba48f23ab4761ddd8a3e05977a04c4323bcef648.scope

This reaches the app's cgroup directory directly, without going through /proc/1/root.

If you have permission to get onto the node, you can also read the app's cgroup directly from the node side. I cover that later in "Reading cgroups from the node".

Permissions: SYS_PTRACE and profiles

The reads of /proc/1/root and /proc/1/environ so far worked because dbg-default has SYS_PTRACE. The table in the background section only covers how far you can reach into the app. Whether you can read what you reach depends on the permissions in this section.

SYS_PTRACE: without it, you can't read the contents

SYS_PTRACE is what decides between reading the contents of /proc/1 and getting Permission denied.

When you read /proc/1/root, /proc/1/environ, /proc/1/maps, the link targets in /proc/1/fd, and similar files, the kernel checks whether the reading process is allowed to inspect the contents of the other process. dbg-default runs as root (uid 0 / gid 0). The app runs as uid 65532, and its gid is also 65532 (you can see this in the owner columns, 65532 65532, of the earlier ls -l /proc/1/fd). When your uid and gid differ from the target's like this, inspecting it requires SYS_PTRACE.

This isn't about file permissions. You're root, and you even have the capability to bypass file permissions (DAC_OVERRIDE), but that doesn't get you past this check.

To confirm this, I also entered with --profile=legacy, a profile without SYS_PTRACE, ran the same commands as with default, and compared the results. This container is dbg-legacy, added to the same Pod app with only --profile changed. legacy is deprecated, and kubectl prints a warning when you add it saying that it's scheduled for removal in v1.39. baseline, which also lacks SYS_PTRACE, gave the same results as legacy.

I compared 9 items that read the contents of the app's process. They are the process column of ss -tanp, the link targets in /proc/1/fd, /proc/1/maps, /proc/1/environ, the listing of /proc/1/root/, and reads of app.json, resolv.conf, cpu.max, and cpu.stat through /proc/1/root/.

All 9 items failed with legacy, and all 9 passed with default.

Here are two examples of the failures after entering app with legacy.

$ cat /proc/1/root/config/app.json
cat: can't open '/proc/1/root/config/app.json': Permission denied

$ ls -l /proc/1/fd
total 0
ls: /proc/1/fd/0: cannot read link: Permission denied
lrwx------    1 65532    65532           64 Sep 27 02:28 0
...

Both are Permission denied. ls -l /proc/1/fd prints the list but can't read the link targets.

The users:(...) column disappears from ss -tanp.

$ ss -tanp
State     Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
LISTEN    0      4096               *:3000             *:*   
LISTEN    0      4096               *:4000             *:*   
...

The :3000 line has no users:(("server",pid=1,fd=4)) either, so you can't tell which process owns the socket.

wchan doesn't produce an error, which makes this case easy to miss. Reading it after entering app with legacy returned 0 instead of futex_do_wait or ep_poll.

$ cat /proc/1/wchan; echo
0

If wchan is 0, first check whether you have SYS_PTRACE.

On the other hand, /proc/1/status, /proc/1/cmdline, and the listing of /proc/1/fd itself were readable even with legacy.

SYS_PTRACE is the capability for inspecting another process, and it's useless if that process isn't visible. --target makes the target visible, and SYS_PTRACE grants permission to inspect it. The commands so far work only with both in place.

uid and gid: when both match, reads worked without SYS_PTRACE

There's another way past this kernel check. In this test, when both the uid and the gid of my process matched the target's, reads worked without SYS_PTRACE. However, every target I tried drops all capabilities with capabilities.drop: ["ALL"]. The kernel check also has a condition on the target's capabilities, and I didn't test targets that keep any capabilities.

To confirm this, I prepared two Pods in addition to app, with only runAsUser and runAsGroup changed. app-uid0 runs the app as uid 0 / gid 65532, and app-uid0-gid0 runs it as uid 0 / gid 0.

Reading each app's /proc/<pid>/status from the node gave the following. The first block is app-uid0, and the second is app-uid0-gid0.

$ grep -E '^(Uid|Gid|Groups|CapInh|CapPrm|CapEff|CapBnd|CapAmb|NoNewPrivs|Seccomp):' /proc/7049/status
Uid:    0   0   0   0
Gid:    65532   65532   65532   65532
...
CapEff: 0000000000000000
...

$ grep -E '^(Uid|Gid|Groups|CapInh|CapPrm|CapEff|CapBnd|CapAmb|NoNewPrivs|Seccomp):' /proc/7042/status
Uid:    0   0   0   0
Gid:    0   0   0   0
...
CapEff: 0000000000000000
...

app-uid0 has uid 0 and gid 65532, and app-uid0-gid0 has uid 0 and gid 0. Both have CapEff set to all zeros, so both run with every capability dropped.

I entered each one with legacy and ran the same commands. The results for app are included for comparison.

Target app	uid / gid	Result with legacy
`app`	65532 / 65532	All 9 items failed
`app-uid0`	0 / 65532	All 9 items failed
`app-uid0-gid0`	0 / 0	All passed

Matching only the uid (app-uid0) isn't enough. The reads pass only when the gid matches as well (app-uid0-gid0).

wchan follows the same condition. Entering app-uid0-gid0 with legacy returned futex_do_wait rather than 0. So the 0 seen in "SYS_PTRACE: without it, you can't read the contents" can be read as the value the kernel returns when this check fails.

The distroless nonroot image uses 65532 for both uid and gid. As long as you enter as root, both the uid and the gid differ from the app's, so the quickest route is to enter with a profile that includes SYS_PTRACE.

Profiles: only general and sysadmin can read /proc/1/root

The capabilities the debug container gets are determined by --profile, and the profile determines which commands fail.

What differs between profiles is the securityContext, the part of the spec that sets things like a container's capabilities. I also added legacy, general, baseline, and netadmin to the same Pod app as dbg-default and dbg-sysadmin, changing only --profile, and ran the same commands to compare them. restricted requires an image that runs as non-root, and it didn't start with netshoot, which runs as root, so I left it out of the comparison. Here are the results.

profile	securityContext	Capabilities	`/proc/1/root`	`iptables`
general (same as no `--profile`)	`capabilities.add: [SYS_PTRACE]`	Base + `SYS_PTRACE`	Readable	Fails
legacy / baseline	None	Base only	Not readable	Fails
netadmin	`capabilities.add: [NET_ADMIN, NET_RAW]`	Base + `NET_ADMIN`	Not readable	Works
sysadmin	`privileged: true`	Nearly all	Readable	Works

"Base" in the table is the set of capabilities the container runtime (containerd) grants when the securityContext specifies nothing.

Without NET_ADMIN, iptables fails with the misleading error Permission denied (you must be root), even when it runs as root. tcpdump worked with every profile in the table, because NET_RAW, which packet capture needs, is in the base set from the start. netadmin adds NET_ADMIN and NET_RAW, but NET_RAW is already there, so the only real addition is NET_ADMIN. That's why the Capabilities column shows "Base + NET_ADMIN".

Reading cgroups from the node

As shown in "cgroup: read from /proc/1/root", an ephemeral container can read the app's cgroup values through /proc/1/root/sys/fs/cgroup/..., and with sysadmin you can follow them directly from /sys/fs/cgroup. Looking from the node is the way out when neither of these is available. Here are two such cases.

Pod Security settings or similar policies on the app's Kubernetes namespace allow neither SYS_PTRACE nor privileged containers
You want to look across multiple Pods on the node

Even in the first case, starting the node debug Pod in a different Kubernetes namespace that allows privileged Pods should let you get onto the node. I didn't test that far in this article, though.

Enter the node with kubectl debug node

kubectl debug node/<node name> starts a debug Pod on the specified node and attaches you to it.

$ kubectl debug node/ded-control-plane -it --profile=sysadmin --image=nicolaka/netshoot:v0.16 -- bash
Creating debugging pod node-debugger-ded-control-plane-sjpv8 with container debugger on node ded-control-plane.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.

This Pod's spec shows that it shares the PID, network, and IPC namespaces with the node, runs privileged, and mounts the node's / at /host.

$ kubectl get pod/node-debugger-ded-control-plane-sjpv8 -o json | jq '{hostPID: .spec.hostPID, hostNetwork: .spec.hostNetwork, hostIPC: .spec.hostIPC, sc: .spec.containers[0].securityContext, mounts: .spec.containers[0].volumeMounts, volumes: .spec.volumes}'
{
  "hostPID": true,
  "hostNetwork": true,
  "hostIPC": true,
  "sc": {
    "privileged": true
  },
  "mounts": [
    {
      "mountPath": "/host",
      "name": "host-root"
    },
...
  "volumes": [
    {
      "hostPath": {
        "path": "/",
        "type": ""
      },
      "name": "host-root"
    },
...

It's privileged, and on top of that it has the node's PID, network, and IPC namespaces and the node's filesystem. That makes it a Pod with very strong permissions. Every process on the node is visible, and so is the node's entire /. When you use it on a production node, keep in mind that this is the level of access you're working with.

Let's look under /host.

$ ls --color=never /host
LICENSES  bin       boot      dev       etc       home      kind      lib       lib64     media     mnt       opt       proc      root      run       sbin      srv       sys       tmp       usr       var

This is the node's /.

Trace the cgroup from the app's process

The PID namespace is shared with the node, so you can find the app's process on the node with ps.

$ ps -eo pid,user,args | grep '[/]server -http='
 4082 65532    /server -http=:3000
 4118 65532    /server -http=:4000

:3000 is the app and :4000 is the sidecar, both running as uid 65532. Take the PID of the :3000 process.

$ P=$(pgrep -u 65532 -f '/server -http=:3000')
$ echo "P=$P"
P=4082

4082 is the same as app's PID on the node in the table from the background section. Reading /proc/<pid>/cgroup for this PID gives the path from the root of the node's cgroup hierarchy.

$ cat /proc/$P/cgroup
0::/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-pod719135d1_ee24_4f46_948a_440e1c6f4f77.slice/cri-containerd-698b44cb65ab91fba808118cba48f23ab4761ddd8a3e05977a04c4323bcef648.scope

The path is the part after 0::. Extract it.

$ CG=$(cut -d: -f3 /proc/$P/cgroup)
$ echo "CG=$CG"
CG=/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-pod719135d1_ee24_4f46_948a_440e1c6f4f77.slice/cri-containerd-698b44cb65ab91fba808118cba48f23ab4761ddd8a3e05977a04c4323bcef648.scope

Append this path to /host/sys/fs/cgroup, which corresponds to the node's /sys/fs/cgroup.

$ cat /host/sys/fs/cgroup$CG/cpu.max
50000 100000

50000 100000 in cpu.max is a limit of up to 50,000 microseconds of CPU time out of every 100,000 microseconds (100 ms). It matches the 500m limit set in "Test setup".

Whether the app actually hits this limit shows up in cpu.stat in the same directory.

$ cat /host/sys/fs/cgroup$CG/cpu.stat
usage_usec 56270
user_usec 34699
system_usec 21570
nice_usec 0
core_sched.force_idle_usec 0
nr_periods 22
nr_throttled 0
throttled_usec 0
nr_bursts 0
burst_usec 0

In cpu.stat, nr_periods is the number of 100 ms cpu.max periods in which the app tried to use the CPU. nr_throttled is the number of those periods in which it hit the limit and was throttled. In this run it was 0 out of 22, so the app isn't hitting the limit.

Limitations and caveats

Added containers can't be removed from the Pod

When you add a container with kubectl debug, its definition is appended to the Pod's spec. The process inside stops using resources once it exits, but the definition stays, even for containers that have already terminated. Here's the list after adding dbg-exit, which exits immediately because it runs -- true.

$ kubectl get pod app -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'
dbg-notarget dbg-default dbg-it dbg-legacy dbg-general dbg-baseline dbg-restricted dbg-netadmin dbg-sysadmin dbg-exit

Every container I added for this article's tests is on the list. An attempt to remove one is rejected by the API server.

$ kubectl patch pod app --subresource=ephemeralcontainers --type=json -p '[{"op":"remove","path":"/spec/ephemeralContainers/0"}]'
The Pod "app" is invalid: spec.ephemeralContainers: Forbidden: existing ephemeral containers "dbg-notarget" may not be removed

The record goes away only when the Pod is recreated. Until then the names stay taken as well. An attempt to add another container named dbg-exit was rejected with existing ephemeral containers "dbg-exit" may not be changed. If you experiment casually on a production Pod, these records stay there.

You can't use `--target` on a stopped container

An app that crashes right after starting and keeps restarting is in a state called CrashLoopBackOff. For testing, I prepared a Pod crash that exits right after it starts. It mounts the same ConfigMap as app at /config and gets the same environment variables (APP_MODE, APP_UPSTREAM).

$ kubectl get pod crash
NAME    READY   STATUS             RESTARTS      AGE
crash   0/1     CrashLoopBackOff   4 (81s ago)   2m50s

It's in CrashLoopBackOff and has already restarted 4 times. To find out why it crashed, start with the logs of the previous container.

$ kubectl logs crash -c app --previous
2026/09/27 02:31:58 exiting on purpose

exiting on purpose is the message from the deliberate exit I built in for this test.

If you add a debug container to this Pod with --target=app, it doesn't start (its state is CreateContainerError). The process it's supposed to share a PID namespace with isn't running.

The workaround that fits this constraint is the --copy-to option of kubectl debug. It creates a copy of the Pod and lets you replace a container in the copy and run it.

First, I replaced only the image with netshoot using --set-image.

$ kubectl debug crash --copy-to=crash-copy --set-image=app=nicolaka/netshoot:v0.16
$ kubectl get pod crash-copy -o jsonpath='{range .spec.containers[*]}{.name} {.image} {.args}{"\n"}{end}'
app nicolaka/netshoot:v0.16 ["-exit"]

The image is now netshoot, but the original Pod's argument -exit is still there.

$ kubectl get pod crash-copy
NAME         READY   STATUS              RESTARTS     AGE
crash-copy   0/1     RunContainerError   2 (4s ago)   20s

The copy didn't start either and ended up in RunContainerError. The reason is in the last termination state.

$ kubectl get pod crash-copy -o jsonpath='{.status.containerStatuses[0].lastState}{"\n"}'
{"terminated":{..."message":"failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: \"-exit\": executable file not found in $PATH","reason":"StartError",...}}

The leftover argument -exit is treated as the command to run. It doesn't exist in netshoot, so the start fails.

So I specified the container to replace with --container=app, and passed both an image with --image and a command after --.

$ kubectl debug crash --copy-to=crash-copy2 --container=app --image=nicolaka/netshoot:v0.16 --image-pull-policy=Never -- sleep infinity
$ kubectl get pod crash-copy2 -o jsonpath='{range .spec.containers[*]}{.name} {.image} {.command} {.args}{"\n"}{end}'
app nicolaka/netshoot:v0.16 ["sleep","infinity"]

This time the command is sleep infinity and the arguments are empty.

$ kubectl get pod crash-copy2
NAME          READY   STATUS    RESTARTS   AGE
crash-copy2   1/1     Running   0          1s

The copy is Running and stays up.

In the copy's app container, the image and command are replaced, but the environment variables and volume mounts are carried over unchanged from the original Pod. So you can check the config file and environment variables the app would read at startup, under the same conditions.

$ kubectl exec crash-copy2 -c app -- cat /config/app.json
{"upstream": "http://backend.default.svc:8080", "timeout_ms": 250}

$ kubectl exec crash-copy2 -c app -- printenv APP_MODE APP_UPSTREAM
production
{"upstream": "http://backend.default.svc:8080", "timeout_ms": 250}

Both match what the original app receives. However, the copy doesn't run the app itself, so you can't see the process at the moment it crashes. What you can check is the set of conditions the app receives at startup.

Wrap-up

A distroless container has no shell, so you can't get in with kubectl exec. If you add an ephemeral container with kubectl debug, though, you can bring your own tools and investigate.

The network is shared with the Pod, so curl localhost, ss, tcpdump, and dig work directly
If you enter with --target, the app's process is visible, and you can read its state and environment variables from /proc/1
The app's filesystem is separate and not directly visible. You can read it through /proc/1/root
/sys/fs/cgroup shows the debug container's own values, except with sysadmin. The app's values are readable through /proc/1/root. If an ephemeral container can't reach them, read them from the node

Whether you can read the contents of what you reach depends on two things.

Inspecting another process requires SYS_PTRACE. You get it by entering with general (no --profile) or sysadmin
If the target drops all capabilities (every target in this test did), reads worked without SYS_PTRACE when your uid and gid both matched the target's

When you use this, watch out for two things.

Added containers can't be removed from the Pod
You can't use --target on a stopped container. For a Pod that keeps crashing, make a copy with --copy-to and investigate the copy

How far you can reach into the app depends on which namespaces you share with it. Whether you can read what you reach depends mainly on SYS_PTRACE. If the target drops all capabilities, matching uid and gid also let you read it. If you keep these two questions apart, you'll know which one to suspect when you get stuck.

Kubernetes: iptables is empty after migrating to Cilium — what to look at next

Kazu — Tue, 15 Sep 2026 13:03:00 +0000

A Kubernetes Service stops responding. The first move is always the same: get on the node and run iptables-save | grep KUBE-SERVICES to see which Pod a given Service address maps to.

kube-proxy wrote those rules. It's a standard Kubernetes component that runs on every node and keeps Service forwarding rules registered in iptables. Looking up where a Service routes means reading those rules.

Cilium is a Kubernetes CNI plugin — the piece that wires up Pod networking so Pods can talk to each other. Flannel and Calico fill the same role; you pick one per cluster.

Cilium has a mode that goes beyond Pod networking and also handles Service forwarding, the job kube-proxy used to own. That mode is called kube-proxy replacement. With kubeProxyReplacement=true, Cilium runs on the assumption that kube-proxy is absent. Cilium itself does not remove kube-proxy; on an existing cluster you have to remove it separately. The test environment in this article is built with kind's kubeProxyMode: none, so kube-proxy was never present.

On a node in that Cilium environment, run the same iptables-save | grep KUBE-SERVICES from before. Nothing comes back. The same setup on a kube-proxy cluster returns 13 lines; here it returns 0. KUBE-SERVICES is the chain kube-proxy creates — the entry point for packets destined to a ClusterIP. It doesn't exist.

Broaden the grep to KUBE- and you get a few lines back, but they're KUBE-FIREWALL and KUBE-KUBELET-CANARY, chains created by kubelet that have nothing to do with Service forwarding. The nat table itself has some rules; it isn't empty. Only the Service forwarding rules are missing.

You try a different table, a different node. KUBE-SERVICES is nowhere, and yet the application is up and Services are reachable.

Nothing is broken. The information just isn't in iptables.

This article covers three things: where Cilium stores the forwarding rules that used to live in iptables; why tcpdump on a ClusterIP produces nothing and conntrack -L returns no matches; and what to check, in what order, when connectivity fails.

The test setup runs two kind clusters side by side — one with kube-proxy, one with Cilium — and runs the same commands against the same Service on both. All output shown is from real machines.

All tests run on kind v0.27.0, node image kindest/node:v1.32.2, Cilium 1.17.3, cilium-cli v0.20.0, kernel 6.8. Both clusters are single-node; multi-node-only behaviors are out of scope. ClusterIPs change each time a cluster is recreated, so IPs may differ between sections.

The reproduction repo is at https://github.com/shinagawa-web/kubernetes-cilium-kube-proxy-replacement-debug-guide

Where forwarding rules live, and when they get written

kube-proxy

Rules were registered in netfilter

A Service's ClusterIP doesn't exist on any machine. When a Pod sends a packet to it, the destination gets rewritten to a real Pod IP somewhere in the forwarding path. kube-proxy was what registered those rewrite rules in netfilter. netfilter is the Linux kernel's packet-processing framework; iptables is the command for reading and writing its rules.

That's why iptables-save worked. The forwarding rule — which Pod a ClusterIP packet goes to — was stored as a text rule registered in the kernel, so dumping everything was enough.

Here's what the packet path looks like when a Pod connects to a ClusterIP:

client Pod
   |  sends packet destined for 10.96.111.7:80 (ClusterIP)
   v
network stack
   |
   v
netfilter nat table
   KUBE-SERVICES → KUBE-SVC-xxxx → KUBE-SEP-xxxx
   |  rewrites destination to 10.244.0.6:80 (Pod IP) here
   v
Pod

tcpdump could capture the pre-DNAT packet

In this path, there's a stretch before the rewrite where the packet still carries the ClusterIP as its destination. Running tcpdump on the node with the ClusterIP as the filter caught those packets. Capturing before and after the rewrite showed exactly which Pod IP the destination changed to.

iptables-save for the rules, tcpdump for the live packets. Those two tools covered Service forwarding.

Cilium

Cluster setup

To enable kube-proxy replacement, you disable kube-proxy at cluster creation and install Cilium as the CNI. The test repo does this with kind. If you already have Cilium running, skip ahead to "No kube-proxy means no chains."

First, the kind cluster config disables both kube-proxy and the default CNI:

# cluster/kind-cilium.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.32.2
networking:
  disableDefaultCNI: true
  kubeProxyMode: none

kubeProxyMode: none keeps kube-proxy off the nodes. disableDefaultCNI: true removes kindnet so Cilium can take the CNI slot.

After creating the cluster, install Cilium via Helm. kubeProxyReplacement=true starts Cilium in the mode where it owns Service forwarding as well.

helm install cilium cilium/cilium \
  --version 1.17.3 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost="${API_SERVER_IP}" \
  --set k8sServicePort="${API_SERVER_PORT}"

k8sServiceHost and k8sServicePort are the address Cilium uses to reach the Kubernetes API server. Without kube-proxy, you have to supply these explicitly. The full setup script is in scripts/setup-cilium.sh.

No kube-proxy means no chains

With this setup, kube-proxy doesn't exist. No process registers rules, so KUBE-SERVICES is never created. That's why the initial grep returned 0 lines.

The forwarding table moves to eBPF maps

The destination mapping moves to eBPF. eBPF lets you load programs into the kernel and run them at predefined hook points — on packet receive, on system call entry — without modifying the kernel itself or rebooting. The attached program can read and modify the data passing through that point.

eBPF programs are stateless across invocations — local variables don't persist between calls. State lives in maps instead: a key-value store the kernel maintains. Programs read and write there. Cilium writes the ClusterIP-to-Pod-IP mapping into these maps; on each connection, the attached program looks up the map to decide where to forward.

The rewrite happens at connect()

If only the storage location changed, you'd just learn a new command. But there's one more difference: the rewrite happens earlier.

The same connection in a Cilium environment no longer goes through netfilter at all, and the rewrite point shifts:

client Pod
   |  calls connect() with 10.96.89.117:80 (ClusterIP) as the destination
   v
eBPF program attached to connect()
   |  rewrites destination to 10.0.0.183:80 (Pod IP)
   v
Pod

Cilium attaches an eBPF program to connect(), the system call where an application tells the kernel what it wants to connect to. At this point, not a single byte of data has been sent. The destination address gets replaced with the Pod IP right here.

The rewrite is done before any packet is constructed, so the resulting packet carries the Pod IP as its destination — no ClusterIP packet is ever created. That's why tcpdump on a network interface never shows the ClusterIP.

This applies to connections from within the cluster. Packets arriving from outside the cluster on a NodePort are handled differently, as covered in a later section.

Two kinds of "no output"

At this point there are two separate "nothing shows up" situations with different causes. iptables-save returning 0 lines is because the forwarding rules moved from iptables to eBPF maps. tcpdump returning 0 packets when filtering on a ClusterIP is because there are no packets with that ClusterIP as a destination in that path. The causes are different, so the replacement commands are different. The next section handles the first one.

Reading the forwarding table

The test environment has a single ClusterIP Service named demo, backed by a Deployment with 2 replicas. Here's how to read the forwarding table for that Service in each environment.

kube-proxy

Take another look at those 13 lines from iptables-save | grep KUBE-SERVICES.

:KUBE-SERVICES - [0:0]
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
:KUBE-SERVICES - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES -d 10.96.111.7/32 -p tcp -m comment --comment "default/demo cluster IP" -m tcp --dport 80 -j KUBE-SVC-73JNOU5FOFXIWZLE
-A KUBE-SERVICES -d 10.96.182.1/32 -p tcp -m comment --comment "default/demo-nodeport cluster IP" -m tcp --dport 80 -j KUBE-SVC-IECF2FXKD7A6IGZV
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS

The first 6 lines are chain definitions and hook points — no forwarding destination information. Lines 7 onward list each Service's entry. -d is the ClusterIP; -m comment --comment contains default/<ServiceName> cluster IP. Find the Service you want by that comment. demo is line 8 — "default/demo cluster IP" — and its ClusterIP is 10.96.111.7.

The -j KUBE-SVC-73JNOU5FOFXIWZLE at the end of that line is the jump to the backend-selection chain. Grep for that chain name to get the backend rules:

iptables-save -t nat | grep KUBE-SVC-73JNOU5FOFXIWZLE

-A KUBE-SVC-73JNOU5FOFXIWZLE ! -s 10.244.0.0/16 -d 10.96.111.7/32 -p tcp -m comment --comment "default/demo cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.6:80" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-4BM33MODP4BKIE3G
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.7:80" -j KUBE-SEP-C7KHQYF2QRUY23U4

The KUBE-SVC chain selects the backend. The first line — ! -s 10.244.0.0/16 — handles masquerading for traffic coming from outside the Pod CIDR; it's not part of backend selection. Lines 2 and 3 are the actual selection: the comment shows default/demo -> <Pod IP>:80 and --probability sets the distribution ratio. The -j KUBE-SEP-xxx at the end is the jump to the chain that executes the DNAT:

iptables-save -t nat | grep -E "KUBE-SEP-4BM33MODP4BKIE3G|KUBE-SEP-C7KHQYF2QRUY23U4"

-A KUBE-SEP-4BM33MODP4BKIE3G -p tcp -m comment --comment "default/demo" -m tcp -j DNAT --to-destination 10.244.0.6:80
-A KUBE-SEP-C7KHQYF2QRUY23U4 -p tcp -m comment --comment "default/demo" -m tcp -j DNAT --to-destination 10.244.0.7:80

--to-destination is the backend Pod IP and port.

Summary of how to read the forwarding table in a kube-proxy environment:

ClusterIP: the -A KUBE-SERVICES -d line (confirm the Service name in the comment)
Backend Pod IP: --to-destination in the corresponding KUBE-SEP chain
Weight: --probability in the KUBE-SVC chain

Cilium

First, get the ClusterIP from kubectl get svc:

kubectl get svc demo

NAME   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
demo   ClusterIP   10.96.24.124    <none>        80/TCP    4s

ClusterIP is 10.96.24.124. cilium-dbg service list -o json shows the frontend (ClusterIP), Service name, and backend Pod IPs all at once:

kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | [
    (.spec.id | tostring),
    (.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
    .spec.flags.type,
    (.spec.flags.namespace + "/" + .spec.flags.name),
    (.spec["backend-addresses"] // [] | map(.ip + ":" + (.port|tostring)) | join(" "))
  ] | @tsv' | column -t

1   10.96.0.1:443      ClusterIP  default/kubernetes       172.18.0.2:6443
5   10.96.0.10:53      ClusterIP  kube-system/kube-dns     10.0.0.88:53     10.0.0.192:53
12  10.96.24.124:80    ClusterIP  default/demo             10.0.0.40:80     10.0.0.8:80
2   0.0.0.0:30080      NodePort   default/demo-nodeport    10.0.0.113:80    10.0.0.195:80
...

The 10.96.24.124 from kubectl get svc matches ID 12. The last column shows the backends: 10.0.0.40:80 and 10.0.0.8:80. The three-hop walk through KUBE-SERVICES → KUBE-SVC → KUBE-SEP in kube-proxy fits in a single line here.

In a multi-node cluster, kubectl -n kube-system exec ds/cilium execs into whichever Pod in the DaemonSet gets scheduled. In a multi-node cluster you only see the local state of that agent. The same applies to hubble observe flows — traffic on other nodes won't appear. If you know which node your target Pod is on, check kubectl get pod -n kube-system -o wide | grep cilium for that node's cilium Pod name and exec into it explicitly with kubectl -n kube-system exec <PodName>.

Packet tracing

Comparing how far tcpdump gets you when chasing ClusterIP traffic in each environment.

kube-proxy

Running tcpdump on the node and filtering on the ClusterIP captured packets. DNAT happened in netfilter, so there was a stretch before the rewrite where packets actually carried the ClusterIP as their destination. That "bracket the DNAT with two tcpdump captures" technique only worked because of that structure.

kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "host 10.96.104.4"

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
11:38:19.980617 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [S], seq 3530419686, win 64240, options [mss 1460,sackOK,TS val 4270529724 ecr 0,nop,wscale 10], length 0
11:38:19.980665 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [S.], seq 2901899611, ack 3530419687, win 65160, options [mss 1460,sackOK,TS val 4170938121 ecr 4270529724,nop,wscale 10], length 0
11:38:19.980675 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1, win 63, options [nop,nop,TS val 4270529724 ecr 4170938121], length 0
11:38:19.980726 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [P.], seq 1:76, ack 1, win 63, options [nop,nop,TS val 4270529724 ecr 4170938121], length 75: HTTP: GET / HTTP/1.1
11:38:19.980741 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [.], ack 76, win 64, options [nop,nop,TS val 4170938121 ecr 4270529724], length 0
11:38:19.983523 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [P.], seq 1:1545, ack 76, win 64, options [nop,nop,TS val 4170938124 ecr 4270529724], length 1544: HTTP: HTTP/1.1 200 OK
11:38:19.983538 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1545, win 67, options [nop,nop,TS val 4170938124 ecr 4270529724], length 0
11:38:19.983641 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], seq 76, ack 1545, win 67, options [nop,nop,TS val 4270529727 ecr 4170938124], length 0
11:38:19.983883 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [F.], seq 1545, ack 77, win 64, options [nop,nop,TS val 4170938124 ecr 4270529727], length 0
11:38:19.983898 vethc1bb8f9e In  IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1546, win 67, options [nop,nop,TS val 4270529727 ecr 4170938124], length 0

10 packets captured
10 packets received by filter
0 packets dropped by kernel

Packets to the ClusterIP (10.96.104.4) are captured. SYN through FIN, all there. DNAT happens in netfilter after this, so the packets with the ClusterIP destination exist in this stretch.

conntrack -L also shows the DNAT in a single entry:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- conntrack -L | grep "10.96.104.4"

tcp  6 118 TIME_WAIT src=10.244.0.5 dst=10.96.104.4 sport=54462 dport=80 src=10.244.0.6 dst=10.244.0.5 sport=80 dport=54462 [ASSURED] mark=0 use=1

dst=10.96.104.4 is the ClusterIP (outbound direction); src=10.244.0.6 is the backend Pod IP (return direction). Before and after the DNAT, on one line.

Cilium

Same experiment on the Cilium cluster. Run tcpdump while curling ClusterIP 10.96.89.117 from the client Pod:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "host 10.96.89.117"

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
0 packets captured
0 packets received by filter
0 packets dropped by kernel

The curl succeeds. Connectivity is there. The empty capture is not a filter mistake — there are simply no packets with that ClusterIP as their destination.

connect() rewrites the destination to a Pod IP before any packet is created, so whatever hits the network already has the Pod IP. The ClusterIP never reaches the network layer.

conntrack -L filtered on the ClusterIP also returns nothing. The conntrack table contains only Pod-IP entries, but since no packets with the ClusterIP pass through netfilter, there's nothing to match.

kubectl debug node/<node> -it --image=nicolaka/netshoot -- conntrack -L | grep "10.96.89.117"

No output.

Seeing the ClusterIP translation with Hubble

tcpdump can't show the ClusterIP. So how do you trace ClusterIP traffic in a Cilium environment? Hubble is Cilium's observability component — it surfaces eBPF events as flow records. The test repo's scripts/setup-cilium.sh installs with Hubble Relay enabled. Curl the ClusterIP from the client Pod and then run hubble observe:

kubectl -n kube-system exec ds/cilium -- hubble observe --last 30 | grep -E "client|demo"

--last 30 outputs the 30 most recent flows.

Sep  7 07:23:31.106: default/client (ID:52940) <> 10.96.89.117:80 (world) pre-xlate-fwd TRACED (TCP)
Sep  7 07:23:31.106: default/client (ID:52940) <> default/demo-864f5f87b9-rvr95:80 (ID:10947) post-xlate-fwd TRANSLATED (TCP)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: SYN)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK)
Sep  7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Sep  7 07:23:31.110: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Sep  7 07:23:31.110: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Sep  7 07:23:31.110: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, FIN)

Focus on the first two lines. pre-xlate-fwd TRACED is the pre-translation flow, destination still recorded as 10.96.89.117:80 — the ClusterIP from kubectl get svc demo. post-xlate-fwd TRANSLATED is after, destination now the actual Pod demo-864f5f87b9-rvr95:80.

This confirms why tcpdump couldn't see the ClusterIP: the translation is done before the packet reaches the network. Hubble records both sides of the translation as eBPF events, so you can read "which ClusterIP became which Pod" directly.

NodePort is visible to tcpdump

Try NodePort on the same cluster and the result is different. Send a curl from outside the cluster to the NodePort (172.18.0.2:30080) while running tcpdump for that port on the node:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "port 30080"

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
07:23:24.736193 eth0  In  IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [S], seq 3175489113, win 64240, options [mss 1460,sackOK,TS val 194961069 ecr 0,nop,wscale 10], length 0
07:23:24.736306 eth0  Out IP 172.18.0.2.30080 > 172.18.0.1.52306: Flags [S.], seq 2271513508, ack 3175489114, win 64308, options [mss 1410,sackOK,TS val 855246694 ecr 194961069,nop,wscale 10], length 0
07:23:24.736333 eth0  In  IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [.], ack 1, win 63, options [nop,nop,TS val 194961069 ecr 855246694], length 0
07:23:24.736421 eth0  In  IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [P.], seq 1:80, ack 1, win 63, options [nop,nop,TS val 194961069 ecr 855246694], length 79
...
10 packets captured
10 packets received by filter
0 packets dropped by kernel

10 packets captured. The difference is the path. NodePort packets arrive from outside the cluster as real packets, so they're not subject to socket LB. Processing happens at the tc layer (TCX), after the NIC, which is why tcpdump sees them.

cilium-dbg status --verbose shows which layers are active:

kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
  cilium-dbg status --verbose | grep -E "Attach Mode|Socket LB|XDP Acceleration"

Attach Mode:            TCX
  Socket LB:              Enabled
  XDP Acceleration:       Disabled

Socket LB: Enabled means Cilium rewrites destinations at the socket layer (the connect() hook). ClusterIP traffic gets translated to a Pod IP there, before any packet is created — which is why tcpdump never sees it.

XDP Acceleration: Disabled means NIC-driver-level (XDP) processing is off. XDP requires a compatible NIC, so it's disabled in kind by default. Instead, as Attach Mode: TCX shows, BPF programs are attached at the tc (Traffic Control) layer. NodePort traffic is real packets from outside the cluster, not subject to socket LB, so it gets processed at that tc layer and passes through the NIC — making it visible to tcpdump.

Packet tracing summary

	kube-proxy	Cilium
tcpdump on ClusterIP	captures (packet exists before DNAT)	nothing (translated at connect())
conntrack on ClusterIP	captures (DNAT before/after on one line)	nothing (doesn't go through netfilter)
tcpdump on NodePort	captures	captures (processed at tc layer)
translation visibility	tcpdump before/after DNAT	Hubble pre-xlate-fwd / post-xlate-fwd

E2E debugging flow

Starting from "Service is unreachable," walk through the investigation steps for a Cilium environment. By the end of this section, you'll have a clear path from symptom to cause.

The test repo's manifests/broken/ has two failure patterns: svc-wrong-selector.yaml (a Service with a selector that matches no Pods) and netpol-deny.yaml (a NetworkPolicy that blocks the client). The full debug script is in tests/debug-cilium.sh.

Case A — selector mismatch

Apply manifests/broken/svc-wrong-selector.yaml. The demo-broken Service has selector app: demo-v2, but every Pod in the cluster has the label app: demo. Nothing matches.

Check the ClusterIP first:

kubectl get svc demo-broken

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
demo-broken   ClusterIP   10.96.171.11    <none>        80/TCP    4s

Send a curl from the client Pod:

kubectl exec -n default client -- \
  curl -s --max-time 5 -o /dev/null -w '%{http_code} (%{time_total}s)' http://10.96.171.11/

000 (0.000063s)

%{http_code} returning 000 means no HTTP response. %{time_total} shows it returned immediately. When a Service has zero backends, Cilium's eBPF program rejects the connection right away — same speed as kube-proxy's REJECT --reject-with icmp-port-unreachable.

Step 1 — check Endpoints

Same command as before Cilium:

kubectl get endpoints demo-broken

NAME          ENDPOINTS   AGE
demo-broken   <none>      8s

<none> means no Pods are registered as backends. Compare against a working Service:

kubectl get endpoints demo

NAME   ENDPOINTS                     AGE
demo   10.0.0.117:80,10.0.0.119:80   34s

kubectl get endpoints works exactly the same as in a kube-proxy environment (from 1.33, kubectl get endpointslices is preferred).

Step 2 — check Cilium's internal state

The JSON output includes the Service name:

kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | [
    (.spec.id | tostring),
    (.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
    .spec.flags.type,
    (.spec.flags.namespace + "/" + .spec.flags.name),
    (.spec["backend-addresses"] | length | tostring)
  ] | @tsv' | column -t

12  10.96.171.11:80   ClusterIP  default/demo-broken  0
8   10.96.249.121:80  ClusterIP  default/demo         2

The last column is backend count. demo (ID 8) has 2; demo-broken (ID 12) has 0. A backend count of 0 indicates endpoint shortage. Cilium registers the Service but has nowhere to forward connections.

In a kube-proxy environment, the same situation shows up in the filter table with a comment:

kubectl debug node/<node> -it --image=nicolaka/netshoot -- iptables-save -t filter | grep "10.96.171.11"

-A KUBE-SERVICES -d 10.96.171.11/32 -p tcp -m comment --comment "default/demo-broken has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable

has no endpoints is right there in the comment. Run that same command on the Cilium cluster and you get nothing. For endpoint-shortage cases (as opposed to NetworkPolicy), iptables stops explaining itself. Use cilium-dbg service list -o json | jq backend count instead. 0 means no endpoints.

Root cause

The most common reason Endpoints is empty is a selector mismatch between the Service and its Pods. Compare them directly:

kubectl get svc demo-broken -o jsonpath='{.spec.selector}'

{"app":"demo-v2"}

kubectl get pod -l app=demo -o jsonpath='{range .items[*]}{.metadata.name}  {.metadata.labels}{"\n"}{end}'

demo-864f5f87b9-dccbz  {"app":"demo","pod-template-hash":"864f5f87b9"}
demo-864f5f87b9-zzngq  {"app":"demo","pod-template-hash":"864f5f87b9"}

The selector points to app: demo-v2; the Pods carry app: demo.

Case B — NetworkPolicy deny

Apply manifests/broken/netpol-deny.yaml. The deny-client-to-demo NetworkPolicy allows Ingress to Pods with app: demo only from Pods labeled app: not-client. The client Pod doesn't match, so all connections to demo are denied.

kubectl exec -n default client -- \
  curl -s --max-time 5 -o /dev/null -w '%{http_code} (%{time_total}s)' http://10.96.249.121/

000 (5.001378s)

Same symptoms as Case A — but %{time_total} is 5 seconds. Case A (no backends) returned in 0.000063 seconds. Here, --max-time was exhausted. Packets are reaching the destination; nothing is sending a response back. Endpoints tell a different story.

Step 1 — check Endpoints

kubectl get endpoints demo

NAME   ENDPOINTS                     AGE
demo   10.0.0.117:80,10.0.0.119:80   34s

Endpoints are healthy. That's the branch point from Case A. Empty Endpoints → check selector match and Pod readiness. Endpoints present → keep going.

Step 2 — confirm Cilium sees active backends

kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | select(.spec["frontend-address"].ip == "10.96.249.121") | [
    (.spec.id | tostring),
    (.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
    .spec.flags.type,
    (.spec.flags.namespace + "/" + .spec.flags.name),
    (.spec["backend-addresses"] | length | tostring)
  ] | @tsv' | column -t

8  10.96.249.121:80  ClusterIP  default/demo  2

Backend count is 2; Cilium knows about them. Verify the eBPF map itself agrees with what the agent reports:

kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list | grep "10.96.249.121"

10.96.249.121:80/TCP (0)    0.0.0.0:0 (8) (0) [ClusterIP, non-routable]
10.96.249.121:80/TCP (1)    10.0.0.52:80/TCP (8) (1)
10.96.249.121:80/TCP (2)    10.0.0.167:80/TCP (8) (2)

service list is the agent's view of what it configured. bpf lb list reads the eBPF maps directly. They usually agree; when they diverge, the agent reports healthy backends the datapath never received. Here they match — the forwarding table is correct. Something else is dropping the packets.

Step 3 — check Hubble for packet drops

Cilium enforces NetworkPolicy directly in eBPF, so nothing shows up in iptables. Use Hubble:

kubectl exec -n default client -- curl -s --max-time 5 http://10.96.249.121/ > /dev/null
kubectl -n kube-system exec ds/cilium -- hubble observe --verdict DROPPED --last 20 | grep -E "client|demo"

Sep  9 08:07:46.753: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Sep  9 08:07:46.753: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) Policy denied DROPPED (TCP Flags: SYN)
Sep  9 08:07:47.777: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Sep  9 08:07:47.777: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) Policy denied DROPPED (TCP Flags: SYN)

Policy denied DROPPED. A NetworkPolicy is blocking the connection. policy-verdict:none INGRESS DENIED is the same verdict from a different angle. Either of these in hubble observe --verdict DROPPED points to NetworkPolicy as the cause.

Root cause

kubectl get networkpolicy deny-client-to-demo -o jsonpath='{.spec.ingress}'

[{"from":[{"podSelector":{"matchLabels":{"app":"not-client"}}}]}]

Ingress is only allowed from Pods labeled app: not-client. The client Pod doesn't qualify.

Debug flow summary

The two cases point to this sequence:

Run kubectl get endpoints (or kubectl get endpointslices from 1.33). <none> → check selector match and Pod readiness.
If Endpoints exist, run cilium-dbg service list -o json | jq and check the backend count. 0 → Cilium doesn't see the backends.
If backends exist but there's no connectivity, run hubble observe --verdict DROPPED. Policy denied → NetworkPolicy is the cause.

Tool mapping from kube-proxy to Cilium:

Old tool (kube-proxy)	New tool (Cilium)	What it shows
`iptables-save -t nat \	grep KUBE-SERVICES`	`cilium-dbg service list -o json \
{% raw %}`iptables-save -t filter \	grep `	backend count from `cilium-dbg service list -o json \
{% raw %}`conntrack -L`	`cilium-dbg bpf ct list global`	Connection tracking table
`tcpdump`	`hubble observe`	Per-flow packet visibility (see "Packet tracing")
(no equivalent)	`hubble observe --verdict DROPPED`	NetworkPolicy denials

Wrap-up

iptables-save returning 0 lines was not a broken cluster. kube-proxy doesn't exist, so nothing writes forwarding rules there — they moved to eBPF maps.

To read the forwarding table, use cilium-dbg service list -o json | jq. One command gets you the ClusterIP, Service name, and backend Pod IPs. The three-hop KUBE-SERVICES → KUBE-SVC → KUBE-SEP walk collapses to one line.

Parts of the troubleshooting workflow carry over unchanged. kubectl get endpoints works fine in a Cilium environment. After that, check cilium-dbg service list -o json | jq for backend count, and if backends are present but connectivity still fails, check hubble observe --verdict DROPPED for NetworkPolicy denials. The has no endpoints comment that kube-proxy wrote in the filter table is gone, but a backend count of 0 means the same thing.

Appendix — iptables evaluation cost grows with endpoint count

Measured while scaling demo replicas from 1 to 16. The kube-proxy column shows the number of KUBE-SVC chain rules; the Cilium column shows LB map backend slots.

replicas   kube-proxy SVC rules   Cilium backend slots
1          2                      1
2          3                      2
4          5                      4
8          9                      8
16         17                     16

Both grow linearly with endpoint count. The difference is lookup cost. The table shows equal growth in entry count; the asymmetry is in how those entries are evaluated per new connection.

kube-proxy evaluates KUBE-SVC chain rules top to bottom to pick a backend. Each rule's --probability check either hits or falls through to the next. When the last endpoint is selected, every rule before it was evaluated first. With 16 replicas, up to 16 evaluations happen. This only applies to the first packet of a new connection — subsequent packets are handled by conntrack and skip the rules entirely. The impact shows up in environments with high new-connection rates and many Services.

-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.10:80" -m statistic --mode random --probability 0.06250000000 -j KUBE-SEP-...
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.11:80" -m statistic --mode random --probability 0.06666666688 -j KUBE-SEP-...
...
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.9:80"  -j KUBE-SEP-...

Cilium computes random() % N to get a slot number and does one map lookup to find the backend. Slot 0 is the service master entry; slots 1+ are backends. With 16 Pods, it's still one lookup. This slot format is cilium-dbg bpf lb list output — it reads the eBPF maps directly, not the agent's in-memory state.

kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list | grep "10.96.210.178"

10.96.210.178:80/TCP (0)    0.0.0.0:0 (8) (0) [ClusterIP, non-routable]
10.96.210.178:80/TCP (1)    10.0.0.207:80/TCP (8) (1)
10.96.210.178:80/TCP (2)    10.0.0.48:80/TCP (8) (2)
...
10.96.210.178:80/TCP (16)   10.0.0.187:80/TCP (8) (16)

Migrating from ingress-nginx to Gateway API: a three-category approach to annotations

Kazu — Tue, 08 Sep 2026 13:02:00 +0000

The Kubernetes project announced the retirement of ingress-nginx. ingress-nginx retires in March 2026. No more patches, no more updates. No successor was named — just a recommendation to migrate to Gateway API.

The reason is straightforward: not enough maintainers. For years, effectively one or two people had kept the project running in their spare time. After the retirement announcement, there were plans to develop a successor controller (InGate) in collaboration with the Gateway API community, but no contributors came forward, and InGate was retired as well. No successor was named because none could be named.

Existing deployments won't break overnight. Helm charts and container images remain usable. But security patches stop. When new vulnerabilities are found, they'll go unfixed. The system keeps running while the risk accumulates slowly.

ingress-nginx had a kind of self-evidence: "if you need HTTP in Kubernetes, use this." Gateway API dissolves that.

This article covers how to choose a Gateway API implementation and how to migrate ingress-nginx annotations by sorting them into three categories — with before/after manifests throughout.

What "migrate to Gateway API" actually means

Gateway API is a Kubernetes specification. It is not software.

In the Ingress era, HTTP routing configuration looked like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service
            port:
              number: 8080

Apply this YAML and it worked. Assuming ingress-nginx was installed in the cluster — but since "using Kubernetes Ingress" and "using ingress-nginx" were nearly synonymous, that distinction rarely came up. Spec and implementation mapped almost one-to-one.

Gateway API is different. Writing an HTTPRoute does nothing unless you have separate software installed that reads those resources and actually routes requests.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-route
spec:
  parentRefs:
  - name: my-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: my-service
      port: 8080

Apply this YAML without a Gateway controller installed in the cluster, and nothing happens. Gateway API is a spec defining what resources exist — it's not a process that handles traffic.

"Migrating to Gateway API" meant picking an implementation and installing it. The question of which software was actually running — obvious in the Ingress world — is now something you have to decide yourself.

Too many implementations

"Pick one" sounds simple, but the options are many. In the Envoy-based family alone: Envoy Gateway, kgateway, Istio, Contour, Cilium. Outside Envoy: Traefik, Kong, NGINX Gateway Fabric. Cloud providers each offer their own. And all of them claim "Gateway API conformant."

It's tempting to think: "if they're all conformant, they'll behave the same within the standard, and I can switch later." That doesn't hold.

Three reasons "conformant" isn't uniform

First, conformance levels vary. Gateway API features are divided into three tiers: Core, Extended, and Implementation-specific. Core is the minimum every implementation must support. Extended is "recommended but optional." Implementation-specific is the open space for vendor extensions. Claiming "Gateway API conformant" only requires Core support. Extended and above vary across implementations.

Second, the features you actually need in production tend to land in vendor extensions. Auth, rate limiting, circuit breaking — these commonly belong to Extended or Implementation-specific. Envoy Gateway uses a SecurityPolicy CRD; Istio uses AuthorizationPolicy. The moment you write one of those resources, you're locked into that implementation.

Third, there's no default. When ingress-nginx was around, there was a clear first choice. Now there isn't. Every team is being forced to make that choice simultaneously, without a settled answer.

The first pick carries weight. The comparison below shows why.

Comparing the major implementations

	Envoy Gateway	Istio	Traefik	NGINX Gateway Fabric
Data plane	Envoy	Envoy	Traefik	nginx
Service mesh support	Partial	Full	None	None
Migration ease from ingress-nginx	Good	Moderate	Good	Excellent
Extended feature coverage	Excellent	Excellent	Good	Good
Development momentum	Excellent	Excellent	Good	Moderate

Istio is the only one with full service mesh support — east-west traffic control and sidecar injection included. The other three have no mesh capabilities. NGINX Gateway Fabric scores highest on migration ease because it shares the same nginx data plane. Istio scores lower because installing the entire service mesh stack is a prerequisite. The difference in Extended feature coverage becomes concrete once you read the Category B before/after examples. For development momentum, Envoy Gateway (CNCF incubating) and Istio (CNCF graduated) lead, while NGINX Gateway Fabric has a lower release cadence.

No recommendation here — the right answer depends on whether you already have a service mesh, how you weigh the annotation rewrite cost, and whether your team has Envoy operational experience. But before picking an implementation, there's work to do: inventory your current ingress-nginx annotations.

Sorting annotations into three categories

The before/after manifests and validation scripts used in this article are at shinagawa-web/ingress-nginx-to-gateway-api. before/ holds the ingress-nginx manifests, after/category-a/ the standard Gateway API equivalents, and after/category-b-envoy-gateway/ and after/category-b-istio/ the implementation-specific ones.

Envoy Gateway and Istio were chosen for the Category B manifests because they have broad coverage of features that can't be expressed in the standard spec — external auth, rate limiting, and so on. Traefik and NGINX Gateway Fabric can handle basic routing just as well, but their vendor extensions are less mature. Putting Envoy-based (Envoy Gateway) next to service-mesh-capable (Istio) also makes the architectural difference visible. Links to the relevant files appear at the end of each category section so you can run them locally as you read.

The goal is to inventory every annotation you have and sort each into one of three categories. The approach differs fundamentally by category. Note that Category C is evaluated line by line within the annotation, not annotation by annotation. A single configuration-snippet can contain lines that fall into Category A and lines that require Category B.

Category A: migrates directly to standard Gateway API

rewrite

A rewrite is the process of rewriting the URL path before the request reaches the backend. If a client hits /api/hello and the app should receive /hello, that's a rewrite. In ingress-nginx, you'd use regex capture groups.

# before/02-rewrite.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rewrite
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: echo-v1
            port:
              number: 8080

The pattern /api(/|$)(.*) captures the second group ($2) and forwards it as the path. A request to /api/hello puts hello into $2, delivering /hello to the backend.

Gateway API expresses the same thing with the URLRewrite filter in HTTPRoute. (URLRewrite is an Extended-tier feature — not Core. Verify your chosen implementation supports it; the four implementations in this article's comparison table all do.)

# after/category-a/02-rewrite.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: rewrite
spec:
  parentRefs:
  - name: eg
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /
    backendRefs:
    - name: echo-v1
      port: 8080

Read it as: replace the /api prefix with /. No regex — prefix replacement is explicit.

canary

Canary deployment means routing a small percentage of traffic to a new version, then gradually increasing it rather than doing an immediate cutover. In ingress-nginx, this required two Ingress resources: a primary route to v1 and a canary route to v2, with the canary's weight set in an annotation.

# before/03-canary.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-primary
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo-v1
            port:
              number: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo-v2
            port:
              number: 8080

Two Ingress resources define the same host: app.example.com and path /. Normally that's a conflict, but ingress-nginx treats any Ingress with canary: "true" as a weighted companion to the other. canary-weight: "20" sends 20% of traffic to echo-v2.

One routing decision split across two resources is the problem. Reading canary-primary alone gives you no visibility into the actual traffic split.

Gateway API fits it into a single HTTPRoute.

# after/category-a/03-canary.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: canary
spec:
  parentRefs:
  - name: eg
  rules:
  - backendRefs:
    - name: echo-v1
      port: 8080
      weight: 80
    - name: echo-v2
      port: 8080
      weight: 20

List v1 and v2 in backendRefs with weight ratios. 80:20 sends 80% to v1, 20% to v2. One resource, full picture.

The conversions in this category can be automated with the Ingress2Gateway CLI (1.0 GA). Worth using to reduce manual work.

before/02-rewrite.yaml → after/category-a/02-rewrite.yaml, before/03-canary.yaml → after/category-a/03-canary.yaml

Category B: requires implementation-specific extensions

external auth

External auth delegates authentication decisions to a separate service rather than the application itself. On each incoming request, the proxy asks the auth service whether to allow it. Only when the auth service responds OK does the request proceed to the app. Because the auth logic lives outside the app, the same auth service can cover multiple apps.

In ingress-nginx, two annotation lines did it.

# before/04-auth.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: auth
  annotations:
    nginx.ingress.kubernetes.io/auth-url: http://auth-service.default.svc.cluster.local/auth
    nginx.ingress.kubernetes.io/auth-response-headers: X-Auth-User
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /protected
        pathType: Prefix
        backend:
          service:
            name: echo-v1
            port:
              number: 8080

auth-url is the auth service endpoint. auth-response-headers lists headers from the auth service response to forward to the backend. When the auth service returns 200, the specified headers are added to the upstream request. Two annotation lines cover delegation and response propagation.

The sample repository includes a mock auth service in app/auth-service.yaml — a Python server that returns 200 for Authorization: Bearer valid-token and 401 for everything else.

With Envoy Gateway, HTTPRoute alone has no place to write auth configuration. You create a separate SecurityPolicy CRD — an Envoy Gateway-specific resource — and bind it to the target HTTPRoute.

# after/category-b-envoy-gateway/04-auth.yaml (excerpt)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: auth
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: protected
  extAuth:
    http:
      backendRefs:
      - group: ""
        kind: Service
        name: auth-service
        port: 80
      headersToBackend:
      - x-auth-user

targetRefs specifies which HTTPRoute the policy applies to — here, the HTTPRoute named protected. extAuth.http.backendRefs points to the auth service.

The intent is clearer than in ingress-nginx. Annotations were scoped to their Ingress resource, so tracking which route had which auth policy meant reading every Ingress. SecurityPolicy stands as an independent resource with its target made explicit via targetRefs.

To forward X-Auth-User from the auth service response to the backend, headersToBackend must be configured explicitly. Unlike ingress-nginx's auth-response-headers, Envoy Gateway does not forward auth response headers automatically — only the headers listed in headersToBackend are propagated.

Migrating to Istio changes the architecture. AuthorizationPolicy evaluates request attributes locally — headers, JWT claims, source principals — rather than calling out to an external service.

# after/category-b-istio/04-auth.yaml (excerpt)
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: auth
spec:
  selector:
    matchLabels:
      app: echo
      version: v1
  action: DENY
  rules:
  - when:
    - key: request.headers[authorization]
      notValues:
      - "Bearer valid-token"

action: DENY means "deny requests that match the condition." Requests where request.headers[authorization] is not Bearer valid-token are denied — only valid-token requests get through. Note that selector.matchLabels scopes the policy to the workload, not to the /protected path — the DENY rule applies to all traffic reaching the matched pods. To restrict it to a specific path, add to.operation.paths to the rule. Where Envoy Gateway's extAuth asks an external service for an OK/NG decision, Istio's AuthorizationPolicy evaluates the policy inside the cluster. The mechanism is fundamentally different from ingress-nginx's auth-url callback model, so migrating to Istio means rethinking the auth design.

AuthorizationPolicy evaluates request attributes locally. For logic that requires an external decision — like checking permissions against a database — Istio provides AuthorizationPolicy with action: CUSTOM plus an extensionProviders entry in meshConfig. This is the proper external auth delegation path in Istio, comparable in ergonomics to Envoy Gateway's extAuth. The EnvoyFilter route can also be used but writes Envoy internals directly.

Category A works the same way regardless of which implementation you choose. Category B changes fundamentally depending on which one you picked. This is the "operational features drift into vendor extensions" reality the comparison table pointed to: you can't form a migration plan for this category until you've chosen an implementation.

before/04-auth.yaml → after/category-b-envoy-gateway/04-auth.yaml / after/category-b-istio/04-auth.yaml

rate limit

ingress-nginx's rate limit was self-contained in annotations, but there's a trap: queueing behavior changes after migration.

# before/05-rate-limit.yaml (excerpt)
annotations:
  nginx.ingress.kubernetes.io/limit-rps: "2"
  nginx.ingress.kubernetes.io/limit-burst-multiplier: "1"

limit-rps: "2" allows at most 2 requests per second per IP. Requests over the limit don't get rejected immediately — they queue up to the limit set by limit-burst-multiplier. With limit-burst-multiplier: "1", the queue cap is limit-rps × 1 = 2. Once the queue fills, subsequent requests get a 503 immediately.

Envoy Gateway uses BackendTrafficPolicy. Like SecurityPolicy for external auth, it's a separate resource from HTTPRoute, with targetRefs pointing at the target.

# after/category-b-envoy-gateway/05-rate-limit.yaml (excerpt)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: rate-limit
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: rate-limited
  rateLimit:
    type: Local
    local:
      rules:
      - limit:
          requests: 2
          unit: Second

requests: 2, unit: Second expresses 2 requests per second, but there's no field corresponding to limit-burst-multiplier. The queueing behavior can't be reproduced.

Istio configures the token bucket directly via EnvoyFilter. There's no abstraction layer like BackendTrafficPolicy — you're writing Envoy internals directly.

# after/category-b-istio/05-rate-limit.yaml (excerpt)
token_bucket:
  max_tokens: 2
  tokens_per_fill: 2
  fill_interval: 1s

tokens_per_fill: 2, fill_interval: 1s replenishes 2 tokens per second (= 2 req/s). max_tokens: 2 is the burst cap — the maximum number of tokens the bucket can hold. This corresponds to ingress-nginx's limit-rps: "2", limit-burst-multiplier: "1". The algorithm differs from nginx's queuing model, but the behavior — 2 requests per second, burst cap of 2 — is equivalent. Whether reproducing limit-burst-multiplier is worth trading against implementation choice is a judgment call that varies by team.

before/05-rate-limit.yaml → after/category-b-envoy-gateway/05-rate-limit.yaml / after/category-b-istio/05-rate-limit.yaml

Category C: no equivalent

configuration-snippet

Probably the most useful annotation ingress-nginx had — and the most problematic.

nginx config has server blocks and location blocks. configuration-snippet lets you inject nginx directives directly into the location block, overriding or extending what ingress-nginx generates. It was effectively an escape hatch for anything the standard annotations couldn't express.

# before/06-configuration-snippet.yaml (excerpt)
annotations:
  nginx.ingress.kubernetes.io/configuration-snippet: |
    more_set_headers "X-Custom-Header: my-value";
    more_set_headers "X-Request-ID: $request_id";

more_set_headers is a directive from the headers-more module bundled with ingress-nginx, used to add or override response headers. "X-Custom-Header: my-value" adds a static header. "X-Request-ID: $request_id" attaches nginx's built-in variable $request_id — a unique ID auto-generated per request — as a response header.

Gateway API has no direct equivalent. But saying "no equivalent" oversimplifies it. Break down the annotation line by line, and the picture splits.

Adding a static header like X-Custom-Header migrates to HTTPRoute's ResponseHeaderModifier filter. That's standard spec — Category A.

# after/category-a/06-response-header.yaml (excerpt)
filters:
- type: ResponseHeaderModifier
  responseHeaderModifier:
    add:
    - name: X-Custom-Header
      value: my-value

Dynamic values like $request_id are a different story. Nothing in the Gateway API standard covers referencing nginx built-in variables. With Envoy Gateway, you can reproduce it by injecting a Lua script via the EnvoyPatchPolicy CRD — Category B.

# after/category-b-envoy-gateway/06-request-id.yaml (excerpt)
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: request-id-lua
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
  - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
    name: "default/eg/http"
    operation:
      op: add
      path: "/default_filter_chain/filters/0/typed_config/http_filters/0"
      value:
        name: envoy.filters.http.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          default_source_code:
            inline_string: |
              function envoy_on_request(request_handle)
                local req_id = request_handle:headers():get("x-request-id")
                if req_id then
                  request_handle:streamInfo():dynamicMetadata():set("lua_ns", "req_id", req_id)
                end
              end
              function envoy_on_response(response_handle)
                local meta = response_handle:streamInfo():dynamicMetadata():get("lua_ns")
                if meta and meta["req_id"] then
                  response_handle:headers():add("x-request-id", meta["req_id"])
                end
              end

Envoy generates an x-request-id header automatically on incoming requests. The Lua script reads it during request processing, stores it in metadata, then attaches it as a response header during response processing. nginx's $request_id becomes Envoy's request ID, but the intent — a unique per-request ID on the response header — is the same.

The same Lua logic works in Istio via EnvoyFilter, but the apply context and metadata namespace differ.

# after/category-b-istio/06-request-id.yaml (excerpt)
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: request-id-response-header
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.lua
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          default_source_code:
            inline_string: |
              function envoy_on_request(request_handle)
                local req_id = request_handle:headers():get("x-request-id")
                if req_id then
                  request_handle:streamInfo():dynamicMetadata():set("lua_metadata", "req_id", req_id)
                end
              end
              function envoy_on_response(response_handle)
                local meta = response_handle:streamInfo():dynamicMetadata():get("lua_metadata")
                if meta and meta["req_id"] then
                  response_handle:headers():add("x-request-id", meta["req_id"])
                end
              end

The Envoy Gateway version uses EnvoyPatchPolicy with JSONPatch to insert the Lua filter at the Listener level, with state stored under metadata namespace lua_ns. The Istio version uses EnvoyFilter with applyTo: HTTP_FILTER and match.context: GATEWAY, with state stored under lua_metadata. The Lua logic is the same, but the configuration layer differs.

What truly has no equivalent in Category C is nginx-specific functionality that EnvoyPatchPolicy can't reach — directives that depend deeply on nginx internals, or snippets that assume concepts Envoy doesn't have. In practice, most configuration-snippet contents, when broken down line by line, land in Category A or Category B. The annotation as a whole isn't Category C — each line has to be read and evaluated individually.

before/06-configuration-snippet.yaml → after/category-a/06-response-header.yaml (X-Custom-Header) / after/category-b-envoy-gateway/06-request-id.yaml (X-Request-ID, Envoy Gateway) / after/category-b-istio/06-request-id.yaml (X-Request-ID, Istio)

Standardization didn't make migration easier

Gateway API gave us a unified spec, so "the interface is now shared." But more implementations means more decisions. The Core tier is consistent across implementations. But operational features drift into vendor extensions, so locking in to an implementation comes after, not before.

The frustration with ingress-nginx — "settings are scattered across annotations, hard to see the structure" — was a real problem. Gateway API was designed to address it. But from the migration side, new decisions piled on: which software to pick, which annotations map to the standard and which need vendor extensions and which get dropped.

"Migrate to Gateway API" wasn't a work order. It compressed two separate jobs into one phrase: choose an implementation, then classify each annotation into one of three categories and handle each accordingly.

Inspect HTTPS Traffic Without a Proxy — Built with Go and eBPF

Kazu — Tue, 01 Sep 2026 12:01:00 +0000

I built a CLI in Go that reads HTTPS traffic as plaintext — no CA certificate installation, no proxy, and no changes to the target app.

The original motivation was simple: spinning up mitmproxy every time I needed to debug something added one too many steps. I found an approach using eBPF, wrapped it into a CLI, and called it tinytap. This article is about how I built it.

Here's what it looks like in action.

Two terminals side by side. The left sends one HTTPS request; the right shows tinytap printing it in plaintext — request line, headers, Authorization token, JSON body, all of it.

What it captures

tinytap targets apps like this — ordinary code that hits an API, in this case the OpenAI API from Python:

from openai import OpenAI

client = OpenAI(api_key="sk-...", base_url="https://127.0.0.1:8443/v1")

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "what is a uprobe"}],
)

And here's what tinytap captures and outputs:

POST /v1/chat/completions HTTP/1.1
Host: 127.0.0.1:8443
Authorization: Bearer sk-demo-xxxx
Content-Type: application/json
User-Agent: OpenAI/Python 2.53.0
...
Request body (decoded): {"messages":[{"role":"user","content":"what is a uprobe"}],"model":"gpt-4o-mini"}

Request line, all headers, the Authorization: Bearer token, and the JSON body. The token in the gif is sk-demo-xxxx — a dummy — but when you point tinytap at a real app, the real token shows up exactly like that. No reformatting.

Nothing on the app side was touched. One command: tinytap.

How eBPF makes this work

A uprobe is a kernel mechanism for dynamically hooking into user-space functions. tinytap attaches a uprobe to SSL_write in libssl. The moment that function is called, tinytap reads the argument buffer — which is still plaintext at that point. The data travels through a ring buffer to user space and gets displayed in the TUI. No modifications to the app or to libssl.

This is not breaking TLS

One thing worth clearing up.

tinytap only looks at the buffer your own process hands to the TLS library — the moment before it gets encrypted. It is not a vulnerability that lets you intercept traffic on the network. What flows over the wire is properly encrypted. Other processes and other people's traffic are not accessible.

App
  │  plaintext
  ▼
TLS library  ◀── tinytap looks here
  │  ciphertext
  ▼
Network
  │  ciphertext
  ▼
API server

It's the same idea as curl printing its own request to your terminal.

Why Go

A few reasons I chose Go.

First, cilium/ebpf. It's a mature library for managing eBPF programs from Go — it absorbs the boilerplate of talking to the kernel, so I could focus on the actual hook logic.

Second, binary distribution. go build produces a single binary you can hand to someone and it just runs. No runtime or external dependencies to install. That matters a lot when you're shipping an open-source CLI.

Third, the TUI. I used Bubble Tea to build a live event display. Even when a burst of eBPF events comes in from the kernel, Go's goroutines and channels handle concurrent ingestion and rendering without trouble.

Try it

tinytap uses eBPF and is Linux-only. On Mac or Windows, set up a Linux container with Docker.

curl -fsSL https://raw.githubusercontent.com/shinagawa-web/tinytap/main/scripts/install.sh | sh
sudo setcap cap_dac_read_search,cap_perfmon,cap_bpf,cap_sys_admin=eip $(command -v tinytap)
sudo chmod +x $(ldconfig -p | grep libssl.so | awk '{print $NF}' | head -1)
tinytap

setcap grants the binary the capabilities it needs for eBPF. The chmod +x step is needed on Debian and Ubuntu because libssl.so.3 ships without the execute bit set there. With both done, tinytap runs without sudo.

If something isn't working, run tinytap doctor. It checks your kernel version, BTF availability, capabilities, and the libssl execute bit, and tells you exactly what's missing.

$ tinytap doctor
tinytap doctor — v0.6.1

[OK      ] kernel version               6.1.0 (>= 5.8 required)
[OK      ] kernel BTF                   present
[OK      ] cap_dac_read_search          present
[OK      ] cap_perfmon                  present
[OK      ] cap_bpf                      present
[OK      ] cap_sys_admin                present
[OK      ] cap_syslog                   not needed on arm64
[OK      ] perf_event_paranoid          4
[OK      ] unprivileged_bpf_disabled    2
[OK      ] RLIMIT_MEMLOCK               soft=unlimited hard=unlimited
[OK      ] syscall tracepoints          available
[OK      ] BPF dry-run load             ok
[DEGRADED] libssl execute bit           /usr/lib/x86_64-linux-gnu/libssl.so.3 not set
    Affects: TLS capture only. Plaintext HTTP capture is unaffected.
    Fix:     sudo chmod +x /usr/lib/x86_64-linux-gnu/libssl.so.3

12 ok, 1 degraded, 0 blocking, 0 info

Python, Ruby, PHP, and Node.js all work. Go apps are out of scope — the standard library's crypto/tls doesn't go through OpenSSL, so there's no SSL_write to hook. tinytap currently understands HTTP/1.1 only.

Targeting implementations that bypass OpenSSL — like Go's crypto/tls or Rust's rustls — would require attaching the uprobe at a different layer. Where exactly that should be is still an open question.

Verified compatibility details are in the docs:

Server compatibility — how much of the response body tinytap can surface for nginx, Caddy, Uvicorn, and other servers
TLS compatibility — the list of supported TLS implementations; OpenSSL-based stacks (Python, nginx, curl, Node.js, etc.)
Platform support — tested Linux distributions and kernel versions; kernel 5.8 or later is required (Ubuntu 20.04's default kernel 5.4 is too old)

Wrapping up

I built a CLI in Go that reads HTTPS plaintext using eBPF uprobes — no CA certificate, no proxy needed. By hooking into SSL_write in libssl, it works with any OpenSSL-based stack: Python, Ruby, Node.js, and others, regardless of language.

Code is on GitHub. Environment requirements and compatibility details are in the docs.

Nginx, Caddy, Traefik, or HAProxy: How to Pick the Right Reverse Proxy for Your Stack (2026)

Kazu — Tue, 25 Aug 2026 13:02:00 +0000

Have you ever heard "Caddy is great" and wondered what actually makes it different from Nginx? Or maybe you were evaluating Traefik for a Kubernetes setup, went down a rabbit hole of comparison articles, and gave up halfway through?

Part 1 ("For People Who've Configured a Reverse Proxy but Can't Explain How It Works") covered how reverse proxies work under the hood. Part 2 — this article — focuses on tool selection: the design philosophies behind Nginx, Caddy, Traefik, and HAProxy, and a mental model for deciding which one fits your use case. If you haven't read Part 1, one baseline is enough: a reverse proxy is software that sits between the browser and the app server, relaying traffic between them.

The four tools and what each one is

A quick one-liner for each:

Nginx: the most widely used general-purpose web server and reverse proxy
Caddy: simple config file, automatic TLS certificate management built in
Traefik: designed for dynamic routing config in container environments
HAProxy: a full-featured load balancer that operates at both the TCP (L4) and HTTP (L7) levels

All four can do load balancing, reverse proxying, and TLS termination. Nginx can handle load balancing. HAProxy can work as a reverse proxy. Capability overlap is the point — what differs is design emphasis, not capability ceiling. That emphasis is what drives the choice.

Consider what "emphasis" means in practice. Nginx was built as a general-purpose machine: serve static files, reverse proxy, act as a load balancer — all from one piece of software. Traefik was born from a specific frustration: having to rewrite config files every time a container starts or stops. The starting point differs, so even when both do the same job, one is working in its natural mode and the other is being stretched.

Read this article through that lens: where does each tool's center of gravity sit?

One tool you won't see in this comparison is Envoy. Envoy has native support for circuit breaking, distributed rate limiting, and gRPC load balancing, and it's usually encountered in the context of service meshes like Istio. The configuration complexity and operational overhead are in a different league. For someone who's only used Nginx, it's too big a jump. Envoy does appear in the bonus section at the end — not as "here's the next step to consider," but as "here's what these four can't reach."

Nginx vs Caddy: what's actually different

Put the minimal HTTPS configuration for each side by side.

The Nginx version:

server {
    listen 443 ssl;
    server_name example.com;

    ssl_certificate /etc/ssl/certs/example.com.crt;
    ssl_certificate_key /etc/ssl/private/example.com.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The equivalent Caddyfile:

example.com {
    reverse_proxy localhost:3000
}

The line count difference might look like Caddy is skipping things — it isn't. With Caddy, TLS certificate retrieval and renewal happen automatically. It fetches a certificate from Let's Encrypt and renews it automatically well before the 90-day expiry, with no manual steps needed. There's no ssl_certificate line because you don't supply the cert file yourself. There are no proxy_set_header lines because Caddy forwards those headers by default.

If you've ever wrestled with Nginx TLS config, Caddy's automatic TLS will be a real relief. Especially if you've had a service go down because a certificate renewal slipped through the cracks.

That said, Caddy's simplicity cuts both ways. The convenient defaults make it harder to reason about what's happening when you need to override something. Nginx's verbosity is a feature: every directive is explicit, which makes troubleshooting more straightforward.

The question to ask when choosing: do you want to control TLS yourself, or do you want to hand it off?

Caddy does have cases where it doesn't fit. Internal services — those not exposed to the internet — won't work with ACME-based automatic TLS. Let's Encrypt challenges require the domain to be publicly reachable, so for a service on an internal DNS, you'd need extra DNS-01 challenge configuration to get Caddy's automatic TLS working. Going in with the assumption that "Caddy means TLS is automatic" will cause problems for internal services.

There's also a debugging gap. caddy reload and caddy validate error messages can be less specific than Nginx's. Nginx's nginx -t gives you the exact line number of the problem; Caddy sometimes only tells you "configuration is invalid." For a team with deep Nginx experience, troubleshooting an unfamiliar tool takes longer than it should.

Why Traefik pairs well with Kubernetes

Traefik's starting point was: "I don't want to rewrite config files every time a container is added."

With Nginx, you write this:

upstream backend {
    server app1:3000;
    server app2:3000;
}

Adding app3 means opening the config file, adding server app3:3000;, and reloading Nginx. Manual every time.

Traefik works differently. With Docker, you attach labels to the container at startup and it joins the routing automatically.

services:
  app:
    image: my-app
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`example.com`)"
      - "traefik.http.services.app.loadbalancer.server.port=3000"

The moment that container starts, Traefik reads the Docker labels and adds the route. When the container stops, the route disappears. No config reload needed.

In Kubernetes, the same thing happens through Ingress annotations. Deploy a new service with kubectl apply, and Traefik detects it and updates routing. The "update config files on every deploy across 20 containers" workflow goes away.

If static config files are enough, Nginx works fine. When containers come and go frequently, that dynamic config automation is the reason to choose Traefik.

Going deeper into Kubernetes: Traefik has its own CRD called IngressRoute. It's more expressive than the standard Ingress resource, but it means writing Traefik-specific config into your K8s manifests. Nginx Ingress Controller is built on the standard Ingress resource and uses annotations to control fine-grained behavior.

Which you choose depends on whether you'd rather write in the proxy's config language or stay close to K8s manifests. If your team already knows Nginx, the Nginx Ingress Controller has a lower learning curve. If your service churn is high and you want routing definitions managed as code inside K8s, Traefik's IngressRoute is where it earns its keep.

Why HAProxy gets chosen as a dedicated load balancer

Ask the question "Nginx can load balance too, so why choose HAProxy?" and the answer surfaces.

A health check in HAProxy config looks like this:

backend api_servers
    balance roundrobin
    option httpchk GET /health HTTP/1.1
    http-check expect status 200
    default-server inter 2s fall 3 rise 2 maxconn 30
    server app1 10.0.0.1:3000 check
    server app2 10.0.0.2:3000 check

inter 2s fall 3 rise 2 means: check every 2 seconds, mark down after 3 consecutive failures, mark up after 2 consecutive successes. Nginx can do something similar, but HAProxy lets you express this level of control directly in the config.

Where HAProxy really stands out is observability — seeing what's happening in real time. The built-in stats dashboard shows connection counts, error rates, and latency per backend, live. If one backend starts spiking in latency because it's running out of memory, you can dig through Nginx access logs to find the cause eventually. Opening HAProxy's dashboard makes it immediately obvious which backend has the elevated error rate. When you're handling high traffic and need to know instantly which backend is the bottleneck, that data matters.

HAProxy also handles both TCP (L4) and HTTP (L7) load balancing. Beyond HTTP apps, you can balance database connections and manage WebSocket traffic in config.

HAProxy makes sense when traffic is heavy and you need fine-grained control over backend behavior. Reaching for it on a simple reverse proxy setup means absorbing configuration complexity without much benefit.

Picking one for your situation

Mapping the discussion to four scenarios:

Scenario	Choice	Main reason
Simple VPS + TLS termination	Caddy	Automatic TLS removes certificate management overhead
Small Docker microservices (stable topology)	Nginx	Proven track record, large community, explicit config
Small Docker microservices (high churn)	Traefik	Routing updates automatically as containers start and stop
Kubernetes cluster	Traefik / Nginx Ingress	Depends on what the team already knows
High-traffic load balancer	HAProxy	Health check granularity and built-in observability

For a simple VPS with TLS termination, Caddy is the pick. Offloading certificate management is a meaningful gain, and the short config file is easy to maintain. It suits personal projects and small services.

For small Docker-based microservices, Nginx or Traefik are both candidates. If the container count is stable and doesn't change much, Nginx is plenty. If you're adding and removing services regularly, Traefik's dynamic config earns its place.

For Kubernetes, Traefik or Nginx Ingress Controller. Traefik has a head start on K8s integration, but Nginx Ingress Controller is mature and fully viable. If your team has deep Nginx knowledge, there's no good reason to switch to Traefik just because it's the trendy choice.

For a high-traffic load balancer, HAProxy. That's where its design philosophy — fine-grained control and first-class observability — actually comes into play.

My last personal choice was Caddy on a VPS. The reason was simple: I'd taken a service down once because I forgot to renew a certificate. Since switching to Caddy, I haven't thought about certificates once. That alone was worth it.

Config files and benchmark

Working examples for all four proxies are in shinagawa-web/reverse-proxy-bench — each proxy has its own directory with a config file and a docker-compose.yml. Run any of them locally with make bench-nginx (requires Docker and k6).

The benchmark runs 60 seconds at 20 concurrent virtual users against a minimal Go backend (/ping → 200 OK), with default configurations:

Proxy	req/s	p50 (ms)	p95 (ms)	p99 (ms)
HAProxy	10,762	1.53	3.45	4.89
Traefik	7,946	2.15	4.62	6.48
Caddy	7,535	2.28	4.93	6.81
Nginx	4,386	4.37	5.43	6.83

These results reflect default configurations on a 2-vCPU GitHub Actions runner. Nginx in particular has significant headroom — worker_processes auto alone would narrow the gap considerably. The intent isn't to declare a winner, but to show how the design emphasis of each tool translates into behavior under the same load.

Bonus: what these four can't do

For intermediate and above: the areas where all four have the same blind spot.

Distributed rate limiting

Nginx, Caddy, Traefik, and HAProxy all have rate limiting, but all of them apply limits per instance. Scale out to four instances and your limit effectively quadruples. A "100 requests per second" rule becomes "400 requests per second" in practice. If you hit a situation right after a scale-out where rate limiting suddenly seems broken, the connection to instance count isn't always obvious.

You can share state via an external store like Redis — each proxy instance increments a Redis counter on every request and returns 429 when the threshold is hit. That turns Redis into a single point of failure, so you also need an availability design for Redis. Every request that touches Redis also adds the round-trip latency — typically 0.5–2ms on a local network, but under traffic spikes Redis itself can become the bottleneck, slowing the proxy down across the board.

The tools that provide this natively are Kong and Envoy. Kong's Rate Limiting Advanced plugin uses a hybrid approach: local counters per instance, synchronized to a shared store on a configurable interval. Requests don't need a Redis write on every hit, which reduces the latency overhead significantly. Envoy is designed to integrate with a dedicated external rate limit service from the start.

Circuit breaking

Suppose an upstream service starts responding slowly. The proxy holds connections open until timeout, and new requests keep trying to connect. The connection pool fills with waiting connections, other backends start backing up too, and failures cascade through the whole system. That's a cascading failure.

A circuit breaker cuts that chain. When it detects consecutive failures to an upstream, it stops sending new requests there and returns errors immediately. Envoy has this natively via outlier_detection — setting consecutive_5xx: 5 and base_ejection_time: 30s is enough to get it working. None of these four have an equivalent. Traefik's circuit breaker plugin uses similar configuration, but it operates at the route level rather than the stream level, which makes it less effective for long-lived streams like gRPC where a single stream can hold a backend connection open for minutes.

Native JWT verification / OIDC

Traefik and Caddy have plugin support for this, but many of the available plugins are community-maintained. How they handle JWKS endpoint caching and key rotation varies, and some don't document their behavior clearly. If a signing key rotates and the plugin doesn't refresh its cache in time, valid tokens start getting rejected. Nginx requires Lua or the NJS module. HAProxy is in a similar position.

If you want to handle authentication at the proxy layer reliably, Kong or Envoy are the more practical choices. Kong's JWT plugin caches JWKS with configurable TTL and handles key rotation predictably — which matters when authentication is in the critical path.

gRPC load balancing

With HTTP/1.1, a new connection is opened per request (or closed quickly), so the proxy distributes at the request level. gRPC runs over HTTP/2, which multiplexes requests over a single long-lived connection.

Put Nginx in front of a gRPC backend as a load balancer, and Nginx maintains one persistent connection to each backend. Those connections stay alive for a long time. Stream distribution becomes uneven, and specific backends end up with concentrated load. You can have 10 backends but effectively route to only 1 or 2 of them.

Nginx can partially work around this with keepalive on the upstream block combined with HTTP/2 proxying, but stream-level routing — deciding per-request which backend to send to — remains limited. Traefik opens multiple HTTP/2 streams per backend connection and rebalances more aggressively than Nginx's static upstream model, so load spreads more evenly across backends. Envoy is built gRPC-first, with stream-level load balancing as a baseline assumption.

You've Configured a Reverse Proxy, But Can You Explain Why It Works? (Part 1)

Kazu — Thu, 20 Aug 2026 12:01:00 +0000

You wrote proxy_pass http://localhost:3000;, checked it in the browser, saw it working, and called it done. But you couldn't actually explain why it works. What's happening between nginx and the app? Where is the request going? You probably couldn't answer that with any confidence.

Getting something working and understanding how it works are two different things. This article (Part 1) is about the how. Once you have that, the next question becomes which tool to use. In Part 2, I'll break down the design philosophies behind nginx, Caddy, Traefik, and HAProxy, and give you a framework for choosing the one that fits your stack.

Where the browser is actually connecting

When a browser sends a GET to https://api.example.com/users, it resolves api.example.com via DNS and opens a TCP connection to port 443 at that IP. Who's on the other end of that connection?

In a setup with a reverse proxy, it's not the app server — it's the proxy.

Browser  ──[TCP connection]──→  nginx (443)  ──[separate TCP connection]──→  App (3000)
         ←───────────────────                ←──────────────────────────────

This is the key point. The browser and the app server are not talking directly. The proxy holds two separate connections and shuttles requests between them. That's what "split communication" means: the browser has no idea what the app server's IP address is, what port it's on, or that it even exists.

This split is also the root cause of several problems I'll get to later. The reason the browser's IP disappears from the app's perspective, and the reason the Host header gets rewritten — both trace back to this structure.

How it differs from a forward proxy

"Proxy" is an overloaded term. Forward proxy and reverse proxy are easy to mix up. Both relay traffic, but they stand in for opposite sides, and they hide opposite things.

A forward proxy acts on behalf of the client. You see it in corporate networks, where internal machines can't go directly to the internet — they route through a proxy server. The destination server sees the proxy, not the individual machines behind it. What's hidden is the client.

Internal PC  ──→  Forward proxy  ──→  External server
                                       ↑
                             Internal PC is not visible

A reverse proxy acts on behalf of the server. The client (browser) connects to the proxy without knowing how many app servers are behind it or how they're arranged. What's hidden is the server infrastructure.

Browser  ──→  Reverse proxy  ──→  App server
                                   ↑
                         Server is not visible

Forward proxies hide the client; reverse proxies hide the server. That distinction comes back when we look at what disappears when you add a proxy.

What happens before a request reaches the app

Between the browser sending a request and the app server receiving it, the proxy does several things. Let's trace through a concrete nginx config.

server {
    listen 443 ssl;
    server_name api.example.com;

    location /users {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

You've probably written something like this. Let's go through each line and why it's there.

proxy_set_header Host $host; — The browser's HTTP request includes a header like Host: api.example.com. When the proxy forwards that request to the app server, the Host header gets rewritten to localhost:3000 by default, because that's the host of the connection the proxy opened. If your app needs to know the original domain — for virtual host routing, or to construct a redirect URL — it needs api.example.com, not localhost:3000. This line passes the original value through explicitly.

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; — X-Forwarded-For (XFF) is a header that records the list of IP addresses a request has passed through. When the proxy forwards a request to the app, the connection comes from the proxy, not the browser. From the app's perspective, the remote IP is the proxy's address. To tell the app what the original client IP was, the proxy adds this header. $proxy_add_x_forwarded_for is an nginx variable that appends to an existing XFF header if one is present, or creates a new one with the connecting IP if not.

proxy_set_header X-Forwarded-Proto $scheme; — When the proxy receives HTTPS on port 443 and forwards it to the app as HTTP, the app has no way of knowing the original request came over HTTPS. X-Forwarded-Proto carries that information.

Each of those settings exists because of the connection split from the first section.

What problems a reverse proxy solves

Consider what happens if there's no proxy and the app server has to handle everything itself.

To serve over HTTPS, you need to manage a server certificate and handle TLS handshakes. A Node.js app doing that on its own looks like this:

const https = require('https');
const fs = require('fs');

const options = {
  key: fs.readFileSync('/etc/ssl/private/server.key'),
  cert: fs.readFileSync('/etc/ssl/certs/server.crt'),
};

https.createServer(options, app).listen(443);

The app needs to know the certificate file paths. Every time you renew the certificate, you restart the app. If you're using Let's Encrypt for automatic 90-day renewals, that means the app restarts on every renewal cycle. Cipher suite selection and protocol version management live in your app code too — every time TLS 1.0/1.1 gets deprecated somewhere, you touch the app.

When nginx handles TLS termination (ending the encrypted connection with the client at the proxy), your app code becomes:

// Only needs to speak HTTP
const http = require('http');
http.createServer(app).listen(3000);

Certificate management and TLS configuration stay in the nginx config file. The app server speaks plain HTTP and never needs to restart because a cert was renewed.

When traffic grows and a single app server can't keep up, you add more. But if browsers connect to app servers directly, there's no way to spread requests across the new instances. With a proxy, you get load balancing across multiple upstreams:

upstream api_servers {
    least_conn;  # send to the server with fewest active connections
    server 10.0.0.1:3000;
    server 10.0.0.2:3000;
    server 10.0.0.3:3000;
}

least_conn sends each request to whichever server currently has the fewest active connections. It distributes load more evenly than the default round-robin behavior when response times vary — slow requests don't pile up on one server.

Routing /api/ requests to the backend cluster and /static/ requests to a file server is also more naturally expressed at the proxy layer:

server {
    listen 443 ssl;
    server_name api.example.com;

    location /api/ {
        proxy_pass http://api_servers;
    }

    location /static/ {
        proxy_pass http://file_server;
    }
}

The routing logic stays out of your app code. Where each request goes is visible in the nginx config.

These are usually described as "reverse proxy features," but what they really are is work that doesn't belong in the app server. The proxy takes on what would otherwise be scattered across every instance, and the app gets to stay focused on what it's actually for.

What disappears when you add a proxy

After adding a reverse proxy, you look at your app's access logs and every IP is 127.0.0.1. This trips people up. I was one of them.

Before the proxy:

# Before proxy
203.0.113.45 - - [12/Aug/2026:10:00:01 +0000] "GET /users HTTP/1.1" 200 512

After adding the proxy, without XFF headers configured:

# After proxy (no XFF config)
127.0.0.1 - - [12/Aug/2026:10:00:01 +0000] "GET /users HTTP/1.1" 200 512

The reason is straightforward: the app's connection comes from the proxy, and if the proxy and app are on the same server, that's the loopback address. What the app sees is "the IP of whatever connected to me," not "the browser's IP."

The fix is to read the X-Forwarded-For header that nginx is attaching:

# Value available when the app reads XFF
X-Forwarded-For: 203.0.113.45

But XFF is a header the client can freely manipulate, so trusting the wrong source means your IP can be spoofed.

If you don't notice that the log format changed after setting up the proxy, debugging gets confusing fast. If your app does any IP-based access control and isn't reading XFF, every request looks like it's coming from the same address.

Wrapping up

Everything in this article comes back to one thing: the communication is split into two separate TCP connections. Because of that split, the Host header has to be passed through explicitly, the client IP disappears, and TLS termination at the proxy becomes a viable pattern. Every line in that config exists because of this split. Which tool to use for the proxy is Part 2.

If what's in this article has stuck, you should be able to answer something like: "The proxy holds two TCP connections and splits the communication. That's why the Host header gets rewritten and the client IP disappears. TLS ends at the proxy." You can now explain what's happening between nginx and your app.

curl works, your app gets 400: read the exact request your SDK sent

Kazu — Tue, 18 Aug 2026 13:01:00 +0000

curl returns 200. The same request from your app returns 400. You followed the docs. The parameters look right. It still doesn't work.

When that happens, what you really want to know is what your app actually sent. But that's the one thing you can't see. The SDK assembles the final request below your code, and TLS encrypts it before it hits the wire — capture the traffic and you get ciphertext.

This post is about seeing that plaintext with one command, no CA certificate install, no proxy, no changes to your app. The tool is tinytap, something I built for exactly this. Here's what it looks like.

curl works, the app gets 400

Here's a concrete example of how this goes wrong. You're hitting a payment API that expects an integer amount in cents. Your code passes 1000. The app returns 400 invalid integer: amount. You hardcode the same value in curl and it goes through fine.

The usual suspects come to mind. Double-encoded body? Type coercion somewhere? Wrong Content-Type? Missing token? All plausible, none confirmable. Your app's logs show the 1000 you passed into the SDK — not the final form that went over the wire.

If you could see the request body with "amount": "10.00" — the value that silently became a string somewhere — you'd have exactly what you need to compare what you meant to send against what you actually sent.

The SDK's final form never appears in any log

The obvious move is to check the logs. But whatever you print or pass to your logger is the value before the SDK gets it. The SDK adds headers, serializes the body, and possibly retries or signs the request — all below the layer your logs can see. You get the intent, not the result.

Wireshark is another option. But the connection is TLS, so you get ciphertext. Decoding that requires setting up SSLKEYLOGFILE and a decryption step, and not every client supports it.

mitmproxy gives you plaintext — but it requires installing a CA certificate in your trust store, configuring your app to route through the proxy, and some SDKs ignore proxy environment variables entirely. Installing a CA cert on your machine is a heavy setup just for debugging.

So there was a gap: seeing the actual plaintext, without touching the app, without installing a certificate, without a proxy in the middle.

Peek at the plaintext before it gets encrypted

That's what tinytap does. The gif above shows it: run your app on the left, and the request's plaintext appears on the right in tinytap.

The left side is running this:

from openai import OpenAI

client = OpenAI(api_key="sk-...", base_url="https://127.0.0.1:8443/v1")

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "what is a uprobe"}],
)

And on the right:

POST /v1/chat/completions HTTP/1.1
Host: 127.0.0.1:8443
Authorization: Bearer sk-demo-xxxx
Content-Type: application/json
User-Agent: OpenAI/Python 2.53.0
...
Request body (decoded): {"messages":[{"role":"user","content":"what is a uprobe"}],"model":"gpt-4o-mini"}

The request line, every header, the Authorization: Bearer token, the JSON body — exactly what the SDK built and sent. The User-Agent: OpenAI/Python line confirms this is real bytes from the real SDK, not something I reconstructed. No proxy, no certificate, no changes to the app. The token in the gif is a dummy (sk-demo-xxxx), but when you point tinytap at your own app, your real token appears in plaintext just like this.

Your code on the left, what it actually sent on the right. The moment you see both side by side, the "I thought I sent X" question is answered on the spot.

This is not breaking TLS

Worth stating upfront before anyone raises an eyebrow: this is not a vulnerability that lets you intercept traffic.

tinytap looks at the buffer your process hands to the TLS library, just before encryption — and the buffer that comes back out, just after decryption.

App
  │  plaintext
  ▼
TLS library  ◀── tinytap looks here
  │  ciphertext
  ▼
Network
  │  ciphertext
  ▼
API server

It hooks into your process from the inside, at the moment the plaintext is still plaintext. TLS itself is untouched. Other people's traffic is not accessible. This isn't about someone with root on your machine — it's about the inside of your own process. It's the same idea as curl printing its own request to your terminal. The ciphertext on the wire is untouched. All you're doing is looking at it from your own side, one step before encryption.

Try it yourself

"Plaintext without a certificate or proxy" sounds like it might be overselling it. So here's how to reproduce it yourself. tinytap uses eBPF, which means Linux only. On Mac or Windows, spin up a Linux container with Docker.

# install
curl -fsSL https://raw.githubusercontent.com/shinagawa-web/tinytap/main/scripts/install.sh | sh

# grant the capabilities needed for eBPF and TLS capture
sudo setcap cap_dac_read_search,cap_perfmon,cap_bpf,cap_sys_admin=eip $(command -v tinytap)

# set the execute bit on libssl (Debian/Ubuntu ships it without one)
sudo chmod +x $(ldconfig -p | grep libssl.so | awk '{print $NF}' | head -1)

tinytap

setcap grants the binary the capabilities it needs for eBPF and TLS capture. chmod +x sets the execute bit on libssl — Debian and Ubuntu ship libssl.so.3 without it by default, so this line is required there. With both steps done, tinytap runs without sudo. Run your app normally and the requests show up.

If something doesn't work, run tinytap doctor. It checks your kernel version, BTF availability, capabilities, and the libssl execute bit, and tells you exactly what's missing.

$ tinytap doctor
tinytap doctor — v0.6.1

[OK      ] kernel version               6.1.0 (>= 5.8 required)
[OK      ] kernel BTF                   present
[OK      ] cap_dac_read_search          present
[OK      ] cap_perfmon                  present
[OK      ] cap_bpf                      present
[OK      ] cap_sys_admin                present
[OK      ] cap_syslog                   not needed on arm64
[OK      ] perf_event_paranoid          4
[OK      ] unprivileged_bpf_disabled    2
[OK      ] RLIMIT_MEMLOCK               soft=unlimited hard=unlimited
[OK      ] syscall tracepoints          available
[OK      ] BPF dry-run load             ok
[DEGRADED] libssl execute bit           /usr/lib/x86_64-linux-gnu/libssl.so.3 not set
    Affects: TLS capture only. Plaintext HTTP capture is unaffected.
    Fix:     sudo chmod +x /usr/lib/x86_64-linux-gnu/libssl.so.3

12 ok, 1 degraded, 0 blocking, 0 info

tinytap only speaks HTTP/1.1 — gRPC and HTTP/2 binary frames aren't readable as-is. Go is also out of scope: its crypto/tls bypasses OpenSSL entirely. Python, Ruby, PHP, and Node.js all work out of the box. The language is a quick guide: Python / Ruby / PHP / Node.js, you're good; Go, you'll need a different approach.

Instead of the self-signed local server in the demo, you can point tinytap at whatever API-connected app you normally run. The Authorization header and body that show up are what you need to debug "curl works, app gets 400."

Which SDK has bitten you? Silent retries, double-encoded bodies, headers disappearing for no reason — if any of that sounds familiar, it's probably the kind of thing that resolves the moment you can see what was actually sent.

So how does it actually see the plaintext?

This post has stayed focused on what tinytap shows, not how it works. The mechanism — eBPF and uprobes, hooking into the TLS library boundary — is a story for another post.

The server returned 400. It's telling you the request was wrong. You don't know what's wrong because you can't see what your code actually sent. If you could see it, you'd just compare what you meant to send against what went out. tinytap is built for that.

nginx doesn't run your config top to bottom: two traps with rewrite, try_files, and if

Kazu — Tue, 11 Aug 2026 13:01:00 +0000

You write a rewrite, then a try_files right under it, and you read it as "the rewrite on top runs first, the try_files below runs after." You put an if inside a location and read it as "when the condition is true, this stuff gets added to the outer config." Both readings trace the config file straight down, line by line, in the order it's written. And both have burned me.

Let me show you the two symptoms first. In each case the config reads cleanly, yet the behavior doesn't line up. Why that happens — the order in which nginx processes a request — is what this post is about.

Symptom 1: rewrite and try_files side by side, and you get a 404

In some location you want to rewrite the request once, then fall back to a static file. The naive version looks like this:

location /app/ {
    rewrite ^/app/(.*)$ /public/$1 last;
    try_files $uri $uri/ =404;
}

The intent is "rewrite /app/foo to /public/foo, serve the file if it exists, 404 otherwise." Since rewrite is on top and try_files is below, the rewrite takes effect first and then try_files handles the fallback — that's how it reads.

But throw /app/logo.png at it and you get a 404, even when /public/logo.png really exists.

$ ls /public/logo.png
/public/logo.png            # the file is definitely there

$ curl -sI http://localhost/app/logo.png
HTTP/1.1 404 Not Found      # and yet, 404

The try_files in this location is never evaluated once.

Symptom 2: config placed inside an `if` block doesn't take effect

In another location you want to add one header conditionally. The naive version looks like this:

location /download/ {
    add_header X-Served-By "nginx" always;

    if ($arg_debug = "1") {
        add_header X-Debug "on" always;   # only want this when ?debug=1
    }

    root /var/www;
    try_files $uri =404;
}

The intent is "always add X-Served-By, and add X-Debug on top of it only when ?debug=1." But hit /download/foo?debug=1 and X-Debug shows up while X-Served-By, which you set on the outside, has vanished. The moment the condition goes true, part of the outer config drops.

Both configs read cleanly when you scan the lines top to bottom. And both miss.

How it works: nginx's HTTP processing phases

nginx runs each request through a fixed sequence of processing phases. Which phase a given directive runs in is determined per directive type, and has nothing to do with where the line sits in the config file. Both symptoms come out of exactly this one fact.

Start with the big picture. Lay out the phases an HTTP request passes through, in order, and you get this (source: the official Phases section of the Development Guide).

post-read       right after the request is read
    ↓
server-rewrite  rewrite directly under server (before location selection)
    ↓
find-config     pick the location
    ↓
rewrite         rewrite / if inside the location
    ↓
post-rewrite    if the URI changed, go back to find-config
    ↓
preaccess       connection count, rate limiting, etc.
    ↓
access          allow / deny / auth
    ↓
post-access
    ↓
precontent      try_files
    ↓
content         serving files via proxy_pass / root
    ↓
log             access log

The same rewrite runs before location selection (server-rewrite) when written directly under server, and after it (rewrite) when written inside a location. Where you put it changes the phase, which changes the order it runs in. What decides the order is not the line sequence but "which block, and which type of directive" — that's how you read this list. Now let's follow how each of the two symptoms travels along this timeline.

Symptom 1, followed by phase

The part that matters for rewrite ... last and try_files is this stretch.

  ┌─▶ find-config     pick the location (first pass: /app/. with last, the second pass picks a different one)
  │       ↓
  │   rewrite         rewrite ... last rewrites to /public/foo
  │       ↓
  └── post-rewrite    URI changed → back to find-config
          ↓
      precontent      try_files runs here. we're now in the different location picked on the second pass.
                      the try_files written in the original /app/ is never reached → 404

The last in rewrite ... last means "once you've rewritten the URI, redo location selection with the new URI." So the instant /app/foo becomes /public/foo, the post-rewrite phase calls find-config back and picks a location all over again. The rewritten /public/foo no longer matches /app/, so it moves to a different location. Meanwhile try_files runs way later, in the precontent phase. It may get evaluated wherever the request landed, but the try_files written in the original /app/ is never reached again. If the destination has no file and no fallback, that's where the 404 comes from. It's not that your =404 fired — the request dead-ended somewhere else, never reaching the original try_files. "Written below, so it runs after" is not what happens; the phases differ, so regardless of line position the rewrite fires first and jumps out of the location.

Symptom 2, followed by phase

if and add_header take effect in two phases that are far apart.

find-config     pick the location (/download/)
    ↓
rewrite         if is true → switch the config context to the inner block
    ↓           (the switch carries over into all later phases)
preaccess       connection count, rate limiting, etc.
    ↓
access          allow / deny / auth
    ↓
post-access
    ↓
precontent      try_files
    ↓
content         apply add_header when the response is being sent
                it looks at the block we switched into (X-Debug only)
                → the outer X-Served-By is gone

An if inside a location is an "implicit nested location" — it creates a config block of its own. And that if is a rewrite-phase directive. When the condition goes true, this phase switches the request's config context into the inner block, and that state carries over into the phases that follow. Much later, when add_header is applied at the point of sending response headers, what it reads is the block we switched into. All that's inside is X-Debug, and on top of that, add_header does not inherit from the outer scope once there's even one in the same block (it replaces rather than accumulates), so X-Served-By disappears entirely. If the condition is false, no switch happens and X-Served-By shows up as usual. The outer header vanishing is not about line order — it's because the switch that happened in the rewrite phase is still in effect in the later phases. What people call if is evil is, at bottom, this "context switch in the rewrite phase."

Fixes

Symptom 1

If you want to keep processing in the same location after the rewrite, change last to break. With break, the post-rewrite return to find-config doesn't happen, so the request can go down to the precontent try_files while staying in the same location.

location /app/ {
    rewrite ^/app/(.*)$ /public/$1 break;   # last → break
    try_files $uri $uri/ =404;
}

If instead you want to route the request to a different location depending on the URI, leave last as is. In that case, put the fallback in the destination location.

location /app/ {
    rewrite ^/app/(.*)$ /public/$1 last;
}

location /public/ {
    root /var/www;
    try_files $uri $uri/ =404;   # put the fallback in the destination
}

Which one is right comes down to a design call: do you want to keep processing in the same location after the rewrite, or hand off to a different one?

Symptom 2

Don't put content-handling directives (add_header, proxy_pass, try_files, and so on) inside an if. Treat return and rewrite as about the only things that are safe inside if.

The condition here is a query argument (?debug=1), and location can't match on the query string. For branching on a value like this, push it out of if and into a map that builds a variable.

map $arg_debug $x_debug {
    default "";
    "1"     "on";
}

server {
    location /download/ {
        add_header X-Served-By "nginx" always;
        add_header X-Debug     $x_debug always;
        root /var/www;
        try_files $uri =404;
    }
}

The two add_header lines sit at the same level, so no nested block replaces anything and both take effect. When $arg_debug isn't 1, $x_debug is empty, and nginx won't emit an add_header with an empty value, so X-Debug shows up only during debugging.

Checking which location you're in right now

The feel for phases sticks once you get your hands dirty. The quick way is to plant one marker in a suspect location to tell whether the request settled there.

location /app/ {
    add_header X-Loc "app" always;   # marker that shows up if you stay in this location
    rewrite ^/app/(.*)$ /public/$1 last;
    try_files $uri =404;
}

Look at the headers with curl -sI http://localhost/app/foo, and if X-Loc: app isn't there, the request has already left this location via rewrite ... last (the add_header you wrote here has no effect once it's gone). Change last to break, hit it again, and if X-Loc: app appears, the request stayed in the same location and made it down to precontent. One marker header lets you see, across phases, which location the request is in right now.

Wrapping up

When you read an nginx config, your eyes follow the lines top to bottom. But what nginx actually does is run the request along a timeline of phases. When you're questioning a config, don't trace the line order — trace it in this order instead: "which phase does this directive run in? and at that point, which location is the request in?" That usually makes things line up.

More places where this phase-based behavior produces silent bugs — add_header inheritance dropping when if switches context, alias path traversal, location selection overrides — are catalogued with reproduction cases in Silent nginx Config Bugs That Pass nginx -t.

nginx Tools for When Your Config Gets Out of Hand

Kazu — Tue, 04 Aug 2026 13:02:00 +0000

nginx config files grow. What starts simple gets layers added on top: another reverse proxy, TLS settings, routing logic that branches and branches again.

Before long you're looking at 20+ location blocks and can't say off the top of your head which one handles which request. You add a single add_header line and some other header silently disappears. You fix an alias path and prod throws 404s while staging is fine. nginx -t says "syntax is ok." But you don't feel ok about it.

Every config change comes with a quiet dread: will this break something? And that dread usually has a basis. nginx config mistakes are silent. Nothing shows up in the error log. The browser gets a normal-looking response. You don't find out something is broken until much later.

nginx itself is well-documented, but the ecosystem around it is harder to survey. Validation tools, config generators, test frameworks — searching turns up scattered information. This article organizes the common pain points into six categories and maps the tools that address each one.

Problem 1: Prevent config mistakes upfront
Problem 2: Generate config files
Problem 3: Automate testing
Problem 4: Collect metrics
Problem 5: Run nginx on Kubernetes
Problem 6: Visualize logs
Gaps that still aren't filled

Problem 1: Prevent config mistakes upfront

nginx -t checks syntax. But syntactically valid config can still have security problems. SSRF, path traversal via alias (e.g., a request to /files/../../etc/passwd reaches files it shouldn't), HTTP splitting — none of these trigger a syntax error.

Gixy: static analysis focused on security

https://github.com/dvershinin/gixy

Originally built at Yandex, now actively maintained as a community fork. It checks for well-known security risks: SSRF, alias traversal, HTTP splitting. Install the gixy-ng fork from PyPI.

pip install gixy-ng
gixy /etc/nginx/nginx.conf

Output looks like this — it tells you which directive is the problem and why:

[WARN] [http_splitting] Possible HTTP-Splitting via HTTP header.
Directive: proxy_set_header X-Forwarded-For $http_x_forwarded_for;

A good fit when you've just inherited an nginx config and want a quick security audit before touching it. Detection is limited to known patterns and there's no custom rule support, but as a first security gate in CI it offers the best return on effort.

crossplane: treat config as data

https://github.com/nginxinc/crossplane

A Python library that parses nginx config files into JSON. Useful not just for validation but for dynamically generating config or loading it in tests.

import crossplane

payload = crossplane.parse('/etc/nginx/nginx.conf')

# Check for parse errors
for error in payload['errors']:
    print(f"Error: {error}")

# Walk the config tree to find specific directives
def find_directives(block, name):
    for item in block:
        if item.get('directive') == name:
            yield item
        if 'block' in item:
            yield from find_directives(item['block'], name)

config = payload['config'][0]['parsed']

# Example: flag any proxy_pass that uses plain HTTP
for d in find_directives(config, 'proxy_pass'):
    url = d['args'][0]
    if url.startswith('http://'):
        print(f"Warning: proxy_pass uses plain HTTP: {url}")

The parsed output is a tree of directives and their arguments. With a traversal like find_directives, you can express project-specific rules in Python: "are all proxy_pass values HTTPS?", "is server_tokens off set?" Where Gixy is limited to known patterns, crossplane lets you write your own.

Scripting against config files and automating structural checks in CI are where this earns its place. If you don't have a concrete need to generate or programmatically inspect config, skip it for now. Adding it "because it looks useful" tends to mean it sits unused.

Which to choose

Goal	Pick
Find security issues quickly	Gixy
Manipulate or validate config in code	crossplane

Gixy alone covers the security CI gate. If you're writing config generation or structural validation logic in Python, you need crossplane. They don't overlap, so using both is fine.

Problem 2: Generate config files

Writing TLS config from scratch means making a lot of small decisions: cipher suites, HTTP/2 support, HSTS headers. Starting from a known-good template beats hand-rolling it and getting something subtly wrong.

nginxconfig.io: generate best-practice config in the browser

https://www.digitalocean.com/community/tools/nginx

A web app by DigitalOcean that generates a complete nginx config from a series of UI choices. Select your TLS cipher suites, gzip settings, HTTP/2, security headers, and it outputs ready-to-use config files. Free, no account required.

Use it when bootstrapping a new server or bringing TLS settings up to current best practices. It's an excellent starting point, but don't deploy the output unchanged. Treat it as a template and adjust for your project's requirements.

nginx-proxy: auto-generate config in Docker environments

https://github.com/nginx-proxy/nginx-proxy

A container that watches Docker start/stop events and automatically updates nginx config. Set a VIRTUAL_HOST environment variable on your container and nginx-proxy generates the reverse proxy config for it.

services:
  app:
    image: my-app
    environment:
      - VIRTUAL_HOST=example.com
  nginx-proxy:
    image: nginxproxy/nginx-proxy
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro

When app starts, nginx-proxy automatically creates config to route example.com traffic to it.

In Docker Compose setups where manually updating nginx on every deploy is friction, this removes that burden. Traefik and Caddy do the same thing. If you're not already committed to nginx (existing config, team familiarity), compare them first. If you're already running nginx, nginx-proxy is the natural fit.

Which to choose

Environment	Pick
Regular Linux server	nginxconfig.io
Docker / Docker Compose	nginx-proxy

No Docker? nginxconfig.io is enough. In Docker environments, nginx-proxy's automation pays off.

Problem 3: Automate testing

Manually sending test requests after every config change doesn't scale. Automating it means the same checks run every time.

Test::Nginx: declarative test framework

https://github.com/openresty/test-nginx

A Perl-based test framework from the OpenResty community. It starts a real nginx instance, sends requests, and checks responses. The format is declarative — you describe what config to use, what request to send, and what response to expect. Tests read clearly as a result.

=== TEST 1: basic proxy
--- config
location /api {
    proxy_pass http://127.0.0.1:8080;
}
--- request
GET /api/users
--- response_body
[{"id":1}]

This fits nginx module development and verifying that config changes don't alter behavior. Perl is a real barrier for most teams today. If you're not writing custom nginx modules, hurl covers most of what you actually need.

hurl: a DSL for HTTP testing

https://hurl.dev/

hurl lets you write HTTP requests and assertions in a plain-text format, then run them from the shell. Easy to slot into CI. Not nginx-specific — it works for HTTP testing in general.

GET http://localhost/api/users
HTTP 200
[Asserts]
header "Content-Type" contains "application/json"
jsonpath "$.length" > 0

hurl --test api.hurl

Reach for it when you want to confirm API behavior hasn't broken after a config change, or when the team has no Perl experience. Because it's not nginx-specific, it doesn't go deep on nginx internals. It works well for endpoint reachability checks, but for testing precise nginx behavior (match precedence, path rewriting), Test::Nginx is the right tool.

Which to choose

Situation	Pick
Testing nginx config or modules directly	Test::Nginx
Checking that the API behind nginx still works	hurl
Team has no Perl experience	hurl

Test::Nginx for precise nginx behavior verification. hurl for HTTP interface testing with a lower adoption cost.

Problem 4: Collect metrics

Without visibility into how many requests nginx is handling and where the bottlenecks are, there's no basis for making informed decisions when problems surface.

nginx-module-vts: embedded stats module

https://github.com/vozlt/nginx-module-vts

A module that adds traffic statistics to nginx — request counts, error rates, latency per virtual host and upstream. Built in at compile time. Exposes a /status endpoint that returns JSON.

vhost_traffic_status_zone;

server {
    location /status {
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
    }
}

Detailed traffic stats are the payoff — when you have a build environment and can compile nginx from source. Requiring a source build is the biggest barrier. If you're using a packaged nginx install, this isn't an option. If you already have Prometheus, the overhead of a custom build may not be worth it.

nginx-prometheus-exporter: bridge stub_status to Prometheus

https://github.com/nginxinc/nginx-prometheus-exporter

Reads nginx's built-in stub_status data (connection counts, request totals) and exposes it in Prometheus format. Runs as a separate process alongside nginx — no changes to nginx itself required.

location /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
}

./nginx-prometheus-exporter -nginx.scrape-uri=http://localhost/nginx_status

In Prometheus + Grafana stacks, this is the easiest way to add nginx metrics without touching nginx itself. stub_status only gives you connection counts and request totals — no per-upstream breakdown. Confirm that's enough before committing.

Which to choose

Situation	Pick
Already running Prometheus	nginx-prometheus-exporter
Need per-host, per-upstream, per-status-code stats	nginx-module-vts
Using packaged nginx and don't want to rebuild	nginx-prometheus-exporter

If you're on Prometheus, nginx-prometheus-exporter is by far the easier path. If you need granular metrics and control your build, look at vts.

Problem 5: Run nginx on Kubernetes

When exposing services externally in Kubernetes, you use an Ingress resource — a rule that maps incoming paths to backend services. nginx-based Ingress controllers are widely used for this. Two controllers with similar names exist, and they're frequently confused.

ingress-nginx: the Kubernetes community controller

https://github.com/kubernetes/ingress-nginx

Maintained by the Kubernetes community (kubernetes/ingress-nginx). Used as the default by many Kubernetes distributions. Large amount of documentation and community knowledge available.

In March 2025, a set of serious vulnerabilities called IngressNightmare was disclosed (CVE-2025-1974 and others, CVSS 9.8). Remote code execution was possible through the admission controller component. Fixed in versions 1.12.1 and 1.11.5. If you're running an older version, upgrade now.

On a general-purpose Kubernetes cluster running standard Ingress, this is the obvious starting point. Switching Ingress controllers later is painful, so the initial choice matters. When in doubt, ingress-nginx has fewer dead ends from a documentation standpoint.

kubernetes-ingress: the NGINX Inc. controller

https://github.com/nginxinc/kubernetes-ingress

Maintained directly by NGINX Inc. (nginxinc/kubernetes-ingress). Supports both open-source nginx and NGINX Plus (the commercial version).

The case for it is NGINX Plus features (advanced load balancing, active health checks) or a support contract with NGINX Inc. If you don't need NGINX Plus, there's little reason to choose this over ingress-nginx. For OSS nginx, the community controller has better coverage.

Which to choose

Situation	Pick
Community resources matter	ingress-nginx
Planning to use NGINX Plus	kubernetes-ingress
Starting on EKS / GKE or similar managed cluster	ingress-nginx

If you go with ingress-nginx, run version 1.12.1 or 1.11.5 or later. Check your current version before anything else.

Problem 6: Visualize logs

nginx access logs pile up as text. Figuring out which paths are getting hammered, or where errors are spiking, is hard to do from raw log files.

GoAccess: real-time stats in the terminal

https://goaccess.io/

A TUI tool that aggregates nginx access logs and displays them in real time. Graphs and stats update live in the terminal. No log server or database needed — just install and run. Can also generate HTML reports for browser viewing.

goaccess /var/log/nginx/access.log -c
# Starts in interactive mode to select the log format

# Stream live logs in COMBINED format
tail -f /var/log/nginx/access.log | goaccess --log-format=COMBINED -

Reach for it when you need answers quickly, especially on a server with no log collection infrastructure in place. It reads local log files directly, so multi-server environments are awkward. Think of it as a per-server quick-look tool.

Loki + Grafana Alloy: long-term log storage and search

https://grafana.com/oss/loki/ / https://grafana.com/docs/alloy/latest/

Loki stores and queries logs efficiently, with Grafana for dashboards. Grafana Alloy handles log collection. Alloy replaces Promtail, which reached end-of-life in March 2026 — for new setups, use Alloy.

Example Alloy config (sending nginx logs to Loki):

loki.source.file "nginx" {
  targets = [{
    __path__ = "/var/log/nginx/access.log",
    job      = "nginx",
  }]
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

It earns its place when you need to aggregate logs from multiple servers, search historical logs by time range, or view nginx logs alongside other metrics in Grafana. Running the full stack yourself carries real operational overhead. Start with Grafana Cloud's free tier, or bring this in when someone on the team already knows how to operate Grafana.

Which to choose

Situation	Pick
Need to check something right now	GoAccess
Already running Grafana	Loki + Grafana Alloy
Need long-term log retention and cross-server search	Loki + Grafana Alloy
Debugging with direct SSH access to the server	GoAccess

GoAccess gives you immediate answers. Loki + Alloy is for ongoing operations. They're not mutually exclusive — GoAccess for fast situational awareness, Loki for accumulating and analyzing over time.

Summary: match tool to problem

Problem	Tool
Find security risks in config	Gixy
Manipulate config programmatically	crossplane
Generate correct config (Linux server)	nginxconfig.io
Auto-manage Docker reverse proxies	nginx-proxy
Test nginx config behavior	Test::Nginx
Add HTTP connectivity checks to CI	hurl
Collect metrics via Prometheus	nginx-prometheus-exporter
Need detailed per-upstream stats (module)	nginx-module-vts
Set up Ingress on Kubernetes	ingress-nginx (1.12.1+)
Need NGINX Plus features	kubernetes-ingress
Visualize logs immediately	GoAccess
Long-term log storage and search	Loki + Grafana Alloy

nginx itself is stable and mature, but the ecosystem moves fast — Promtail's EOL is a good example. Periodically check the maintenance status of tools you rely on, or you'll find out about a support cutoff at the worst time.

Gaps that still aren't filled

The tools covered here leave some problems unsolved or underserved. If you know tools that address any of these, drop them in the comments.

Dynamic upstream management

Updating upstreams without an nginx reload is a common requirement. With open-source nginx, the only options are OpenResty (Lua scripting for dynamic upstream control) or NGINX Plus (API-driven management). Vanilla nginx requires rewriting config and reloading.

Config drift tracking across environments

There's no standard tool for tracking and visualizing how nginx config differs between dev, staging, and production. Git-based management is the practical answer, but cross-environment diff review and detecting prod-only config still means manual work.

IDE / LSP support

VSCode extensions for nginx.conf exist, but completion and inline validation are incomplete. No editor today reliably explains location block match priority or how proxy_pass transforms paths.

WAF

ModSecurity has a solid track record as an nginx WAF, but rule management and false-positive tuning are heavy ongoing work. There's nothing you can drop in and operate at a low maintenance level.

https://github.com/owasp-modsecurity/ModSecurity-nginx

Distributed tracing

The nginx OpenTelemetry module (ngx_otel_module) shipped in 2023 but is still maturing. Trace collection with Jaeger or Tempo is possible, but documentation and real-world operational experience are thin.

https://github.com/nginxinc/nginx-otel

DEV Community: Kazu

High p99 latency in Node.js at low CPU usage: a blocked event loop

How Node.js handles requests

Why overall CPU looks flat

Spillover to unrelated endpoints

Observing lag

Measuring event loop lag

Per-core CPU

Effect and limits of an overload guard

Finding and fixing the blocking work

Finding it

Fixing it

Summary

Buttons don't work for some users after a deploy

Impact

Reproduction

Without clicking the button first

With the button clicked first

Narrowing it down

Access log

Old and new mixed during the deploy

Server-side cache

Tabs left open

Browser HTTP cache

Relationship to time since modification

Time since modification

Reload

What is happening

What decides it

Fixes

Set a cache lifetime for the HTML

Add no-cache to index.html

Add max-age to index.html

Reload when loading fails

Effect of the fixes

no-cache

max-age

Reload on failure

Scope

Browsers

When the previous version's JS is kept

When missing paths don't return 404

Other ways of serving

When file timestamps are pinned

With a Service Worker

Built with webpack

Appendix: how the times were produced

High p99 latency at low CPU usage is caused by CPU throttling

Impact

Reproduction

Narrowing it down

The accept queue

Downstream calls

CPU usage

cpu.stat

Conditions for hitting the limit

Usage

Number of workers

CPU time per period

What is happening

Waiting before accept

The period's 50ms

The whole container stops

Tracing each request

Mapping to the symptoms

What decides it

Remedies

Raise the CPU limit

Add nr_throttled and nr_periods to monitoring

Effect of the remedy

Notes

No shell in a distroless Pod: what kubectl debug can and can't see

Test setup

Background: container isolation is set per namespace

Adding an ephemeral container

--target changes what ps shows

What you can see directly

Network: use it directly

Process state: read from /proc/1

Environment and arguments: read from /proc/1

`cpu.stat`

Add `nr_throttled` and `nr_periods` to monitoring

`--target` changes what `ps` shows

You can't use `--target` on a stopped container