DEV Community: Sai Pisey

One header in the config. A thousand on the wire.

Sai Pisey — Sat, 24 Oct 2026 10:22:08 +0000

An Alloy user had one header configured on pyroscope.scrape, an X-Api-Key. About two hours in, profiling stopped. Their packet capture showed a single GET carrying roughly 1,000 copies of that header, and the target answering 431 Request Header Fields Too Large.

The bug was in prometheus/common, the library behind http_config in Prometheus, Alertmanager and the exporters. The reporter had already traced it to cloneRequest. I wrote the upstream fix, #982, which shipped in v0.72.1.

Reproducing it

One header in the config, the same request sent four times:

$ go run .   # prometheus/common v0.71.0
server saw X-Api-Key x1
server saw X-Api-Key x2
server saw X-Api-Key x3
server saw X-Api-Key x4

$ go run .   # prometheus/common v0.72.1
server saw X-Api-Key x1
server saw X-Api-Key x1
server saw X-Api-Key x1
server saw X-Api-Key x1

The copy that copied nothing

r2 := new(http.Request)
*r2 = *r
// Deep copy of the Header.
maps.Copy(r.Header, r2.Header)

The comment says deep copy. It isn't one. *r2 = *r leaves both requests pointing at the same header map, and maps.Copy copies that map into itself. Anything the library adds afterwards lands in the caller's request. http_headers uses Header.Add, so a request that gets reused picks up one more copy per send.

Checking your binaries

$ go version -m ./alloy | grep prometheus/common
    dep github.com/prometheus/common    v0.71.0 h1:...

v0.69.0 to v0.72.0 are affected. v0.72.1 or later is fixed. You only hit it with an affected version, an http_headers block, and code that sends the same request object twice.

Prometheus 3.15 pins the same broken version and never shows the bug, because an otelhttp wrapper clones each request first. The full post has the exposure table for five tools and the regression test that went in with the fix.

Read the full post on saipisey.com →

One command tells you if your next Kubernetes upgrade will brick a node

Sai Pisey — Sat, 17 Oct 2026 07:52:54 +0000

Since v1.35, the kubelet setting failCgroupV1 defaults to true. On a node that still runs cgroup v1, the kubelet refuses to start, so the node goes NotReady and stays there. On a control-plane node it is worse: the kubelet starts the API server and etcd as static pods, so after a reboot they do not come back either.

kubeadm warns you first: its SystemVerification preflight check errors on init, join and upgrade once the kubelet is v1.35 or later. Do not skip it with --ignore-preflight-errors.

Under the hood, cgroup v1 and v2 lay out the same pod very differently, which is why one stat call can tell them apart:

Check one node

$ stat -fc %T /sys/fs/cgroup/
cgroup2fs

cgroup2fs: cgroup v2. Nothing to do.
tmpfs: cgroup v1. After an upgrade to v1.35 or later, its kubelet will not start.

Check every node without SSH

$ for n in $(kubectl get nodes -o name); do
    printf "%s " "$n"
    kubectl debug "$n" -n default -q -i --profile=general --image=busybox \
      -- chroot /host stat -fc %T /sys/fs/cgroup
  done
node/worker-1 cgroup2fs
node/worker-2 tmpfs

Keep the chroot /host. Without it the command runs busybox's own stat, which prints UNKNOWN instead of cgroup2fs on a healthy v2 node. I hit that while testing on a v1.35 cluster.

Which nodes are likely on cgroup v1

CentOS 7, RHEL 7 and 8, Ubuntu 18.04 and 20.04, and Amazon Linux 2 all default to cgroup v1. Self-managed kubeadm clusters on long-lived VMs are where this bites.

The full post covers what a node needs for cgroup v2, which nodes can switch in place and which need a new OS image, how to check the kubelet and containerd cgroup drivers, and the failCgroupV1: false escape hatch with its expiry date.

Read the full post on saipisey.com →

Your cluster says 3/3. It will still go dark with one zone.

Sai Pisey — Sat, 10 Oct 2026 10:36:48 +0000

Every deployment in this three-zone cluster reports ready:

$ kubectl get deployments
NAME            READY   UP-TO-DATE   AVAILABLE
checkout-api    3/3     3            3
session-store   1/1     1            1
web             3/3     3            3

Now take us-east-1a away:

$ kubectl survive-zone
Losing us-east-1a  ->  2 lost, 0 degraded, 1 impaired by a dependency
  WORKLOAD       PLACEMENT                               VERDICT
  checkout-api   us-east-1a:3                            LOST
  session-store  us-east-1a:1                            LOST
  web            us-east-1a:1 us-east-1b:1 us-east-1c:1  IMPAIRED

Two workloads are simply gone. The third still has two healthy replicas, and stops serving anyway.

Nobody made a mistake

checkout-api was scheduled while us-east-1a was the only zone. The scheduler put all three replicas there, correctly. Two more zones were added later, and nothing moved them.

That is not a bug. Topology spread constraints are applied when a pod is scheduled, and never checked again. The Kubernetes docs say it plainly: there is no guarantee they stay satisfied once pods move.

So the manifests say multi-AZ, and the cluster quietly says something else.

The row a linter would pass

web is the interesting one. It is spread one replica per zone, which is exactly what a spread linter wants to see. It still goes down, because the only session-store pod lives in the zone that died.

Zone survivability is a property of the dependency graph, not of each workload on its own. kubectl survive-zone follows each Service to its EndpointSlices to build that graph, so a workload can be marked impaired even when its own pods look perfect.

It will not hand you a fix that causes an outage

Most zone fixes can make pods unschedulable. Tighten a spread constraint from ScheduleAnyway to DoNotSchedule and you can end up with replicas stuck in Pending, which is the outage you were trying to avoid.

So kubectl survive-zone fix only prints a patch after proving two things: that it is schedulable on your real nodes, using the upstream kube-scheduler's own Filter plugins, and that it actually survives the zone loss.

And it keeps watching

Placement decays with nothing changed in git. A node drains at night, pods reschedule, and a workload that was spread now sits in one zone. Run as a Prometheus exporter, the tool turns that into an alert while the zone is still up.

Two things the full post covers that surprised me

A node drain that hangs forever, behind a disruption budget that looks completely fine. Budget arithmetic says the drain is safe. It is not, and the reason only shows up when you simulate the replacement pod against the real scheduler.

An ordinary rolling update that quietly broke an enforced maxSkew: 1. I found it by accident while recording the demo, and the fix is a one-line field most spread constraints do not set.

The full write-up also covers how every verdict is checked nightly against a real control plane in 200 randomised scenarios, and what the harness caught that no unit test did.

Read the full post on saipisey.com →

Try it in under a minute. It is read-only and never writes to your cluster:

$ kubectl krew install survive-zone
$ kubectl survive-zone