Patch the image, not the version

October 21 was a busy day for anyone who runs servers. PHP fixed two critical remote code execution bugs. GitLab fixed one rated 10.0. Node.js shipped security releases for its 22, 24 and 26 lines, including an out-of-bounds write in the Brotli decoder that the advisory says can lead to remote code execution.

Different stacks, same job: patch it, then prove the patch is running.

The Node one looked like a one-line change:

- FROM node:22.12.0-alpine
+ FROM node:22.23.4-alpine

Bump the number, rebuild, deploy. I did this on a set of containerized services that week, and the version bump was the easiest part. The real work was finding every place the version lived, noticing what else the new image changed, and checking which pods actually picked it up.

Diagram of the whole trip (① pick the patch, ② change every pin, ③ the image moves its OS packages too, ④ only rebuilt services get the fix, ⑤ deployed isn't running, so check the pods): see it on maruf.ing

1. Patch now, upgrade later

The advisory lists a fixed version per line: 22.23.4 for 22.x, 24.22.1 for 24.x. Both carry the same security fixes, so the only question is how much else you want to change today.

	stay on 22	move to 24
Change	patch only	major
Risk	low	APIs, native modules, deps
Support ends	Apr 30, 2027	Apr 30, 2028

I took the 22 patch. A security fix should ship fast, and if it rides along with a major upgrade, a failure could come from either change. The move to 24 is real work (native modules rebuild, deprecated APIs, dependency ranges) and gets its own change and its own testing.

That doesn't make 24 optional. Node 22 has about six months of support left, so the patch buys time to do the upgrade properly, not a reason to skip it. And if your line is already end of life, the order flips: there is no patch to take.

2. Find every pin

"Which Node do we run?" usually has several answers in one repository:

• the FROM lines in each Dockerfile, often two per file in a multi-stage build
• the image the CI jobs build and test in
• engines in package.json, .nvmrc, .tool-versions
• dev containers and local compose files

These drift. A new service gets copied from an old template and keeps its old tag. CI tests on one version while production runs another. A fix applied to "the version" lands in some of these places and quietly misses the rest.

So the first step is a search, not an edit:

# find-pins.sh
# Every place a Node version might be pinned.
# Review each hit; not every match is a runtime pin.
grep -rnE 'node:[0-9]|"node":|nodejs' \
  --include='Dockerfile*' --include='*.y*ml' \
  --include='package.json' --include='.nvmrc' \
  --include='.tool-versions' .

One hit to leave alone: @types/node has its own version numbers and isn't the runtime.

The longer fix is fewer pins. One build argument every Dockerfile reads, and CI and local tooling reading the same file. Then the next advisory is one edit.

3. The image is part of the change

This was the part that surprised me.

A tag like node:22.23.4-alpine doesn't say which Alpine it's built on. It means whatever Alpine the maintainers build on today. The image I was replacing was built on an older one, so the new tag moved Alpine forward two releases.

Each Alpine release ships its own versions of every system package. Anything your service installs with apk add moves with it. For me that was ffmpeg, which one service uses to process uploaded media: 6.1 on Alpine 3.21, 8.0 on 3.23. A Node security patch had become a major upgrade of a media tool, and nothing in the diff said ffmpeg.

Yours might be ImageMagick, libvips, a Postgres client, git, fonts or the CA certificates. Same thing: the tag you bumped says Node, and something else moved.

A container image is a stack: base OS, system packages, runtime, your dependencies, your code. Changing the tag can move any layer under it. Call it base-image drift.

# Dockerfile
# Floating base: Alpine moves when this tag is rebuilt.
FROM node:22.23.4-alpine

# Explicit base: only Node moves.
FROM node:22.23.4-alpine3.23

Naming the Alpine release makes the change visible in review. It doesn't always let you keep the old base, since new Node patches are only published for the Alpine releases still being built on. When the old one isn't offered, the base move is part of the patch, and you test it that way. Debian tags like -bookworm already name the release, so the OS stays put across Node patches, though its packages still pick up updates on rebuild.

Either way, before shipping, compare the system packages your service actually uses in the old and new image. apk info -v (Alpine) or dpkg -l (Debian) in each is a quick diff.

careful: A major bump in a system tool can pass a smoke test and still behave differently: new defaults, removed flags, slower output. Running the service's exact command in a test container only proves the command runs. Push real input through it on staging (an upload, an import, a report) and compare the result.

4. Green isn't proof

Two things can make a successful pipeline lie to you.

Path filters. Many pipelines rebuild only the services whose files changed. A service you didn't touch keeps its old image and its old Node. If a service gets its runtime from a shared base image instead of its own FROM line, the filter won't see that it needs a rebuild. Touch every affected service or force a full rebuild.

"Deployed." With Helm, STATUS: deployed means the manifests were applied and the release was recorded. Without --wait, it says nothing about whether new pods started, passed readiness, or replaced the old ones. A pod stuck on a bad image sits behind a release that says deployed.

So check what is actually running:

# check-rollout.sh
# Did the rollout finish, and which image runs?
kubectl rollout status deploy/api --timeout=5m
kubectl get pods -l app=api \
  -o jsonpath='{.items[*].spec.containers[*].image}'
# The runtime itself, from inside a live pod.
kubectl exec deploy/api -- node --version

The first waits for the rollout. The second shows the image the pods started from. The third answers the original question from inside a running container. Do it for every affected service, not one sample.

In my case the deploy log showed every release upgraded, but my cluster login had expired, so for a while that log was all I had. After logging back in I checked the pods, and they were running the patched version. Only then was it done. A deploy log is evidence that a deploy happened, not that the fix is running.

5. Make the pipeline check it

Checking every service by hand is tedious, which is a good sign it belongs in the pipeline:

• rebuild everything that depends on a shared base image when that image changes
• deploy with --wait and a timeout, so the pipeline fails when pods don't come up
• after rollout, read the runtime version from a pod and compare it to the expected one

For one service with no system packages, most of this collapses into bump, deploy, check. The list grows with the number of places the version lives and the number of things the image carries besides Node.

The short version

The advisory gives you a version number. It doesn't tell you where that number lives, what else moves with the image, or which pods picked it up.

The version bump is one line. Proving it reached production is the work.

---

First published on maruf.ing. I write about boring backend systems.