Oleksandr Kuryzhev

Posted on Nov 9 Originally published at kuryzhev.cloud

Docker Engine 29 upgrade gotchas for production hosts

#devops #docker

Originally published on kuryzhev.cloud

A typical failure looks like this: unattended upgrades pull a new docker-ce package overnight, the daemon restarts, and by morning the reverse proxy no longer discovers containers. The proxy logs complain that its client API version is too old, while docker ps works fine from the shell. A Docker Engine 29 upgrade can produce exactly this, and a few other surprises, on hosts nobody planned to touch.

Docker Engine 29 was released on 2025-11-10 (see the official release notes), and the line is now at 29.9.3. A year on, Docker's apt repository offers 29.x for supported Ubuntu releases, including 26.04 LTS. Older 28.x hosts are being swept up by routine patching, and anything still on a floating package channel will get it. The time to find the breaking edges is before the package manager finds them for you.

Short answer: pin the package, inventory every API client that talks to /var/run/docker.sock, and upgrade one canary host first. If a client still speaks an API older than the new minimum, upgrade the client, and use DOCKER_MIN_API_VERSION only as a temporary bridge. Pin your image store in daemon.json so it does not change underneath you.

What changed and who is affected

The items below come from the Docker release notes and engine docs. Verify each against the notes for your exact target version, because details shift between 29.x patch releases.

Minimum API version raised. The daemon now rejects clients that speak very old API versions (the documented floor is 1.44). Affected: older reverse proxies, monitoring agents, CI runners, and homegrown scripts built on pinned Docker SDKs.
containerd image store as the default for new installs. Fresh installs use it. Existing hosts are documented to keep their current graph driver. Affected: anyone who rebuilds hosts from images or automation and expects the old overlay2 layout.
nftables firewall backend (experimental). An opt-in alternative to iptables. Affected: only hosts that enable it, but worth knowing about if your security team is pushing nftables.
Live restore does not cover major upgrades. The documented behavior of live-restore is limited to patch-level daemon upgrades. Plan for containers to restart across a major jump.

Not affected in practice: hosts that only run docker compose and the CLI from the same package set, since the client and daemon move together. Compose files should already omit the legacy top-level version: key.

Before/after config for a Docker Engine 29 upgrade

A typical pre-upgrade /etc/docker/daemon.json on a production host relies on defaults for the storage layer:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" },
  "live-restore": true
}

🧪 Lab — try this in your browser

Docker Compose Auditor — Parses docker-compose YAML to audit services for security, reliability and portability issues, with a scored report and service inventory.
Docker Workbench — Build, convert, audit and analyse Docker Compose files and Dockerfiles, with a reference library.

The target config makes the storage layer explicit, so a rebuilt host behaves like the one it replaces. It does two things:

It disables the containerd image store through the containerd-snapshotter feature flag.
It pins the classic overlay2 graph driver.

Switch to the containerd image store later, deliberately, not as a side effect:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" },
  "live-restore": true,
  "storage-driver": "overlay2",
  "features": { "containerd-snapshotter": false }
}

Some old clients cannot be upgraded before the maintenance window. For those, add a systemd drop-in as a stopgap. Create /etc/systemd/system/docker.service.d/min-api.conf:

[Service]
# Temporary bridge: lets pre-1.44 API clients connect. Remove once clients are upgraded.
Environment="DOCKER_MIN_API_VERSION=1.24"

Watch out for: this override keeps you on a deprecated client path. It buys time and nothing else, so put a removal date on the ticket.

Step-by-step migration

Run this on a canary host first, with the same workloads and the same socket consumers as production.

Inventory socket clients. Search for anything that mounts the socket or uses DOCKER_HOST: proxies, log shippers, CI agents, deploy tools. Check each tool's changelog for its Docker API support.
Record the baseline. Save the current versions, storage driver and container list so you can diff afterwards.
Drain and stage config. Drain the host from the load balancer before installing anything. The docker-ce package restarts the daemon during installation, and live restore will not carry you across a major version. Write the target daemon.json and the optional drop-in before the install, so that restart picks them up.
Pin and upgrade. Choose an explicit version string rather than "latest", and hold the package afterwards.

# 1. Baseline (store the output with the change ticket)
docker version --format '{{ "{{" }}.Server.Version}} api={{ "{{" }}.Server.APIVersion}} min={{ "{{" }}.Server.MinAPIVersion}}'
docker info --format 'driver={{ "{{" }}.Driver}}'
docker ps --format '{{ "{{" }}.Names}}\t{{ "{{" }}.Image}}' | sort | sudo tee /root/containers.before

# 2. Drain the host, then stage /etc/docker/daemon.json and the drop-in (if used)
sudo systemctl daemon-reload

# 3. Find the exact package string available for 29.x
apt-cache madison docker-ce | head

# 4. Install the pinned version (replace with the string from step 3); this restarts dockerd
sudo apt-get install -y docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING>

# 5. Stop unattended upgrades from moving it again
sudo apt-mark hold docker-ce docker-ce-cli containerd.io

Installation instructions differ by distribution. Follow the Docker Engine install docs for yours. More runbooks for this kind of host work live on kuryzhev.cloud.

Verify and roll back

Verification is a diff, not a feeling. Compare the post-upgrade state with the baseline and check that every socket client reconnected.

# Server version and the API floor it enforces
docker version --format '{{ "{{" }}.Server.Version}} api={{ "{{" }}.Server.APIVersion}} min={{ "{{" }}.Server.MinAPIVersion}}'

# Storage driver must match the baseline (expect overlay2 if you pinned it)
docker info --format 'driver={{ "{{" }}.Driver}}'

# Same containers, same images, all running
docker ps --format '{{ "{{" }}.Names}}\t{{ "{{" }}.Image}}' | sort | sudo diff /root/containers.before - && echo "containers match"

# Rejected API versions may appear here, depending on daemon log level
journalctl -u docker --since "-15min" | grep -i "api version"

Expected shape:

The server reports a 29.x version.
The driver matches the baseline.
The diff prints "containers match".

The daemon returns API version rejections to the client, and it may not log them at the default level. An empty grep is therefore not proof. Check each socket client's own logs for "too old" errors. Also watch your proxy's discovered-backend count and your monitoring agent's container metrics for a drop.

To roll back, run sudo apt-mark unhold docker-ce docker-ce-cli containerd.io. Then install the previous pinned 28.x string with sudo apt-get install -y --allow-downgrades docker-ce=<PREVIOUS_VERSION> docker-ce-cli=<PREVIOUS_VERSION> and hold the packages again. Downgrades across majors are not guaranteed to be clean for every on-disk state, so take a VM or volume snapshot before the upgrade.

The cost is real: snapshotting a host with a large /var/lib/docker takes time and storage. Where it is too expensive, the canary-first approach is your safety net.

Gotchas after the switch

Images appear to vanish after switching image stores. The classic graph driver and the containerd image store keep images separately. Enabling "features": {"containerd-snapshotter": true} on an existing host shows an empty docker images list until you pull or load again. The old images are still on disk, and switching back brings them back. Budget for double disk usage during the transition, and pre-pull critical images before cutover.

Watch out for rebuilt hosts. A freshly provisioned node can land on the new default store while the rest of the fleet runs overlay2. Pin both storage-driver and containerd-snapshotter in your configuration management to remove the mixed fleet. Confirm with the docker info check above in your provisioning tests.

Do not treat live restore as an upgrade tool. Containers may restart during a major upgrade even with live-restore enabled. Anything without a restart policy or a health check will stay down. Check with docker ps -a for exited containers after the restart.

Firewall surprises. The nftables backend is experimental. Do not enable it on production without testing published ports, inter-network traffic and any existing iptables-based rules. If you have not decided to adopt it, leave it off and read the Docker Engine documentation section on packet filtering first.

Embedded SDK users. Internal tools that import the Docker Go client may need dependency updates, since module layout and supported API ranges have moved. Check the release notes for the exact module path changes before bumping versions in your build.

When not to upgrade yet: a critical socket client has no release supporting the newer API floor, and you cannot afford the temporary override. In that case, stay on your current 28.x patch, keep it pinned, and revisit once the vendor ships a fix.

DEV Community