One upgrade, three outages: glibc, cgroups, and a placeholder bug

This is the last of three parts. In part one I split my personal services onto their own two-node cluster. In part two I read the orchestrator's source and fixed two bugs, which shipped in the next release. This part is what happened when I installed that release.

It broke the cluster three separate ways. None of them shared a root cause. Here they are in the order I hit them.

Failure one: glibc floor

I copied the official release binary to the second node and the service went into a crash loop:

/usr/local/bin/orca: /lib/x86_64-linux-gnu/libc.so.6:
  version `GLIBC_2.32' not found (required by orca)

The node runs Ubuntu 20.04, which ships glibc 2.31. The official builds are linked against something newer, so the binary simply cannot load there. The controller node is newer and took the release fine. The old node could not.

The fix was to stop using the official binary on that node and build one linked against its glibc. Compiling inside a matching container does it:

docker run --rm -v "$PWD":/src -w /src rust:1-bullseye sh -c '
  apt-get update && apt-get install -y protobuf-compiler &&
  cargo build --release'

rust:1-bullseye is Debian 11, glibc 2.31, so the resulting binary runs on the 2.31 node.

The lesson: your oldest host sets your glibc floor for every prebuilt binary. Either build for that floor or retire the host. I now had a standing rule that any orchestrator upgrade on that node needs the container build. I also had a fresh argument for replacing the box.

Failure two: status=219/CGROUP

With a working binary in place, the unit still refused to start, now failing differently:

Main PID: ... (code=exited, status=219/CGROUP)

Exit 219 from systemd means it could not set up the service's cgroup. Not a binary problem at all. The node is a Virtuozzo container, and in that environment the systemd cgroup hierarchy has a hard cap:

$ cat /sys/fs/cgroup/systemd/cgroup.subgroups_limit
100

Then the count:

$ find /sys/fs/cgroup/systemd -type d | wc -l
101

At the cap. systemd could not create even one more service cgroup, so any new unit failed with 219, working binary or not.

What filled it was the interesting part. Two contributors. First, hours of crash-looping from failures one and two churned units. Second, and larger, leaked SSH session scopes. Every SSH login created a session-*.scope cgroup that this systemd version never cleaned up on logout. With zero users actually logged in, 96 of the 101 cgroup directories held no processes at all. My own repeated logins to debug the box had been quietly eating the budget.

The recovery was to remove the process-less session scopes, clear the failed state, and start:

for d in /sys/fs/cgroup/systemd/user.slice/user-*.slice/session-*.scope; do
  [ -s "$d/cgroup.procs" ] || sudo rmdir "$d"
done
sudo systemctl reset-failed orca-agent
sudo systemctl start orca-agent

The lesson: on a constrained container, the cgroup budget is a real, exhaustible resource. Automation or debugging that opens a lot of SSH sessions can silently consume it, and the failure it produces (219/CGROUP) points at systemd, not at whatever you were actually trying to start.

Failure three: the placeholder regression

The binary loaded, the unit started, and the cluster was still broken. Every domain on the controller node returned 404 while the containers underneath ran perfectly fine and served correct responses on localhost.

The orchestrator status showed those services as 0/1 stopped. But docker ps showed the containers up for days. The controller thought the services were not running; the containers disagreed.

The release I had just installed added a startup path that registered each service, and it treated services pinned to the controller's own hostname as remote placeholders: entries that mean "this belongs on some other node, wait for that node to report in". The controller then waited forever for a node that was itself, never adopted the already-running local containers, and never registered their routes. Hence 404 at the edge while the containers ran.

If that sounds familiar, it is the same family as the redeploy bug I fixed in part two: the controller's self-registration matching a placement lookup it should be excluded from. My fix covered the redeploy path. This was a different code path, added later, with the same blind spot.

Two things made it stickier than expected. The placeholder registration also persisted a stop-mark, so simply removing the pins from config was not enough to bring the services back; the reconciler still saw them as stopped. And the new release had dropped the CLI start verb, so clearing the mark meant calling the REST endpoint per service:

for s in app-a app-b app-c; do
  curl -s -X POST -H "Authorization: Bearer $TOKEN" \
    "http://127.0.0.1:PORT/api/v1/services/$s/start"
done

The durable fix was in configuration, and it is the through-line of this whole series: do not pin controller-hosted services at all. Unpinned services default to the controller anyway. Only services that genuinely live on a remote node get an explicit pin. I removed every controller pin, kept the one real remote pin, and filed the regression as orca#151.

The through-line

Across all three parts, one trap kept resurfacing: pinning a service to "the controller's own node" is a landmine in this orchestrator. It caused the redeploy 503 in part two and the placeholder 404 here, through two different code paths. The reliable rule is to never name the controller in a placement; let it be the default and reserve pins for remote nodes.

The other lesson is blunter. An end-of-life host turned a routine upgrade into a multi-hour, three-failure incident: glibc too old for the binary, a Virtuozzo cgroup cap I did not know existed, all of it made worse by the debugging itself. The single best fix for two of these three outages is to retire that box. That is now at the top of the list.

What I would keep, and what I would change

Keep:

• Declarative reconcile with git-push deploys. It is fast and legible, and the same property that makes it dangerous makes recovery a git revert.
• Reading the source. Twice it beat guessing outright.
• Verifying in production against the real path, not in my head.

Change:

• Never pin a service to the controller's own node.
• Retire end-of-life hosts before they set your glibc floor and hand you exotic kernel limits.
• Watch the cgroup budget on constrained containers, and do not let your own debugging sessions leak scopes into it.

That is the split, the bug hunt, and the upgrade. Three posts, one small orchestrator, and a much cleaner separation between the things I run for money and the things I run for fun.