DEV Community: Jörg Halitschke

rsync -a Stole the setgid Bit, and systemd Blocked the Rollback

Jörg Halitschke — Sun, 15 Nov 2026 17:43:04 +0000

The setup was deliberately boring. A small Node.js app with SQLite on a single VM, a release-directory layout with a current symlink, a CI workflow that uploads each release over SSH, runs npm ci, flips the symlink, restarts the systemd unit and waits for /healthz. If the health check fails, flip the symlink back and restart again.

Every piece of that looked right in review. The first time CI ran it end to end, the deploy went red, the app went down, and the rollback — the part that exists precisely for this moment — failed as well.

Neither failure was in the deploy logic. Both were in what the tools around it did by default.

TL;DR: rsync -a includes -p and -g, so it copies the sender's mode and group onto the target — including onto the release directory itself, which silently clears the setgid bit the server relied on. npm ci then wrote node_modules with the deploy user's private group, and the app, running as a different user that only shares a group, couldn't read its own dependencies. The app crash-looped into systemd's StartLimitBurst, after which systemd refused every start — including the rollback's — until systemctl reset-failed. Fix: normalise group and setgid after the upload, call reset-failed before every restart, and stop waiting on a health check once the unit has visibly crashed.

Why Two Users in the First Place?

Because the process that writes code to the server shouldn't be the process that runs it.

The app runs as an unprivileged runtime user with no sudo and no SSH key. Deploys run as a separate deploy user whose key is restricted and whose sudo rights cover exactly two commands: restarting the unit and reset-failed (why the second one is on the list is most of this post). The two share one group, and the shared directories are set up so both can write:

# once, at setup
chgrp -R app-shared /srv/app
chmod -R g+rwX /srv/app
find /srv/app -type d -exec chmod g+s {} +   # new files inherit the group

The setgid bit on a directory is what makes this work without either user having to think about it: every file created inside inherits the directory's group instead of the creator's primary group. Combined with umask 0007 on both sides, everything the deploy user creates is group-readable and group-writable, and everything outside the group sees nothing.

That's the property the first deploy lost.

What Did rsync Actually Change?

The workflow creates the release directory first, then uploads into it:

ssh deploy-target "mkdir -p '/srv/app/releases/${REL}'"
rsync -az ./ "deploy-target:/srv/app/releases/${REL}/"

The mkdir happens inside releases/, which has setgid, so the new directory is created correctly: shared group, setgid set. Then rsync runs, and -a is shorthand for -rlptgoD. Two of those letters do the damage:

-p copies permissions from the sender. On the CI runner, the checkout directory has no setgid bit — why would it. A trailing slash on the source (./) means rsync syncs the attributes of the directory itself as well as its contents, so it chmods the freshly created release directory to the runner's mode. setgid gone.
-g copies the group. A non-root receiver can only assign groups it is a member of, so rsync falls back to the deploy user's own primary group for everything it creates.

Nothing about this looks wrong in a listing unless you're looking for it. The files are there, the sizes match, the owner is the deploy user as expected. The difference is one letter in the mode column and one name in the group column.

The next step made it fatal. npm ci runs on the server, as the deploy user, inside a release directory that no longer has setgid. Every file in node_modules got the deploy user's private group, with umask 0007 stripping all "other" bits. The runtime user — different UID, member only of the shared group — couldn't read a single dependency. require('express') failed on the first line of server.js.

Why Didn't the Rollback Save It?

Because by the time the script tried to roll back, systemd had stopped taking orders.

The unit had sensible crash-loop protection:

[Unit]
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=5

Crash, wait five seconds, restart, crash again. Five starts inside five minutes and systemd marks the unit failed with start request repeated too quickly. That's exactly the right behaviour for a broken binary at 3 a.m. — it stops a crash loop from hammering the machine.

What's less obvious is that the start limit doesn't distinguish who's asking. Once it trips, a manual systemctl restart is refused just like the automatic ones. The deploy script flipped current back to the previous, known-good release and called restart — and systemd rejected it, because the limit belongs to the unit, not to the code the unit happens to point at.

So the sequence was: new release can't read its dependencies, unit crash-loops through its burst, health check eventually times out, rollback flips the symlink correctly and then can't start anything. The old release was sitting on disk, intact, one symlink away, and the service stayed down until someone ran reset-failed by hand.

What Does the Fix Look Like?

Three changes, each small, each aimed at one of the mechanisms rather than at the symptom.

Normalise the release after upload, before anything runs in it. The group to use is read from releases/ itself, so the script doesn't hard-code a group name:

GRP="$(stat -c %g "$BASE/releases")"
find "$REL_DIR" -exec chgrp "$GRP" {} +
find "$REL_DIR" -type d -exec chmod g+rwxs {} +

This runs before npm ci, so node_modules is created under a setgid directory again and inherits the right group. An alternative is to change the rsync flags — --no-p --no-g --chmod=Dg+s,ug+rw — and that works, but it puts the correctness of the server layout into a CI file that lives somewhere else. The find after the fact is idempotent and lives next to the thing it protects.

Reset the start limit before every restart, not just before rollbacks:

restart_service() {
  sudo -n systemctl reset-failed "$SERVICE" 2>/dev/null || true
  sudo -n systemctl restart "$SERVICE"
}

reset-failed on a healthy unit is a no-op, so there's no reason to make it conditional. It's also why the deploy user's sudo rule has to name it explicitly — a rule that only allows restart reproduces the original failure with a permission error on top.

Stop waiting once the unit has visibly crashed. The original health check polled /healthz ten times, three seconds apart, plus curl timeouts — up to about 35 seconds of a dead app behind the reverse proxy returning 502. But there's no point polling an HTTP endpoint on a process systemd has already reported as dead:

healthy() {
  for i in $(seq 1 "$HEALTH_RETRIES"); do
    curl -fsS --max-time 5 "http://127.0.0.1:${PORT}/healthz" >/dev/null 2>&1 && return 0
    n="$(systemctl show -p NRestarts --value "$SERVICE" 2>/dev/null || true)"
    if [ "$(systemctl is-active "$SERVICE")" = "failed" ] || [ "${n:-0}" -ge 1 ]; then
      log "service crashed - not waiting for the health check any longer"
      return 1
    fi
    sleep 3
  done
  return 1
}

A unit that has restarted even once since the deploy's own restart is not going to come good on the next poll. Failing fast here shortens the outage of a bad release from half a minute to a few seconds, and gets to the rollback while the start limit still has budget left.

Couldn't This Be Caught Before It Reaches Production at All?

Yes — and that became the next change. The three fixes above make a broken release fail faster and recover reliably. They don't stop a broken release from being switched in. The deploy now boots every release against a copy of the production database on a spare port before touching current, and the very first thing that preflight does is look for exactly the failure from this post: files in the release that the shared group can't read.

bad="$(find "$REL_DIR" \( -type d ! -perm -g=rx -o -type f ! -perm -g=r \) | head -n 3)"
[ -z "$bad" ] || { log "not group-readable: $bad"; return 1; }

That check exists because the preflight itself runs as the deploy user, not the runtime user — so a permissions problem would sail straight through a "does it boot" test. The rest of that design is the next post.

What's the Generalizable Lesson Here?

rsync -a is "archive", and an archive preserves the sender's metadata. When the receiver's metadata is the part that matters — shared groups, setgid, ACLs — either strip -p/-g explicitly or normalise afterwards. A trailing slash on the source applies the source directory's attributes to the target directory too.
setgid on directories is a quiet invariant: nothing breaks when it's removed, until the next file is created. Anything that relies on it should re-assert it, not assume it.
With separate deploy and runtime users, "it works for me" during deploy proves nothing about the runtime user. Check readability as the group, not as yourself.
systemd's start limit is per unit and blocks manual starts as well as automatic ones. Any script that restarts a unit after a possible crash loop — a rollback above all — needs reset-failed first, and the sudo rule needs to permit it.
Don't keep polling an HTTP health endpoint on a process the supervisor has already declared dead. Ask the supervisor first; it knows sooner.
A rollback that has never run against a real failure is a hypothesis. The interesting state isn't "previous release on disk" — it's "unit just crash-looped through its start limit", and that's the one to rehearse.

Boot the Release Against a Copy of Production Before You Switch

Jörg Halitschke — Sun, 15 Nov 2026 17:43:02 +0000

The deploy script already had a rollback. Switch the current symlink, restart the systemd unit, poll /healthz, and if that fails, switch back and restart again. On paper that's a safe deploy.

In practice, every broken release still produced an outage. The reverse proxy returned 502 from the moment the old process stopped until the rollback had brought it back — up to about 35 seconds in the worst case, and a few seconds even after the health check learned to fail fast. A post-switch health check can't prevent that. It runs after the damage, and its only job is to bound it.

The fix was to move the question "does this release start?" to before the switch, and to ask it against the real data.

TL;DR: Before touching current, the deploy now copies the live SQLite database with VACUUM INTO, starts the new release on a free localhost port against that copy with a throwaway config and session secret and every outbound job disabled, and waits for /healthz. If it doesn't come up, the deploy aborts with the test server's last log lines and production never notices. The first check in the preflight is whether the release is readable by the runtime group — because the preflight runs as the deploy user and would otherwise miss permission bugs entirely. A deliberately broken release was then pushed through the real pipeline: the deploy failed, the site served HTTP 200 throughout.

What Does a Post-Switch Health Check Actually Protect?

Less than it looks like. The sequence is:

switch current to the new release
restart the service — the old process is gone now
poll /healthz
on failure: switch back, restart, poll again

Everything between step 2 and the end of step 4 is downtime if the release is broken. The health check decides whether you stay down, which is valuable, but the outage itself is already happening by the time it reports anything.

What breaks a release in a small Node.js app is rarely subtle. A missing package after npm ci, a syntax error in a file the tests didn't load, a schema migration that throws on real data, a file the runtime user can't read. All of those fail within the first second of node server.js. None of them needs the release to be live to show up. They just need the release to start.

How Do You Start a Second Copy Without Touching the First?

Three things have to be separate from production: the port, the data, and anything the app does to the outside world.

The port. Ask the kernel for one instead of picking a number:

port="$(node -e 'const s=require("net").createServer().listen(0,"127.0.0.1",()=>{console.log(s.address().port);s.close()})')"

There's a tiny race between closing that socket and the app binding it, and on a single-tenant VM it doesn't matter. Binding to 127.0.0.1 matters more — the preflight instance must never be reachable from outside, not even for the few seconds it lives.

The data. The app runs on SQLite in WAL mode, which means copying the .db file with cp while the app is running can produce a file that's missing whatever is still sitting in the -wal file, or one that's torn mid-write. VACUUM INTO asks SQLite itself for a consistent snapshot, WAL contents included, as a single self-contained file:

vacuum_into() {
  TARGET="$1" node -e '
    const { DatabaseSync } = require("node:sqlite");
    const db = new DatabaseSync(process.env.DB_FILE);
    db.exec("VACUUM INTO \x27" + process.env.TARGET.replace(/\x27/g, "\x27\x27") + "\x27");
    db.close();
  '
}

It reads while the live app keeps writing, and it's the same call the deploy already used for its pre-switch backup. Using the real data rather than a fixture is the point: migrations run on startup, and a migration that works on the test schema but fails on a production row is precisely the bug a fixture can't find.

The side effects. This is the part that's easy to get wrong. A second instance of the app is a second instance of everything the app does on startup — including its scheduler. This one syncs orders from an external marketplace every couple of hours and sends mail. A preflight instance that starts its scheduler would make real API calls with real credentials, and if it ever pointed at the wrong database, import orders twice. So the preflight instance is made inert:

( cd "$REL_DIR" && exec env \
    PORT="$port" BIND_HOST=127.0.0.1 \
    DB_FILE="$tmp/app.db" CONFIG_DIR="$tmp/config" \
    NODE_ENV=production EXTERNAL_SYNC_DISABLED=true \
    SESSION_SECRET="$(head -c 24 /dev/urandom | od -An -tx1 | tr -d ' \n')" \
    node server.js ) >"$tmp/server.log" 2>&1 &

Its config directory holds only the checked-in template, not the local file with SMTP and API credentials. Its session secret is random, so a cookie from the live app means nothing to it. And the sync job has an off switch — which is a design requirement worth adding before you need a preflight, because retrofitting "don't do anything on startup" into an app that assumes it's the only instance is much harder than it sounds.

When Has the Preflight Passed?

When /healthz answers. When the process dies, it has failed — and there's no reason to keep polling a dead PID:

for i in $(seq 1 40); do
  kill -0 "$pid" 2>/dev/null || break
  if curl -fsS --max-time 2 "http://127.0.0.1:${port}/healthz" >/dev/null 2>&1; then ok=0; break; fi
  sleep 0.5
done
kill "$pid" 2>/dev/null || true
wait "$pid" 2>/dev/null || true

On failure the script prints the last fifteen lines of the test server's log into the CI output, trimmed to 300 characters each. That turned out to be the most useful part of the whole change: a failed deploy now says why in the same place it says that, instead of sending someone to journalctl on the server to find a stack trace from a process that's already been replaced.

/healthz itself is deliberately cheap — it runs SELECT 1 and returns 503 if the database layer throws. By the time it can answer at all, the app has loaded every module, opened the database, and run its startup migrations against the copy. That's the whole set of things this check is for. Whether the business logic is right is the test suite's job, and it ran before the deploy started.

What Can the Preflight Not See?

Being explicit about this kept the check honest.

It runs as the deploy user, not the runtime user. The deploy user owns the release and can read everything in it. The runtime user only shares a group. A file that isn't group-readable boots fine in the preflight and crashes in production — the exact failure from the previous post. So before it starts anything, the preflight checks permissions directly:

bad="$(find "$REL_DIR" \( -type d ! -perm -g=rx -o -type f ! -perm -g=r \) 2>/dev/null | head -n 3)"
if [ -n "$bad" ]; then
  log "files not readable by the group (the app could not open them):"
  printf '    %s\n' $bad
  return 1
fi

It doesn't exercise integrations. Outbound sync and mail are off by design. A broken API client starts fine and fails two hours later. That's a trade, not an oversight — the alternative is a preflight with real side effects.

It can't run on the first deploy. No database yet, nothing to copy. The script logs that it's skipping and carries on; the first deploy initialises the schema instead.

It doubles memory for a few seconds. On a small VM that's worth checking once: on a box already near its memory limit, a preflight could trigger the OOM killer on the live process, which would be a fairly ironic way to cause an outage.

How Do You Know It Works?

Two ways, and both are needed.

In the test suite, the deploy script runs for real against a temporary directory layout. npm, sudo and systemctl are stubs; node, curl and bash are real. One release's server.js is a one-line HTTP server that answers /healthz. The other is one line too:

throw new Error('deliberately broken');

The tests assert that the broken release aborts the deploy, that current was never switched, that no pre-switch backup was taken, and that the thrown error text appears in the output. A third case drops group-read from server.js and expects the permission check to name the file; a fourth lets the healthy release through and checks it switches only after preflight. A deploy script with no tests is a deploy script whose failure branches have never run — and the failure branches are the whole reason it exists.

Against the real server, a deliberately broken release went through the real CI pipeline while the site was watched from outside. The deploy went red with the thrown error in its log, and the site answered HTTP 200 the entire time. That's the only result that actually proves the claim, because it's the only one that includes the reverse proxy, systemd, and the real filesystem.

What About the Restart That Still Happens?

A release that passes preflight still needs a restart, and a single-process app on a single VM has a gap of a second or two while that happens. Two small changes make the gap invisible in practice rather than eliminating it:

The reverse proxy maps 502/503/504 to a static maintenance page served from outside the release directory, returned as 503 with Retry-After rather than a bare 502. Crawlers and monitors read that as "temporary", humans see a sentence instead of an error page.
The frontend retries idempotent GET requests briefly when it sees a 502/503/504, instead of treating any non-2xx as "session expired" and bouncing the user to the login screen — which is what it had been doing.

True zero-downtime would need two processes and a handover, which for an app this size is more machinery than the problem justifies. The preflight moved every predictable failure out of the restart window. What's left is a planned blip, not an outage.

What's the Generalizable Lesson Here?

A health check after the switch measures an outage; it doesn't prevent one. Anything that can fail at startup can be checked before the switch, against a non-serving instance.
Test against a copy of production data, not a fixture. Startup migrations are the most likely thing to fail on real rows, and the least likely to be covered by test data.
Copy a live SQLite database with VACUUM INTO, never cp. In WAL mode a file copy can be missing committed data or be torn mid-write; VACUUM INTO gives you one consistent file while the app keeps running.
A preflight instance must be inert: localhost-only, its own port, its own data, a throwaway secret, no schedulers, no outbound calls, no real credentials. Design the off switches in early — an app that assumes it's the only instance is hard to start twice.
Write down what the preflight can't see. Here: it runs as the wrong user, so it checks permissions explicitly; it skips integrations, by design.
Put the failed instance's last log lines into the CI output. A failed deploy that explains itself in the place it fails saves the trip to the server.
Prove it end to end once with a deliberately broken release and an external probe. The test suite proves the script's logic; only the real run proves the proxy, the supervisor and the filesystem agree with it.