DEV Community

Cover image for A Symlink Swap Is Not a Zero-Downtime Deploy
Nasrul Hazim
Nasrul Hazim

Posted on

A Symlink Swap Is Not a Zero-Downtime Deploy

#devops #linux #nginx #php

TL;DR — I had Capistrano-style releases on a plain VM: build into releases/<ref>, swap a symlink, restart the service. Atomic, I thought. Then I put curl in a loop and watched an actual deploy: 502 on every single one, for the length of the app's boot. The files were atomic. The process never was. And the nginx directive everyone reaches for to paper over that does absolutely nothing in the shape most people write it.

The half that was already right

The release layout is the well-trodden one, and there's nothing wrong with it:

/srv/app/
  releases/20260831-101500-a1b2c3d/
  releases/20260830-093000-9f8e7d6/
  current -> releases/20260831-101500-a1b2c3d
Enter fullscreen mode Exit fullscreen mode

Build into a fresh directory, then move the pointer. One detail worth getting right, because two of the three obvious ways to write it are broken:

# WRONG — `-sf` follows an existing symlink-to-a-directory and
# quietly creates /srv/app/current/20260831-101500-a1b2c3d
ln -sf "$RELEASE" /srv/app/current

# STILL WRONG — `-n` fixes that, but this is unlink() then symlink().
# There is a window where the path resolves to nothing.
ln -sfn "$RELEASE" /srv/app/current

# RIGHT — create beside, then rename(2) over the top. Atomic.
ln -sfn "$RELEASE" /srv/app/current.tmp
mv -Tf /srv/app/current.tmp /srv/app/current
Enter fullscreen mode Exit fullscreen mode

mv needs -T for the same reason ln needed -n: without it, moving onto an existing symlink-to-a-directory moves into it.

Get that right and your files switch instantly. A request that started on the old release finishes against the old tree; the next one gets the new one. Genuinely atomic, genuinely nice.

Which is exactly why the next line goes unexamined for months.

The half nobody looks at 

systemctl restart app
Enter fullscreen mode Exit fullscreen mode

restart is stop, then start. Between those two, nothing is listening on your app's port.

t+0.00  mv -Tf current            files switch, atomic ✓
t+0.01  systemctl stop app        port closes
t+0.01  GET /  →  502
t+0.4   GET /  →  502
t+1.2   GET /  →  502
t+2.8   app finished booting, binds port
t+2.9   GET /  →  200
Enter fullscreen mode Exit fullscreen mode

The deploy is atomic right up to step 3. Everything red after it is the part the symlink swap never covered.

Three seconds of hard 502s, on every deploy, recovering by itself. That last part is why it survives so long: nobody files a bug for something that fixed itself before they could screenshot it. It gets called "flaky."

It isn't flaky. It's exactly what you told the machine to do.

Two shapes, two different 502s

If you serve a process on a port, it's the obvious one:

location / {
    proxy_pass http://127.0.0.1:3000;
}
Enter fullscreen mode Exit fullscreen mode

Process dies → port closes → connect() refused → nginx has nothing to say but 502.

The second shape catches people out, and it caught me. Conventional PHP on a VM: nginx owns the docroot and only .php reaches the app.

root /srv/app/current/public;

location ~ \.php$ {
    include fastcgi_params;
    fastcgi_pass unix:/run/app/php-fpm.sock;
    fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
}
Enter fullscreen mode Exit fullscreen mode

The reasoning goes: nginx serves the files itself, so a restart can't matter much. It matters completely — if the fpm master runs under the same systemd unit as the app. Which is often exactly how you want it, because then logs, resource limits and lifecycle all work per-app instead of being shared through one system-wide pool.

The cost of that choice: restarting the unit kills the master, the socket file is unlinked, and fastcgi_pass gets ENOENT until it comes back. Same 502, different mechanism, and the static assets keep serving perfectly the whole time — which makes it look like an application bug rather than a deploy bug.

The directive that does nothing

Everyone's first fix, mine included:

location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_next_upstream error timeout;   # ← inert
}
Enter fullscreen mode Exit fullscreen mode

This changes nothing. proxy_next_upstream retries the next peer in the upstream group. A proxy_pass at a literal host:port is a group of one. There is no next peer, so there is nothing to retry to, and you get the same 502 you got before — now with a config line that makes you think you handled it.

The version that works looks like a typo:

upstream app {
    server 127.0.0.1:3000 max_fails=0;
    server 127.0.0.1:3000 max_fails=0;   # yes, the same address twice
}

location / {
    proxy_pass http://app;
    proxy_next_upstream error timeout invalid_header non_idempotent;
    proxy_next_upstream_tries 3;
    proxy_connect_timeout 2s;
}
Enter fullscreen mode Exit fullscreen mode

Listing the address twice is the entire mechanism. Now there is a next peer, so a refused connect gets retried instead of reported.

max_fails=0 is not decoration either. The default is max_fails=1 fail_timeout=10s, and these two "servers" are one process — so a single refused connect marks both peers down and nginx serves 502 for the next ten seconds. You'd have converted a 200 ms gap into a ten-second outage while believing you'd added resilience.

And while you're in there, stop showing people the nginx default page:

error_page 502 504 = @unavailable;

location @unavailable {
    default_type text/html;
    add_header Retry-After 5 always;
    return 503 '<!doctype html><title>Starting up</title><p>This service is starting. Refresh in a few seconds.';
}
Enter fullscreen mode Exit fullscreen mode

503 with Retry-After, not 502. A process that is starting is not a broken upstream, and the status code is the only part of that a client, a CDN or a health checker can act on.

But be clear-eyed: all of this buys you a retry, not zero downtime. It covers a sub-second gap. It does not cover a three-second boot. For that you need to stop having a gap at all.

What actually fixes it: two of everything

Blue-green, but with systemd doing the work. A template unit gets you both colours from one file:

# /etc/systemd/system/app@.service
[Unit]
Description=app (%i)
After=network-online.target

[Service]
Type=simple
User=app
WorkingDirectory=/srv/app/slots/%i
EnvironmentFile=-/srv/app/slots/%i.env
ExecStart=/usr/bin/node server.js
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

systemctl start app@blue and app@green are now two independent services. The layout grows a slots directory:

/srv/app/
  releases/<ref>/
  current      -> releases/<ref>     the version that is SERVING
  slots/blue   -> releases/<ref>     one colour's release
  slots/green  -> releases/<ref>
  slots/blue.env                     PORT=41000
  slots/green.env                    PORT=41001
Enter fullscreen mode Exit fullscreen mode

And the deploy becomes a sequence rather than a restart:

active=$(systemctl is-active --quiet app@blue && echo blue || echo green)
standby=$([ "$active" = blue ] && echo green || echo blue)

# 1. Build the release. Nothing serving is touched.
build_into "$RELEASE"
printf 'PORT=%s\n' "$STANDBY_PORT" > "/srv/app/slots/$standby.env"
ln -sfn "$RELEASE" "/srv/app/slots/$standby.tmp"
mv -Tf "/srv/app/slots/$standby.tmp" "/srv/app/slots/$standby"

# 2. Start the standby beside the one that is serving.
systemctl restart "app@$standby"

# 3. Prove it answers, on its OWN port — not through the proxy,
#    which is still pointing at the version we are replacing.
curl -fsS --retry 20 --retry-delay 1 "http://127.0.0.1:$STANDBY_PORT/up" >/dev/null

# 4. The switch. A reload does not drop connections in flight.
write_upstream "$STANDBY_PORT"
nginx -s reload

# 5. Only now is the old one expendable.
systemctl disable --now "app@$active"
Enter fullscreen mode Exit fullscreen mode

Same deploy, two colours. Blue keeps answering until step 4, and step 4 is a reload, not a restart.

Step 3 is the one people skip and shouldn't. Probe the standby's own port, never the public URL — the public URL is still answered by the version you're trying to replace, so a probe through it always passes and proves nothing.

Step 4 is the moment traffic moves, and it is the only moment anything visible changes. An nginx -s reload starts new workers for the new config and lets the old workers finish what they're holding. Nobody gets cut off mid-response.

The thing container people never have to think about

If you've only done blue-green with containers, this is the part that doesn't transfer.

Each container gets its own network namespace, so both colours can bind port 3000 and only the network alias tells them apart. The name moves; the number stays.

On a plain host there is one port space. Two processes cannot both bind 3000. So the number is what moves, and the hostname stays 127.0.0.1. Everything downstream follows from that one fact:

  • your proxy repoint has to rewrite the port, not the upstream host;
  • you need somewhere to keep "which port is this colour on" — I write it to slots/<colour>.env and read it back at switch time;
  • pick your slot ports above the registered range and below ip_local_port_range (32768–60999 on most boxes), or you'll eventually collide with an ephemeral port the kernel handed out for an outbound connection. 41000–41999 is a quiet neighbourhood.

And one consequence worth refusing rather than shipping: if your app serves two domains on two ports, a colour only rebinds the primary one. The second domain keeps answering from the colour you are about to kill. That's a half-switched release, which is worse than an honest restart — so detect it and refuse, loudly, with a message that says why.

Two rules I'd argue for in any implementation

A colour's WorkingDirectory points at its own release, never at current. This sounds like a detail and it's the whole thing. Two versions can only run at once if each has its own working directory. Point both at current and you've rebuilt the restart with extra ceremony.

current moves at promote, not at deploy. It should keep meaning "the version that is serving" right up until the switch. That matters because your companion units follow it:

# worker unit — deliberately on `current`, not on a colour
[Service]
WorkingDirectory=/srv/app/current
ExecStart=/usr/bin/php artisan queue:work
Enter fullscreen mode Exit fullscreen mode

Restarting that at promote — and only at promote — is also exactly right for a queue worker. A worker holds the code it started with, and the moment the new version becomes canonical is the moment it should pick it up. Not while it's still a candidate.

Gate on what you can actually reach

Back to the php-fpm shape, because it's where I had to be honest with myself.

A standby fpm master listens on a unix socket and nothing else. There's no port to curl. And the only nginx on the box is still pointing at the colour that's serving — so an HTTP probe "at the standby" through nginx is answered by the version you're replacing. It passes every time. It's not a gate, it's a gate-shaped thing that can only ever say yes.

Real options: stand up a second loopback listener just for the gate, or speak FastCGI to the socket directly. Both are actual work.

What I did instead was record that switch as unverified rather than dressing it up as passed, and keep the checks that genuinely are reachable — did the unit stay up through its first few seconds, and did the pre-start migration exit non-zero. That's less than an HTTP probe and it is not nothing.

An honest "we could not verify this one" is worth more than a green tick you invented. A check that cannot fail is indistinguishable from a check that never ran, and six months later nobody can tell you which one it was.

Prove it with curl, not with your eyes

Whatever you do, the test is two lines and you should run it before you believe anybody, including me:

while :; do
  printf '%s ' "$(curl -s -o /dev/null -w '%{http_code}' http://localhost/)"
  sleep 0.1
done
Enter fullscreen mode Exit fullscreen mode

Leave it running. Deploy. Watch.

before:  200 200 200 502 502 502 502 502 502 502 200 200
after:   200 200 200 200 200 200 200 200 200 200 200 200
Enter fullscreen mode Exit fullscreen mode

That's the entire acceptance criterion, and it's the one your users are actually running.

The takeaway

Atomic file swaps are the easy half, and they're the half that gets all the attention because they're satisfying to get right. Nobody's curl loop can see them.

Ask the boring question instead: between the old process exiting and the new one accepting connections, what answers the phone? If the answer is "nginx, with a 502", you don't have a zero-downtime deploy — you have an atomic file swap and an outage you've agreed not to measure.

And if you're about to fix it with proxy_next_upstream on a single-target proxy_pass: that line does nothing. List the peer twice.

Top comments (0)