DEV Community

Sergey Shinder
Sergey Shinder

Posted on

Our deploy script ran ln -sf on the current link and every release after the first landed inside the one before

Our carrier gateway runs on four hosts we built in September. The systemd unit starts /opt/gateway/current/bin/gateway, and the deploy script was short: unpack the release into /opt/gateway/releases/ followed by the version, run ln -sf from that directory to /opt/gateway/current, restart the service, and check that systemctl reports it active.

The first deploy on each new host worked. Five weeks and six releases later a fix for a carrier's new service codes went out, and the carrier kept rejecting the same codes. ls -l /opt/gateway/current still pointed at the September release, and inside that directory sat six links, one named after each release since.

The trouble is how ln treats its last argument. If it is an existing directory, ln makes the new link inside it, under the target's own name. A symbolic link to a directory counts as a directory, because ln follows it by default, and -f does not change that: it only forces replacement of a file ln has decided to overwrite. On the first deploy current did not exist, so ln created it. On every deploy after that, current was a link to a directory, so ln went through it and created a new link in the September release. The restart started September's code each time, systemctl said active, and the script reported success.

ln has a flag for this, -n, which treats an existing link to a directory as a plain file to be replaced. We did not stop there. ln -sfn removes the old link and then creates the new one, and for that instant there is no current at all, so a restart landing in that gap fails to start. The script now creates the new link under a temporary name, current.new, and moves it over the old one with mv -T. That is a single rename, which replaces the link atomically. After the restart the script compares readlink -f current with the release it unpacked, and the version the service reports with the version it meant to deploy, and fails if either differs.

We cleaned the stray links out of the old release and deployed the fix forty minutes later.

The command had been tested once, and once was the only time it behaves that way. A deploy step has to be tested on its second run.

– Sergey Shinder

Top comments (0)