I Left Sixty-Eight Backups; nginx Read Exactly One

Before I touch a live configuration file, my hands type the same line on their
own: cp -p file file.bak-$(date +%Y%m%d-%H%M%S). An old reflex. If the change goes
wrong, the copy is right there, two keystrokes away. On the evening of 28 September, while reworking the blog's nginx
configuration, that line got typed again.

Four days later the copy was still there. Not merely there — inside the
directory nginx reads. Over those four days nginx restarted once
(30 September, 21:09) and reloaded its configuration seven times; every one of
those times it parsed that backup too, took its server block into memory, and
told me so, in sixteen lines of warnings. That nobody read those warnings for four days is an inference
from the backup still being there: once the "test is successful" line at the
bottom of nginx -t shows up, the job counts as done and the sixteen lines
above it go unread.

The backup did not take over the site. Nothing I did prevented it. This piece
is about what that "nothing" actually was: the safety of the habit comes not
from me but from two conventions I never chose.

First, the count: how widespread is the habit

Wondering about one file is no reason to look at only one file. A scan of /etc
down to three levels, for every name that smells like a backup:

sudo find /etc -maxdepth 3 \
  \( -name "*.bak" -o -name "*.bak-*" -o -name "*.old" -o -name "*.disabled" \
     -o -name "*.orig" -o -name "*.save" -o -name "*~" -o -name "*.pre-*" \
     -o -name "*.before-*" -o -name "*.next" \) -type f | wc -l

The answer: 68. Broken down by directory:

Directory	Copies	How does the system read this directory?
/etc/nginx/sites-available	16	Never read (link targets only)
/etc/systemd/system	10	Only recognised unit suffixes
/etc (root)	7	Files are called by exact name
/etc/nginx/snippets	4	Only via include snippets/<name>
/etc/nginx/conf.d	2	include conf.d/*.conf — suffix filtered
/etc/nginx/sites-enabled	1	include sites-enabled/* — everything

Those six rows account for 40 files. The remaining 28 are scattered two and
three at a time across /etc: application config directories, ssh, apt,
fail2ban — and five of them in the two directories already set aside for the
purpose, /etc/nginx/backups and /etc/nginx/disabled-backups. The blog's deploy
directory holds five more; those come later.

Those three system directories got a separate check, because the claim in the
title depends on them too: the line in sshd_config reads
Include /etc/ssh/sshd_config.d/*.conf, so it is suffix filtered, and apt and
fail2ban go by extension as well. In all three, my backups are not read. The
bare wildcard exists in exactly one place.

The table was a relief, and immediately afterwards an irritation. The relief:
67 of the 68 copies sit where the system does not care. The irritation comes
from the same place — which ones the system ignores was not something I knew
while dropping the copy. If sixty-seven of sixty-eight shots land, that is not
marksmanship; that is the shape of the distribution.

Which directory gets read, and which does not

What draws the line is not my attention. It is two lines inside nginx.conf:

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

The first imposes a suffix. The file
mustafaerbay-cache.conf.bak-20260928-191601 does not end in .conf, so it
never enters the conf.d sweep — my backup sits there like an invisible stone.
The second imposes nothing. A bare * makes everything in that directory part
of the configuration: .bak, .disabled, the ~ file an editor left behind,
all of it.

Debian's sites-available / sites-enabled pair exists precisely for this. The
real file lives in sites-available, and sites-enabled holds a symlink to it.
In that arrangement, a copy dropped "beside" the file lands automatically on the
sites-available side — the side nginx does not look at. The convention itself
is a guardrail.

On the server, sites-enabled holds 45 entries. Of those, 41 are symlinks and
4 are real files. The blog's configuration is one of those four. Its backup is
the second. So the copy went into one of the four places where the guardrail is
switched off, and that is not a surprise either — the reason a real file lives
there is also me. The live configuration now ships from the repository; the file
on the server is byte-for-byte identical to the 222-line version under
deploy/nginx-live/. Automation wrote a plain file where the distribution
expected a symlink. And then a copy went next to the plain file.

nginx did not pass over this in silence. Its nginx -t output carries sixteen
lines:

nginx: [warn] conflicting server name "mustafaerbay.com.tr" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "www.mustafaerbay.com" on [::]:443, ignored
...

Four hostnames (mustafaerbay.com.tr, mustafaerbay.com and the www form of
each) times four listening sockets (IPv4/IPv6 over 80/443) equals sixteen
warnings. Each says the same thing: you defined this name twice, and the second
one is being ignored.

So which one does it ignore?

The word "second" is the crux here, because the ordering is not mine. Rather
than touch production, the question went to a lab: nginx 1.29.8 in Docker, two
configuration files, the same server_name, the same listening socket. One
returns GERCEK-KONFIG, the other YEDEK-KONFIG. The files sit under conf.d,
but the include line was switched to the server's risky pattern —
include /etc/nginx/conf.d/*;, a bare wildcard. Otherwise conf.d's own suffix
filter would have cancelled the whole experiment.

In the first run the backup carries a suffix, as on the server: site and
site.bak-20260928.

# nginx -T load order
/etc/nginx/conf.d/site
/etc/nginx/conf.d/site.bak-20260928

$ curl -H 'Host: lab.example.com' ...   → GERCEK-KONFIG
$ curl -H 'Host: baska.example.com' ... → GERCEK-KONFIG

In the second run a single thing changed — the backup was renamed to bak-site.
Same content, same line count, same permissions. Only the name now sorts ahead
of the real one:

# nginx -T load order
/etc/nginx/conf.d/bak-site
/etc/nginx/conf.d/site

$ curl -H 'Host: lab.example.com' ...   → YEDEK-KONFIG
$ curl -H 'Host: baska.example.com' ... → YEDEK-KONFIG

The backup took over the site. It also answered the non-matching Host header,
because as the nginx documentation puts it, the default server is the first
block listed for that port, and "the default server is a property of the listen
port and not of the server name".

Those lab files are synthetic; their only job was to report which one answers.
The lab ran 1.29.8 while the server runs 1.30.0; nothing changed on the include
side — the glob call in the nginx source still reads the same on today's
master.
The backup on the server, by contrast, holds the 29 July version: 228 lines, 28
of them different from the live file. Had the name sorted the other way, a
two-month-old configuration would have been serving the site.

The ordering comes from POSIX. nginx expands the pattern with glob() and passes
zero as the second argument:

n = glob((char *) gl->pattern, 0, NULL, &gl->pglob);

Since GLOB_NOSORT is not set, sorting stays on. The glibc
manual on the server is
terse: "By default, the returned pathnames are sorted."
POSIX
says what that order actually is: "Ordinarily, glob() sorts the matching
pathnames according to the current setting of the LC_COLLATE category." So file
names decide which configuration serves the site — though "alphabetical" is not
quite the right word, since the server's locale sets the order.

This is where it became clear that luck was not the operative force, and that is
the part that nags: the file.bak-timestamp shape is structurally safe.
Because it appends a suffix rather than a prefix, the copy always falls behind
the original name; the string site is a prefix of site.bak-20260928, and the
shorter one sorts first — a relation that holds whatever the collation setting
is. Had the reflex settled on bak-file, 00-file or
old-file instead, this would be an incident report. The
safety of the habit comes from the direction of the naming, not from its purpose.

One habit, three other directories, three different verdicts

The interesting part is that the same cp reflex produces entirely different
outcomes elsewhere.

All three verdicts got measured.

The systemd unit directory. /etc/systemd/system holds ten in-place
backups: alert-monitor.service.bak-20260703-131021,
kopru-gateway.service.bak-20260730-115225,
vpsman-agent.service.bak-20260911-1550 and their siblings. systemd loads none
of them, because none carries a recognised unit suffix. Asked directly, it is
unambiguous:

$ systemctl status alert-monitor.service.bak-20260703-131021
Unit alert-monitor.service.bak-20260703-131021.service could not be found.

It appended .service to the name and found nothing. The list of loaded unit
files matches zero of these patterns. All ten copies are entirely harmless — and
entirely invisible. Anyone opening that directory has to read timestamps to work
out which file is live.

Drop-in directories. Here the rule inverts. In the words of the
systemd.unit(5) documentation, files with the .conf suffix "will be merged in
the alphanumeric order and parsed after the main unit file itself has been
parsed", and multiple drop-ins with different names "are applied in lexicographic
order". Whatever is applied later overrides the same setting from earlier. In
nginx the first wins; here the last wins. Two directories, two opposite rules,
one habit.

Two units on the server have two drop-ins each. Under kopru-web.service.d sit
20-live-readiness.conf and order-key.conf, and systemctl cat prints them in
that order. They do not collide, because each defines a differently named
credential and that setting is the kind that appends to a list. But that is
luck: had both written the same single-valued setting, order-key.conf would
have won, and my assumption that the numbered file takes precedence because of
its number would have survived untested.

cron.d. Here the trap runs in reverse. Debian's cron(8) manual states that
names in /etc/cron.d "must consist solely of upper- and lower-case letters,
digits, underscores, and hyphens"; a file containing dots is ignored. Copy a
cron.d job to job.bak, delete the original, and what remains is a file that
never runs and never complains. My own cron.d holds ten entries: nine ordinary
jobs and a tenth called .placeholder — a file silently ignored exactly as that
rule intends.

The real cost of a copy is not being read

Three of the seven copies at the root of /etc are, as their names make plain,
backups of files that carry credentials: one access-token file and two env
files. Their permissions checked out — all three are -rw------- and
root:root, same as the originals. There is no exposure here.

The risk sits elsewhere: these copies freeze the old value of a token. Rotate the
token and the copy does not rotate with it; what remains inside is either a
string that no longer works or — worse — one that still does, because it was
never revoked. Sitting in /etc as a date-stamped file, in no inventory.

On all three, the name and the timestamp disagree. Two of them carry 20260914
in the name while their content was last written on 25 June — eighty-one days
apart; on the third the gap is twenty-one days. The copies were made in a way
that preserves timestamps, so the name records when the copy was taken and the
timestamp records when the content was written.
Anyone reading that directory without knowing the convention trusts the wrong
file.

The alarm that rings about the wrong thing

While counting backups, the blog's own working tree came under the same look,
because the live site is served from it: the blog-serve container mounts
/opt/mustafaerbay read-only as /app and runs dist/server/entry.mjs from
inside it. In that directory git status reports six modified tracked files and
a HEAD stuck on 16 May. Today is 2 October; 139 days sit in between.

On first read that looked alarming. A four-and-a-half-month-old HEAD on a live
tree, six dirty files — as though edits nobody knows about had piled up on the
server. Then each file went head to head with today's origin/main:

File reported as "modified"	Against today's origin/main
deploy/backup.sh	Identical (84 lines)
scripts/notify-subscribers.mjs	Identical (278 lines)
package.json	Identical
package-lock.json	Identical
deploy/pipeline-health.sh	One line differs (line 207)
deploy/update.sh	Not in the repository at all

So four of the six files git status calls "modified" have not changed at all;
they merely look different against an old HEAD. The dirt is not drift, it is a
stale reference. The alarm rings, but points at the wrong thing — and because it
has pointed at the wrong thing for four and a half months, the habit of looking
at it is long gone.

The one-line difference turned out to be a familiar face. The health script on
the server still prints the repository's old address in its alert emails:
github.com/mustafa_itwise/mustafaerbay. After the repository moved to the
organisation on 28 September, three scripts still calling the old name got
cleaned up that day;
this is the fourth. It was inside my own alerting machinery — in the very line
meant to warn me when something breaks.

The debt that stays quiet

The last row of that table is the real finding. deploy/update.sh was deleted
from the repository on 26 June, as part of the incremental-build work. The copy
on the server was last modified on 14 September. A script that does not exist
in the repository was touched 80 days after its deletion, and the copy beside it
carries that same day's migrate tag in its name. I am the only person who does
that work on that server. It has three
siblings: update.sh.bak-1780461942 and update.sh.bak-20260914-migrate (both
3 June, 07:45) and update.sh.disabled (11 May).

Worse, a unit file still calls it:

Description=mustafaerbay.com deploy pull job
ExecStart=/bin/bash /opt/mustafaerbay/deploy/update.sh

The good news: the timer is off. For mustafaerbay-deploy.timer, is-enabled
answers disabled; the service is static and inactive, with not one journal
line since 26 June. The bad news: that timer's content reads
OnUnitActiveSec=1min. A single systemctl enable --now would start running,
once a minute, a script no repository knows about. The gun is not loaded, but it
sits next to the trigger with no label saying which repository, which version.

The same directory also holds an 830 MB dist-persistent: 13,445 files, the
newest dated 29 June. Leftovers from a build experiment that never merged.
Next to it, dist-old was refreshed today — that one is not a backup but the
rollback copy the deployment rotates on every run. The two sit side by side, one
part of a live mechanism, one a ninety-five-day-old fossil. In a directory
listing they are indistinguishable.

Where the copy belongs

After the measurements, the rule got sharper, and all of it hangs on one
question: does anything read this directory, and by what criterion?

• If the directory is read with a bare wildcard (*), a copy goes there never. For the blog's configuration the right place is sites-available or a separate backup directory. The server already has /etc/nginx/backups and /etc/nginx/disabled-backups — past me did the right thing, then forgot.
• If it is read with a suffix filter (*.conf), the copy is safe but invisible. Invisibility is not free: three months later you work out which file is live from its timestamp.
• Know the direction of the sort. In nginx a conflicting name goes to whichever loads first — that one is not in the documentation; it is behaviour measured in a lab and in the source. In systemd drop-ins it goes to whatever applies last, and that one is documented. Naming a file 00- moves it to the front in one system and to the back in the other.
• Use a suffix, not a prefix. file.bak-timestamp falls behind the original name; bak-file jumps ahead of it. That single structural detail is what makes the habit safe.
• If configuration ships from a repository, break the symlink convention deliberately, not by accident. A deploy script that writes a plain file also switches off the protection the distribution handed you.
• Build the detection too, because nginx -t returns zero alongside those sixteen warnings — measured in the lab. No deployment gate will ever see them. Three checks are enough: treat warnings as fatal with nginx -t 2>&1 | grep -q "\[warn\]", read the real load order with nginx -T | grep "# configuration file", and count the non-symlinks with find /etc/nginx/sites-enabled -maxdepth 1 -type f (mine returned four).
• If you leave a backup of a file that carries credentials, add that backup to the rotation list too. Even a copy with correct permissions can keep an unrevoked old secret alive indefinitely.
• For a git working copy sitting on a live tree, either keep HEAD current or stop believing git status in that directory. There is no middle ground: a stale HEAD manufactures fake debt daily, and hides the real debt inside that noise.

Conclusion

Of sixty-eight copies only one mattered, and what kept that one from doing harm
were two conventions I had never thought about: a naming habit that grows by
suffix, and Debian's symlink farm. Neither is my design. One arrived through
muscle memory, the other comes bundled with the distribution.

The finding that stings most is the warning lines. For four days nginx said, in
sixteen lines, "you defined this name twice"; those lines got skipped, and the
one at the bottom — "test is successful" — was taken as the end of the job. The git status output that shows six dirty files has been lying
every day for four and a half months, so I stopped looking at it. An alarm that
rings about the wrong thing is more dangerous than one that stays silent,
because it trains you against your own instrument. The day half-finished work
got counted four ways and
produced four different answers
taught the same lesson; apparently learning it once is not enough.

The copy left inside that directory was not a safety net. It was part of the
configuration. What separated the two was the alphabetical order of its name.

Official Sources

• nginx — How nginx processes a request
• nginx — ngx_core_module: include
• nginx — Server names
• nginx source — ngx_open_glob (src/os/unix/ngx_files.c)
• systemd — systemd.unit(5) manual source
• Debian — cron(8)