One Second Without DNS, Eight Hours Offline

A syndication job noticed before I did

A scheduled task publishes one blog post a day to a developer community. It fetches the article from my own site, converts it, and posts it. At 10:00 it failed four times with this:

Server error '521 <none>' for url
'https://neuragrowth.co/blog/schema-grammar-ceiling/'

521 is Cloudflare saying the origin server did not answer. So the interesting failure was not in the syndication job at all. My whole site was down, and had been for over three hours by then.

The server itself was fine: four days of uptime, load under 0.2, disk at eight percent. But systemctl is-active nginx said failed, and nothing was listening on 80 or 443.

nginx resolves your upstreams before it starts

The journal had the whole thing in three lines:

06:49:54 systemd[1]: Stopping nginx.service...
06:49:54 nginx[36027]: [emerg] host not found in upstream
                       "example-backend.tld" in
                       /etc/nginx/sites-enabled/site:104
06:49:54 nginx[36027]: nginx: configuration file test failed

Line 104 was a small proxy I had added months earlier so the public site could forward one form endpoint to a backend on a different host without revealing its name:

location = /api/lead-capture {
    proxy_pass https://example-backend.tld/api/lead-capture;
    proxy_ssl_server_name on;
    proxy_set_header Host example-backend.tld;
}

When proxy_pass contains a literal hostname, nginx resolves it while parsing the configuration, and treats failure as a fatal config error. That resolution happens inside ExecStartPre=/usr/sbin/nginx -t, so a name it cannot look up means the unit never starts.

The config was not wrong. It was valid before the restart and valid after, and nginx -t passed by hand seven hours later. It was invalid for about one second.

Why DNS was gone for exactly that instant

Ten seconds of journal, reconstructed:

06:49:44  apt-daily-upgrade.service starts
06:49:53  "Reexecution requested ... (unit apt-daily-upgrade.service)"
06:49:53  systemd reexecuting (it had just upgraded itself)
06:49:53  certbot.timer fires in the same second
06:49:54  nginx stopped; ExecStartPre fails on DNS
06:59:20  first alert from a watchdog on the other site
14:29:32  a human starts nginx by hand

The unattended upgrade included systemd itself, which triggered a re-exec, which took the local resolver with it for a moment. Services restarted in that window had to survive without DNS. nginx did not.

None of this is exotic. It is the default upgrade timer on a default Ubuntu install, hitting a default nginx package. The only unusual thing was that it happened at 06:49 instead of during the night.

Make it a request-time lookup

nginx has a documented way out: if the upstream in proxy_pass comes from a variable, resolution is deferred to request time and needs a resolver.

resolver 127.0.0.53 valid=30s ipv6=off;

location = /api/lead-capture {
    set $backend "example-backend.tld";
    proxy_pass https://$backend/api/lead-capture;
    proxy_ssl_server_name on;
    proxy_set_header Host example-backend.tld;
}

The trade is worth naming, because it is the whole point. Before: a DNS hiccup at startup takes down every page on the server. After: a DNS hiccup at request time breaks one endpoint while everything else keeps serving. Same failure, two orders of magnitude less blast radius.

One check worth running afterwards, because a symlinked sites-enabled quietly hides files from a recursive grep:

grep -rn "proxy_pass" /etc/nginx/sites-available/ \
                     /etc/nginx/sites-enabled/

Anything pointing at a literal hostname is a startup dependency on DNS. Anything pointing at an IP or a unix socket is not. My first grep returned nothing at all and I nearly took that as a clean result; it had simply not followed the symlinks.

The outage was not caused by DNS

The DNS failure lasted about a second. The outage lasted seven hours and forty minutes. Those two numbers are not the same problem, and fixing the first one does not fix the second.

nginx ships with Restart=no. A failed start is final until a person intervenes. So the layer above the config fix is a drop-in:

# /etc/systemd/system/nginx.service.d/restart.conf
[Unit]
StartLimitIntervalSec=600
StartLimitBurst=40

[Service]
Restart=on-failure
RestartSec=15

The numbers matter as much as the directive. Forty attempts fifteen seconds apart is ten minutes of patience: long enough that a transient cause is gone by the second try, short enough that a genuinely broken config still ends in failed rather than looping forever. An unbounded restart on a bad config is not resilience, it is a service that lies about its state.

And say the uncomfortable part out loud: as of writing, that restart policy has been verified only by reading systemctl show. It has not been tested by actually killing nginx, because the honest test costs a few seconds of downtime on a shared host. An untested safety net is a belief, not a net, and it goes on the list rather than in the win column.

Nothing told me for three and a half hours

A second application on the same box had its own watchdog. It noticed in nine minutes and sent mail. My site had a health check too, running once a day at 04:15, which had already passed for the day two and a half hours before the outage began.

So the thing that eventually surfaced the failure was a syndication job at 10:00 that could not fetch an article. And that job recorded the error in a database row and said nothing else, which is the same as saying nothing at all. An error nobody hears is not a smaller problem than an error nobody records.

That got fixed too: the job now raises one alert per day when it fails, and the message distinguishes "our own site is down, go look at the web server" from "the destination is unhappy, go look at the API key". Those are different first moves, so they deserve different sentences.

If you run nginx in front of anything

• A literal hostname in proxy_pass is a startup dependency on DNS. Move it into a variable with a resolver and the dependency moves to request time, where a failure costs one endpoint instead of the whole server.
• Check Restart= on anything you would notice being down. nginx defaults to no restart. The default is defensible; relying on it without knowing is not.
• Bound your restart policy. Retries have to outlast a transient cause and give up on a permanent one, or you have swapped a visible outage for an invisible loop.
• A once-a-day health check is a report, not a monitor. If it runs at 04:15, an outage at 06:49 is invisible for nearly a full day.
• A recorded error that rings nothing is an unrecorded error. Mine sat in a database for three and a half hours with the right diagnosis in it.
• Unattended upgrades restart your services at the time of their choosing. That is fine, and much better than not upgrading, but it means every service must be able to come back on its own.

---

Originally published at neuragrowth.co. I run a one-person digital-products studio and write up what breaks in production.

If you write CLAUDE.md files, I keep a set of working templates here: neuragrowth.co/free/claude-md-templates.