One Dead Container Stopped nginx From Starting At All

Eight containers behind one nginx. One of them was stopped. nginx would not start.

Not "returned 502 for that service" — would not start, at all, with every other service running fine:

host not found in upstream "importer"

That is one dead background service taking down the entire public entry point: the frontend, the API, the tile server, all of it. This is a short post about why, the three-line fix, and the two things the fix breaks that nobody mentions.

Why it happens

The usual framing is "nginx caches DNS". That's true but it undersells it. With a literal hostname:

location /importer/ {
    proxy_pass http://importer:8001/;
}

nginx resolves importer once, while parsing the config, and bakes the address into the running configuration. Two consequences follow, and the second is the dangerous one:

1. If the container restarts and Docker gives it a new IP, nginx keeps sending traffic to the old one until you reload.
2. If the name doesn't resolve at parse time, the config is invalid. nginx exits.

Point 2 makes the blast radius total. Every upstream becomes a startup dependency of every route. Restart your reverse proxy at the wrong moment — during a deploy, after a host reboot when containers come up in an unlucky order — and it will refuse to come back because of a service nothing on the critical path needs.

The fix

Put the address in a variable and give nginx a resolver:

resolver 127.0.0.11 valid=10s ipv6=off;
resolver_timeout 5s;

set $svc_importer http://importer:8001;
set $svc_api      http://api:8000;
set $svc_tiles    http://tiles:3000;

location /importer/ {
    proxy_pass $svc_importer;
}

When proxy_pass contains a variable, nginx defers resolution to request time. Now a missing container is a 502 on its own route, which is what you wanted all along. Everything else keeps serving.

Two details in that resolver line:

127.0.0.11 is Docker's embedded DNS server, present inside every container on a user-defined network. On another platform it's whatever your service discovery exposes.

ipv6=off is not optional decoration. Without it nginx asks for AAAA records too, and if it gets one for a service listening only on IPv4, the connection fails in a way that looks nothing like a DNS problem. Turn it off unless your services actually speak IPv6.

valid=10s caps how long a resolved address is reused. Short enough that a container restart heals on its own; long enough that you're not doing a DNS lookup per request.

What the fix breaks, part one: the trailing slash

This is where it stops being a three-line change.

These two are not the same directive:

proxy_pass http://tiles:3000/;   # note the trailing slash
proxy_pass $svc_tiles;           # variable

With a literal URI and a trailing slash, nginx strips the matched location prefix before forwarding. /tiles/catalog arrives upstream as /catalog.

With a variable, nginx cannot do that — it doesn't know at parse time what the URI part is — so it forwards the full original path. /tiles/catalog arrives upstream as /tiles/catalog, and your tile server returns 404 for everything.

You have to do the stripping yourself:

location /tiles/ {
    rewrite ^/tiles/(.*)$ /$1 break;
    proxy_pass $svc_tiles;
}

Nothing warns you. The config is valid, nginx starts, and the route 404s.

What the fix breaks, part two: if below rewrite

And now the second-order one, which cost me considerably more time.

CORS preflight is normally handled with a short-circuit:

location /tiles/ {
    rewrite ^/tiles/(.*)$ /$1 break;

    if ($request_method = 'OPTIONS') {
        add_header 'Access-Control-Allow-Origin' '*';
        add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
        return 204;
    }

    proxy_pass $svc_tiles;
}

That if never runs.

rewrite and if both belong to ngx_http_rewrite_module, which evaluates its directives in order — and the break flag stops that evaluation for the rest of the block. Every rewrite-module directive after it, if included, is skipped.

So the preflight falls through to proxy_pass, the browser gets whatever the upstream says about OPTIONS, and your CORS headers never appear. The failure shows up in a browser console as a CORS error, which sends you looking at add_header and origins — the two things that were correct all along.

The fix is ordering, not content:

location /tiles/ {
    # OPTIONS must come ABOVE the rewrite: `rewrite ... break` halts
    # ngx_http_rewrite_module, and `if` is a directive of that same module.
    if ($request_method = 'OPTIONS') {
        add_header 'Access-Control-Allow-Origin' '*';
        return 204;
    }

    rewrite ^/tiles/(.*)$ /$1 break;
    proxy_pass $svc_tiles;
}

Note that this ordering trap only appears because of the earlier fix. With a literal proxy_pass and a trailing slash, there is no rewrite, so there is nothing to halt the module and the if works wherever you put it. Fixing the startup dependency introduced the rewrite; the rewrite introduced the ordering constraint.

Worth knowing

Test your reverse proxy with a service stopped. Not a service returning errors — a service not running. That's the state a host reboot produces, and it's the one that turns a single failure into a total one. It takes thirty seconds to check and it's the only way to find this before it finds you.

A variable in proxy_pass changes URI handling, not just resolution timing. If you're converting an existing config, every proxy_pass that ended in a slash needs a rewrite to replace it.

In nginx, module ordering beats block ordering. Directives from the same module run as a sequence, and break ends that sequence. if, rewrite, return and set all belong to the rewrite module, so their relative order matters in a way that add_header and proxy_set_header don't.

Three lines to fix. Two more to keep it working.