In September we retired two of the six application servers behind our internal API, and the next morning the warehouse scanners at one site started failing about a third of their requests with 502. Part of our nginx tier was still sending traffic to both retired servers. The upstream list in its configuration had not mentioned them since the fifth of August.
The nginx tier is forty virtual machines configured by an Ansible playbook. The task that renders nginx.conf from a template notifies a handler that reloads nginx. Handlers run at the end of the play, once, on each host where a notifying task reported changed. On the fifth of August the template changed on all forty. Later in the same play, a task installing a monitoring agent failed on eleven hosts because our package mirror timed out. By default a host that fails a task drops out of the rest of the play, and the handlers it had notified never run. Those eleven had the new file on disk and the old configuration in memory.
We fixed the mirror and ran the playbook again, and it was green everywhere. On the second run the template matched what was already on disk, so the task reported ok rather than changed, notified nothing, and the reload never happened. Ansible knows a file needs a reload only during the run that changed it. When that run ends, nothing remembers the debt.
For seven weeks those hosts served the configuration of July. The difference did not matter until the backends it listed stopped existing.
force_handlers is now true in our ansible.cfg, which makes notified handlers run on a host even after a later task fails there. Each role that changes a service's configuration calls meta: flush_handlers straight after its config tasks, so the reload happens before anything unrelated has a chance to fail. And the template now writes its own version into a response header on a local status page. The last task of every play requests that page on each host and fails if the header does not match the version it just rendered, which reads what nginx is actually running rather than what is on disk. Its first run found two more stale hosts in a tier we had not touched in months.
A green run told us every file was right. It said nothing about whether the processes reading them had heard.
– Sergey Shinder
Top comments (1)
The debt that only exists during the changing run is the scary part — eleven hosts got the new file, never reloaded, then a green second run saw "ok" and notified nothing, so they kept serving July upstreams until backends disappeared. force_handlers plus flush_handlers, then asserting a version header from the live process instead of the file on disk, is exactly the right check.
Peer note: I work on CDNCA, an open-source CDN control plane. CDN nodes and a plugin marketplace sit on one control plane, and marketplace features roll out release by release. Fleet config drift of "written but not applied" is the same failure mode we design around: a successful push means nothing until every node reports the version it is actually running.
Do you expose that config version only on a local status page, or also on a response header you can scrape from the load balancer path?