Our customer portal runs on sixteen web servers behind one load balancer. When a customer asks for an invoice, the server that handles the request returns a link signed with an HMAC key, and the download is verified by whichever server the load balancer picks next. Every server has to hold the same key. An Ansible playbook generates it and writes it to every server, and we rotate it every quarter.
The October rotation ran on a Tuesday evening and finished green. By Wednesday morning three invoice downloads in four were failing with invalid signature, and support had more than four hundred tickets before nine.
The task that generates the key is a command running openssl rand, marked run_once and registering its output. With run_once, Ansible runs the task on one host and gives its result to every other host it is working on. In July, after a deploy restarted all sixteen servers at once, we added serial: 4 to the same playbook, so it would work through the servers four at a time.
The Ansible documentation states the combination plainly. When a play uses serial, a run_once task runs once in each batch. So the rotation generated four keys, one per batch, and each group of four servers received its own. A link signed by one server was verified by the same group only when the load balancer happened to choose one of those four, which is one time in four.
Nobody had connected the two changes. The serial line was reviewed as a safety improvement to restarts, and it was one. The key task had not changed in two years, and the July deploy was not a rotation, so the first rotation after it was the first time the two met.
The key is now generated in a separate play on the control node with no serial, stored in our secrets manager, and every server reads it from there. Verification accepts the current key and the previous one for a day, so a rotation never breaks links that are already out. The playbook ends by fetching the key's fingerprint from all sixteen servers and fails if they differ.
run_once means once per batch, and our playbook had become batched for a reason that had nothing to do with keys.
– Sergey Shinder
Top comments (0)