I got called into a financial services client running 40 Linux servers across VMware and AWS — customer-facing APIs, payment processing, internal microservices, all under strict uptime requirements.
Their Ansible deployments were failing. Randomly.
Not every time. Not on the same hosts. Some nights the maintenance window closed clean. Other nights, a chunk of the fleet came back UNREACHABLE mid-run — and the ones that already succeeded stayed changed, while the failed ones didn't. That's the part that should scare you: it doesn't fail loud, it fails partial.
The result: configuration drift across a production banking cluster. Different app versions on load-balanced nodes. Monitoring agents updated on some boxes, not others. Firewall rules inconsistent within the same tier. Ops had to freeze all further deployments until we could explain what was happening.
Root cause, once we dug into the logs:
The playbook ran gather_facts: yes on every execution, pulling full system facts from all 40 hosts before touching a single task. Combined with forks = 100 and a 10-second SSH timeout in ansible.cfg, the control node was pushing far more concurrent SSH sessions than it — or the bastion host — could actually sustain. Under any latency spike, healthy hosts got misclassified as unreachable before they'd even finished responding.
It wasn't a network problem. It was a capacity-planning problem wearing a network problem's clothes.
What fixed it:
→ Scoped fact-gathering to only what each task actually needed
→ Raised SSH connection timeouts to match real production latency
→ Tuned forks down to what the control node and bastion could actually handle
→ Moved from "deploy to all 40 at once" to controlled rolling batches
→ Added retry logic for transient SSH failures before marking a host failed
→ Set a failure threshold that halts the run instead of letting drift compound
Result: deployment success went from 82% to over 99.5%. No more manual remediation after every push. Maintenance windows became boring — which, in banking infra, is exactly what you want.
The lesson that stuck with me: parallelism isn't free speed. Every fork you add is a promise your control node, your bastion, and your network all have to keep at the same time. Break that promise once, and you don't get a clean failure — you get a fleet that's half-updated and doesn't know it.
Top comments (0)