Ansible Zero-Downtime Deployment: Draining the Load Balancer Right

Originally published on kuryzhev.cloud

---

The scenario

We had a Flask API running on 8 app servers behind HAProxy, deployed with a straightforward Ansible playbook. It worked fine for a year — until traffic grew and someone finally noticed the pattern: every deploy, without exception, produced a 20-40 second window of 502s. Nobody had flagged it because deploys were scheduled at 3am specifically to dodge the pain, which tells you everything about how "acceptable" the workaround had become.

The root cause was embarrassingly simple. The playbook used serial: 100%, meaning Ansible hit every host at once, restarted every service at once, and HAProxy's health checks failed simultaneously across the entire backend pool. There was no ansible zero-downtime deployment strategy in place — just a full-fleet restart dressed up as automation. Users hit dead backends for the exact duration it took systemd to bring the process back up and pass its first health check.

The fix isn't exotic. It's serial plus max_fail_percentage plus explicit load balancer deregistration, wired together so that a node stops receiving traffic before it gets touched, and only rejoins the pool once it's actually ready — not just "port open." That last distinction matters more than people expect, and I'll get to why.

Prerequisites

Before touching the playbook, make sure the environment actually supports this pattern:

• Ansible core 2.15+ — serial and max_fail_percentage are play-level keywords, not task-level, and older versions have inconsistent batch-rounding behavior.
• An inventory group (webservers) with SSH access and become privileges configured.
• A load balancer that exposes some form of programmatic control. We used HAProxy's runtime socket (/var/run/haproxy/admin.sock), but if you're on AWS, the amazon.aws.elb_target module (collection amazon.aws >= 5.0.0) does the equivalent against an ALB target group.
• An app-level /healthz endpoint that returns 200 only when the app is actually warmed up — database pool initialized, cache connected — not just "process is alive."

If any of these are missing, fix that first. Building rolling deploy logic on top of a health check that lies is worse than not having one, because it gives you false confidence right before it breaks in production.

Step 1 — Structure the playbook with serial and batching

The skeleton starts at the play level, before any load balancer logic gets involved. serial: "25%" tells Ansible to work through the inventory in batches, and it rounds up — on an 8-host inventory that's batches of 2, but on a 7-host inventory, 25% rounds to batches of 2 as well since Ansible ceils the fraction rather than truncating it. Don't assume the math lines up neatly; check it against your actual host count.

max_fail_percentage: 0 combined with any_errors_fatal: true gives the strictest possible behavior: if a single host in a batch fails, the entire run stops immediately, including batches that haven't started yet. That's intentional. A partial bad rollout across half your fleet is a much worse night than a deploy that stops early.

order: sorted matters more than it looks — default inventory order can shift between runs if you're using a dynamic inventory script, which makes batch membership non-deterministic. That's a debugging nightmare when you're trying to reproduce a failure that only happened on "whichever three hosts landed in batch two."

Step 2 — Deregister from the load balancer before touching the host

This is the step naive rolling deploys skip, and it's the number one mistake I see. Restarting the service before pulling the node out of rotation means HAProxy is still routing live requests to a process that's mid-restart. You don't get a clean 502 — you get dropped connections mid-request, which is worse for anything transactional.

The fix is a pre_tasks block that disables the backend server first, then pauses long enough for in-flight connections to drain before anything else happens.

pre_tasks:
  - name: Deregister node from HAProxy backend
    ansible.builtin.shell: |
      echo "disable server backend_app/{{ inventory_hostname }}" | \
      socat stdio /var/run/haproxy/admin.sock
    changed_when: true

  - name: Wait for in-flight connections to drain
    ansible.builtin.pause:
      seconds: "{{ drain_wait }}"

The drain wait should roughly match your load balancer's connection draining timeout. AWS ALB defaults to a 300-second deregistration delay, which is absurd for a rolling deploy — most teams tune it down to 15-30 seconds. Whatever value you land on, keep the Ansible pause in sync with it, or you'll drain traffic on paper while the LB config still thinks it needs five minutes.

Step 3 — Deploy, restart, and health-check before re-registering

Once the node is out of rotation, deploy the artifact, restart the service, and — critically — wait on a real readiness check before doing anything else. This is where the second big mistake shows up: checking wait_for: port=8080 and calling it done. A port being open tells you the process bound to it. It tells you nothing about whether the DB connection pool finished initializing or the JIT warmed up. I've watched a "successful" deploy re-register a node into the pool and immediately start throwing 500s for the next ten seconds, because the health check was checking the wrong thing.

tasks:
  - name: Deploy latest release artifact
    ansible.builtin.unarchive:
      src: "/opt/releases/app-{{ app_version }}.tar.gz"
      dest: "/opt/app/current"
      remote_src: false

  - name: Restart app service
    ansible.builtin.systemd:
      name: myapp
      state: restarted
      daemon_reload: true

  - name: Wait for app to report healthy
    ansible.builtin.uri:
      url: "{{ healthcheck_url }}"
      status_code: 200
      return_content: true
    register: health
    until: >
      health.status == 200 and
      (health.content | from_json).status == "ok"
    retries: 10
    delay: 3

The /healthz response should include the dependencies that actually matter — something like {"status":"ok","db":"connected","cache":"connected"} — not just process liveness. If you're running DB migrations as part of the deploy, keep them in a separate play with run_once: true, not inline in this one. If the first host in the first batch runs migrations and then fails before the app deploy step completes, you're left with a schema change applied but no app deployed to match it — a nasty state to debug at 3am.

Step 4 — Re-register and move to next batch

Only after the health check passes does the node go back into rotation. Re-enable it on HAProxy, add a short soak period, then let Ansible move to the next batch.

post_tasks:
  - name: Re-register node with HAProxy backend
    ansible.builtin.shell: |
      echo "enable server backend_app/{{ inventory_hostname }}" | \
      socat stdio /var/run/haproxy/admin.sock
    changed_when: true

  - name: Soak period before next batch
    ansible.builtin.pause:
      seconds: 10

That soak period isn't decorative. It gives your monitoring a window to catch a regression on this one node before the next batch also goes down for deployment — cheap insurance against compounding a mistake across the whole fleet. If you need finer-grained concurrency control within a batch — say, only letting one host at a time run a heavy cache-warm task even though serial allows three — throttle: N on that specific task overrides the batch concurrency without changing your overall rollout shape.

Also worth flagging on the security side: whatever credentials Ansible uses to talk to the LB — HAProxy socket permissions or an AWS IAM role — should be scoped to target group registration only, not full ELB admin. Store them via Ansible Vault or an IAM role, never plaintext in inventory. It's an easy thing to overlook once the deploy pipeline is "working."

Verify and test

Don't take zero-downtime on faith — prove it. Run a continuous load test against the LB VIP for the duration of the deploy and check for non-2xx responses:

hey -z 90s -c 10 http://lb.internal/ | tee load_result.txt
grep -E "\[[45][0-9]{2}\]" load_result.txt && echo "FOUND ERRORS" || echo "CLEAN RUN"

In parallel, watch the HAProxy stats socket to confirm nodes actually transition to MAINT and back to UP rather than dropping out silently:

watch -n1 'echo "show stat" | socat stdio /var/run/haproxy/admin.sock | \
  awk -F"," "{print \$2, \$18}"'

# Expected during a healthy rolling deploy:
# node1  UP
# node2  MAINT   <- briefly, while deregistered
# node3  UP
# node2  UP      <- back after health check passes

Then SSH into the host being deployed and confirm it shut down gracefully instead of getting killed mid-request:

ssh node2 "journalctl -u myapp --since '2 min ago' | grep -i 'sigterm|shutting down'"

That last check ties back to your systemd unit — set TimeoutStopSec=30 and make sure the app actually handles SIGTERM by finishing in-flight requests, or you'll get hard kills even after LB deregistration if the app takes too long to drain on its own. Finally, before you ever point this at production, run ansible-playbook site.yml --limit webservers --check against staging, then a real run with -vv so you can see the exact HTTP/socket calls hitting the LB before they touch anything that matters. See the DevOps_DayS archive for more on load balancer health check patterns if you're running this against something other than HAProxy.

Closing

Zero-downtime isn't a flag you flip — it's a chain of three things that all have to hold at once: batched rollout via serial, explicit load balancer state changes wrapping the actual deploy, and a health check that means "ready" instead of just "alive." Drop any one of those and you silently reintroduce the exact outage window you thought you'd fixed, and it usually won't show up until someone's staring at 502 logs wondering why the "zero-downtime deploy" isn't. Once this pattern is solid on a single playbook, the natural next step is wiring it into CI — GitHub Actions or GitLab pipelines with a manual approval gate between batches — so a human can eyeball metrics before the rollout continues past the first few hosts. For more on the underlying HAProxy and load balancer mechanics, the HAProxy documentation and the Ansible strategies guide are worth keeping bookmarked.

Related

• More Ansible playbook patterns and bootstrap troubleshooting
• CI/CD pipeline design and approval gate patterns
• Kubernetes rolling update and rollout strategies for comparison