Originally published on kuryzhev.cloud
A single ansible-playbook site.yml that restarts every web node at once turns a routine release into a short outage. An Ansible rolling deployment avoids that by updating hosts in small batches. Each host is taken out of the load balancer first and put back only after it proves healthy. The tips below are independent, so use whichever fits your setup.
The examples assume an inventory group web, an HAProxy host lb1, and a /healthz endpoint on each app node. Swap in your own load balancer module and health check as needed.
1. Start with one host, then widen the batches
Use a list for serial so the first batch acts as a canary.
serial accepts integers, percentages, or a list of both. A list such as [1, "25%", "100%"] updates one host, then a quarter of the fleet, then everything that remains. If the canary breaks, the blast radius is one node. Percentages are calculated against the hosts in the play, so they scale as the fleet grows.
- name: Rolling application release
hosts: web
serial: [1, "25%", "100%"] # canary, then a quarter, then the rest
max_fail_percentage: 0 # any failure in a batch stops the rollout
tasks:
- name: Deploy release
ansible.builtin.include_role:
name: app_release
Watch out for: max_fail_percentage is evaluated per batch, not across the whole play. With a one-host canary and a 30% threshold, a single failure is already 100% of that batch. Check how the threshold behaves at your smallest batch size.
2. Drain each node before touching it
Remove the host from the load balancer in pre_tasks and delegate the call to the balancer.
Restarting a service that is still receiving traffic drops in-flight requests, even when the batches are small. Disable the backend server first and wait for connections to finish. The delegate_to keyword runs the task on the balancer while still using the current app host's variables.
pre_tasks:
- name: Drain node from HAProxy
community.general.haproxy:
backend: app_backend
host: "{{ inventory_hostname }}" # must match the server name in haproxy.cfg
state: disabled
drain: true # let existing sessions finish
wait: true
wait_retries: 30
wait_interval: 2
delegate_to: lb1
The module talks to HAProxy through its runtime stats socket. That socket must be configured with level admin in haproxy.cfg, and its path set with the socket option if it differs from the default /var/run/haproxy.sock.
Check the module options with ansible-doc community.general.haproxy, since parameters can differ between collection versions. On AWS, the equivalent is deregistering the target from the target group and waiting for the deregistration delay to pass.
3. Gate re-entry on a real health check
A service that is "started" is not a service that is ready.
For Type=simple or Type=forking units, systemd reports the start as successful as soon as the process is launched. It does not wait for the application to be ready to serve. Only Type=notify units that send an explicit readiness signal change that. Even then, the app may still be warming caches or waiting for its database pool.
Poll the health endpoint with until, and re-enable the node in post_tasks only after it returns 200.
post_tasks:
- name: Wait for application health
ansible.builtin.uri:
url: "http://{{ inventory_hostname }}:8080/healthz"
status_code: 200
register: health
until: health.status == 200
retries: 20
delay: 3 # up to ~60 seconds before the host is marked failed
- name: Return node to rotation
community.general.haproxy:
backend: app_backend
host: "{{ inventory_hostname }}"
state: enabled
delegate_to: lb1
Make the health endpoint check real dependencies, not just return a static "ok". Otherwise this gate only proves the web server process is running.
4. Flush handlers before the health check
Restart handlers run late by default, so force them to run inside the batch.
Handlers run at the end of the current section (pre_tasks, tasks, post_tasks), not right after the task that notifies them. If your restart handler is notified in tasks, it fires after tasks finishes, which is before post_tasks. That ordering is usually fine. It breaks as soon as someone moves the health check into tasks. Make the order explicit instead of relying on it.
- name: Render app config
ansible.builtin.template:
src: app.conf.j2
dest: /etc/app/app.conf
notify: Restart app
- name: Apply pending restarts now
ansible.builtin.meta: flush_handlers # restart happens here, not later
Watch out for: if a task fails before the handler runs, the notified handler is skipped for that host. Set force_handlers: true (or pass --force-handlers) to change that. A node with changed files but no restart is a typical source of "the playbook failed but the files changed" confusion.
5. Decide what a failed host looks like
Know whether you want a broken node left drained or put back.
When a host fails, Ansible drops it from the rest of the play. In an Ansible rolling deployment, that means it stays disabled in the load balancer, because the re-enable task never ran. That is usually the safe outcome, since a bad node does not get traffic. Still, make it deliberate and visible, because a silently drained node reduces capacity until someone notices.
Use block/rescue if you want a rollback step, such as reinstalling the previous artifact. Be aware that a host whose rescue succeeds is not counted as failed. It continues into post_tasks, does not count toward max_fail_percentage, and will be returned to rotation if the health check passes. If you want a rolled-back node to stay drained, or the rollout to stop, end the rescue with ansible.builtin.fail.
Use any_errors_fatal: true if one failure should abort the whole rollout. The current task finishes on the batch, then the play ends for all hosts, and no further batches run.
Pair either choice with an alert or a final summary task. That way a reduced pool is reported, not discovered later.
Also confirm that the remaining healthy nodes can carry the load. With serial: "50%", half the fleet is out of rotation during every batch. The percentage should reflect your headroom, not just your patience.
6. Keep migrations out of the serial play
Run schema changes once, in a separate play, and keep them backward compatible.
The documented behavior of run_once combined with serial is that the task runs once per batch, not once per play. A migration marked run_once: true inside a serial play will execute again for every batch. Put it in its own play ahead of the rolling one.
- name: Database migration (once)
hosts: web[0] # a single host runs the migration
tasks:
- name: Run additive migrations
ansible.builtin.command: /opt/app/bin/migrate --additive-only # placeholder for your migration tool
changed_when: true # the command gives no change signal; verify with your tool
- name: Rolling application release
hosts: web
serial: [1, "25%", "100%"]
# ... pre_tasks, tasks, post_tasks from the tips above
During a rollout, old and new code run side by side against the same database. Use the expand/contract approach: add columns and tables first, deploy the code, and drop old structures in a later release. Without that, even perfect batching causes errors on the nodes still running the previous version.
For serial, max_fail_percentage, and run_once, see the official Ansible documentation on playbook strategies and rolling batches. For delegate_to, see the official Ansible documentation on controlling where tasks run. More automation and CI/CD notes are on kuryzhev.cloud.
Top comments (0)