What held up across 28 hosts, and what absolutely did not.
02:13.
That was when the last Debian 11 VM in our payments group stopped accepting the old OpenSSH cipher list, halfway through an Ansible run that had looked boring for 41 minutes. Boring is usually a compliment in infrastructure work. At 02:13, it became a theory.
We moved 28 application and utility hosts to Debian 12 over three evenings: 14 API nodes, six Celery workers, four reporting boxes, two Redis replicas, and two machines nobody could name without checking NetBox. We ran ansible-core 2.17.7 from a pinned Python 3.12 virtual environment on our control host. The exercise produced a familiar set of opinions from people who have used Ansible once, inherited it reluctantly, or seen a YAML file commit a minor crime.
Most of those opinions contain a grain of truth. Grains are how people end up eating sand.
“Ansible Is Just SSH With YAML”
Verdict: false, though SSH is still where many bad evenings begin.
For ordinary Linux hosts, Ansible’s default transport is SSH, and that fact encourages people to dismiss the rest as a collection of remote shell commands with more indentation. We did use SSH. We also used inventory groups, facts, idempotent modules, handlers, privilege escalation rules, check mode, variable precedence, and a run history we could inspect after someone asked why cron had restarted on a reporting node at 01:47.
Calling all of that “just SSH” is like calling PostgreSQL “just files on disk.” Technically defensible, professionally unhelpful.
The distinction mattered during the cutover because our hosts were not alike. The API nodes run gunicorn behind Nginx. Celery workers have different systemd limits. The reporting boxes mount an NFS share that the application hosts never see. A shell loop would have needed its own logic for each difference, along with error handling and a way to tell whether a retry had changed anything. We have written those loops before. They age like yoghurt in a server room.
Here is the small part of the play that put the right APT source on each Debian 12 host and notified Nginx only when its file changed:
- name: Configure internal package source
ansible.builtin.deb822_repository:
name: oasis-internal
types: [deb]
uris: "https://apt.devopsoasis.internal/debian"
suites: ["{{ ansible_distribution_release }}"]
components: [main]
signed_by: "https://apt.devopsoasis.internal/keys/archive.gpg"
state: present
notify: Reload nginx
- name: Install application packages
ansible.builtin.apt:
name:
- oasis-api={{ api_package_version }}
- nginx
state: present
update_cache: true
handlers:
- name: Reload nginx
ansible.builtin.service:
name: nginx
state: reloaded
The Debian repository module and the APT module gave us useful state instead of an optimistic curl | tee sequence. We could rerun the play after fixing the cipher issue and see 23 hosts report ok, four make the expected package changes, and one fail on a stale internal repository key.
That said, Ansible does not remove SSH’s sharp edges. A stale known_hosts entry, a host key rotation, an expired jump-host certificate, or a control socket left under /tmp can still stop a run before a task begins. We had all four categories represented in the preceding month, because apparently our infrastructure enjoys variety.
Ansible gives SSH a memory and a grammar. It does not grant SSH better manners.
“If A Playbook Is Idempotent, Rerunning It Is Safe”
Verdict: dangerously false.
Idempotence means a task should reach the same desired state when applied repeatedly. It does not mean every action surrounding that task is harmless. A package install can be idempotent while the package’s post-install script restarts a service. A template can be idempotent while its handler drains a connection pool at a bad moment. A database migration can be “already applied” and still leave a node with an incompatible binary if deployment order goes wrong.
We had a clean example on the Celery workers. The role installed oasis-worker, rendered a systemd unit, and restarted the service when either changed. On Debian 11, the worker used a unit file with Restart=always. On Debian 12, we changed it to Restart=on-failure after discovering that a bad RabbitMQ credential caused a very energetic restart loop and 9 GB of logs over an on-call shift.
The first run changed the unit file. Good. The second run should have reported no changes. Instead, it restarted all six workers because a task used state: restarted as a convenient substitute for thinking.
We replaced it with this:
- name: Install worker systemd unit
ansible.builtin.template:
src: oasis-worker.service.j2
dest: /etc/systemd/system/oasis-worker.service
owner: root
group: root
mode: "0644"
notify:
- Reload systemd
- Restart oasis worker
- name: Ensure worker is enabled and running
ansible.builtin.systemd_service:
name: oasis-worker
enabled: true
state: started
handlers:
- name: Reload systemd
ansible.builtin.systemd_service:
daemon_reload: true
- name: Restart oasis worker
ansible.builtin.systemd_service:
name: oasis-worker
state: restarted
The handler behaviour documented by Ansible is useful here: notifications coalesce, so six changed tasks still cause one restart at the end of a play. That only helps if we notify on real changes and avoid casually restarting a process because the playbook happened to visit it.
There is a second trap: changed_when: false. People add it to quiet noisy shell tasks, then forget that they have hidden the only signal a downstream handler had. We found one role doing exactly that around update-ca-certificates. It was marked unchanged even when it had imported a new internal CA. The application unit therefore kept running with an old trust store until its next scheduled restart.
Silence in Ansible output is not proof that nothing happened.
We now treat reruns differently by class of work. Package and file convergence can run repeatedly. Service reloads need a reason. Schema changes get a separate deployment step and a named human watching the output. We still have an argument going about whether certificate rotation belongs in the same play as application rollout. Marta, who carries the Thursday primary rota, wants it split permanently. We have not settled it, mostly because nobody enjoys owning the extra release button.
“Roles Make Every Ansible Repository Easier To Maintain”
Verdict: false after the third layer of includes.
Roles are useful. We keep roles for things with a real lifecycle: oasis_nginx, oasis_api, oasis_worker, node_exporter, and vector_agent. Each has defaults, templates, handlers, and tests. That structure lets a new engineer find the file that owns /etc/nginx/sites-enabled/oasis-api.conf without hiring a tracking dog.
Then roles become a way to hide choices.
Our oldest repository had a common role that installed base packages, set SSH settings, configured journald, added users, installed monitoring, patched APT sources, mounted volumes, and once, for reasons lost to history, configured mailutils. Every host received it. Its defaults file had 67 variables. The role’s README said “foundation configuration,” which meant nobody knew whether changing it would restart half the fleet.
We removed common during the Debian 12 work. Not renamed it. Removed it.
The replacement is less elegant to people who enjoy directory trees. The base play now names what it does:
roles:
- role: base_packages
- role: ssh_hardening
- role: journald
- role: node_exporter
- role: vector_agent
That is more repetitive, and we prefer it. A deployment plan should make its dependencies visible. When ssh_hardening changes, we can run it against the staging bastion first rather than wonder what else common believes it owns this week.
We also stopped putting application-specific variables in group_vars/all.yml. That file had become the cupboard under the stairs: Redis memory limits beside deploy keys, Nginx rate settings beside a legacy hostname, each one apparently too small to deserve a proper home. Variable precedence in Ansible is powerful enough to support almost any arrangement, which is a polite way of saying it will let us build a difficult mess. The precedence rules are clear; the harder part is declining clever arrangements before they spread.
Roles help when they represent a bounded unit of ownership. A role that changes SSH, packages, users, metrics, and mail is a landfill with a defaults/main.yml.
“Check Mode Tells Us What Production Will Do”
Verdict: partly true, and we still run it on every merge.
ansible-playbook --check --diff catches a surprising amount. It spots malformed templates, missing variables, unexpected file diffs, and package changes that would otherwise appear during a production window. For the Debian 12 migration, check mode caught an Nginx template referring to ssl_protocols TLSv1.2 TLSv1.3; on a host group where TLS terminated at the load balancer. The task was valid. The configuration was unnecessary. That distinction saved us a small amount of clutter, which is the usual reward for doing review properly.
It cannot predict everything.
Modules vary in their check-mode support. Commands and scripts cannot know their impact unless we tell Ansible with creates, removes, or explicit changed_when logic. External systems are worse. Our deployment task calls an internal release API to register the package version. In check mode, we skip it. Running a fake registration against production would turn a dry run into an incident rehearsal.
Package managers also have opinions. APT can calculate planned installs, but the final answer depends on repository contents, dependency resolution, held packages, and whether the mirror has finished syncing. On the second cutover night, our internal mirror had libpq5 15.8 for eleven minutes before its metadata caught up. Check mode at 22:06 looked fine; the real run at 22:17 installed nothing until we forced an index refresh.
This only held because our package mirror is local and the sync lag is usually under 15 minutes. We have not tried this process across 40 nodes or across regions with separate mirrors. The current serial batch size of two works because someone can read the output before the next pair begins, not because two is a mystical number.
Check mode is a review aid. Treating it as a production simulation invites a very confident failure.
“Ansible Is Too Slow For Real Deployments”
Verdict: mostly true, if “real deployments” means every host deserves parallel attention.
Ansible is not slow in the way people mean when they complain about it. A play spends time connecting, gathering facts, copying files, waiting for package locks, and asking remote services whether they already match the requested state. That work is visible. A bash fan-out script hides the same waiting behind interleaved output and gives it a name like deploy-final-v4.sh.
For 28 hosts, our full Debian 12 convergence play took 18 minutes and 36 seconds with forks = 10. Fact gathering alone consumed just under four minutes on the first run. We could turn it off, and we did for the application-only deployment play. We did not turn it off for the OS migration because several templates depend on distribution release and interface facts, and discovering that late is tedious.
The larger problem is that Ansible’s default execution model makes safe rolling work feel deliberate. We set serial: 2 on API nodes, wait for each host to return to the load balancer, and only then move on. That is slower than replacing all 14 at once. We prefer slower. The incident report from 2023, when a bad Nginx include removed health-check responses across an entire pool, remains available to anyone tempted by speed.
We do use the free strategy for low-risk work such as updating node exporter and Vector. It prevents a slow reporting host from making every other machine wait at the same task boundary. We do not use it for application releases, where matching order across a small batch is easier to observe and roll back.
Ansible has a ceiling. We would not choose it to coordinate thousands of ephemeral containers, and we dislike pretending a tool should solve jobs it was not built for. For a few dozen long-lived servers with awkward local differences, the minutes it spends checking state are cheaper than the minutes we spend explaining a broad outage.
Top comments (0)