Fix Ansible Bootstrap Playbook Failing on Fresh Cloud Servers

#devops

Originally published on kuryzhev.cloud

Your Ansible bootstrap playbook fails on the first run against a freshly created instance, then works perfectly when you re-run it thirty seconds later. If that sounds familiar, you're dealing with an ansible bootstrap playbook failing intermittently — and it's almost never the network. I've hit this exact pattern on AWS EC2, Hetzner, and DigitalOcean droplets, and the fix is the same every time: stop treating "instance running" as "instance ready."

Symptoms

Three failure signatures show up, usually in combination, and they're maddening because they're non-deterministic. First: UNREACHABLE! => Failed to connect to the host via ssh on a brand-new instance, gone entirely on the second attempt. The exact error is:

fatal: [10.0.1.42]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.0.1.42 port 22: Connection refused",
    "unreachable": true
}

That "Connection refused" — not "timed out" — is the tell. It means the port is open at the network layer but sshd hasn't bound to it yet. Second symptom: become: true tasks blow up with {"msg": "Missing sudo password"} even though you can SSH in manually and run sudo whoami without a prompt. Third: the playbook sits at "GATHERING FACTS" for over a minute and then times out, or the very next apt task dies with E: Could not get lock /var/lib/dpkg/lock-frontend.

Individually these look like three unrelated bugs. Together, they're one problem: the control node is racing cloud-init.

Root cause

Cloud-init hasn't finished configuring the SSH daemon and user accounts by the time Ansible tries its first connection. Your instance is "running" in the API's eyes long before it's actually usable — this is a timing race, full stop, not a firewall or security group misconfiguration.

On top of that, ansible_python_interpreter auto-discovery picks the wrong binary on distros that still ship a python2 symlink alongside python3, which breaks modules silently instead of failing loudly. Meanwhile, nobody provisioned a NOPASSWD sudoers entry for the bootstrap user, so become fails unless ansible_become_pass is set — and it usually isn't, because whoever wrote the base image assumed interactive sudo was fine.

Finally, the cloud provider's own unattended-upgrades process or cloud-init's initial apt run grabs the dpkg lock at /var/lib/dpkg/lock-frontend exactly when your playbook's apt: update_cache=yes fires. Two processes, one lock, one loser.

Fix #1 — Wait for SSH and cloud-init readiness before running tasks

Don't gather facts until the host proves it's ready. Use wait_for_connection as the literal first task in the play — it ships in ansible.builtin since Ansible 2.3, no extra collection install needed.

- name: Wait for SSH to become available
  ansible.builtin.wait_for_connection:
    delay: 10       # give cloud-init a head start before probing
    timeout: 300     # cold boots on some providers take 2-4 minutes
    sleep: 5

Then check for the actual cloud-init completion marker instead of guessing with a sleep():

- name: Wait for cloud-init to finish
  ansible.builtin.wait_for:
    path: /var/lib/cloud/instance/boot-finished
    timeout: 180

Gotcha: if you add retry logic to the SSH connection plugin but leave gather_facts: true on the play, Ansible tries to gather facts before your readiness tasks even run — the whole play times out before your retries kick in. Set gather_facts: false at the play level and gather facts explicitly, after readiness checks pass. Also bump ansible.cfg's defaults — the stock [ssh_connection] retries = 3, timeout = 10 is tuned for warm hosts, not cold-boot cloud instances.

Fix #2 — Fix become/sudo failures with proper privilege escalation

The cleanest fix happens before Ansible ever connects: bake a sudoers file into cloud-init user-data at instance creation time.

# cloud-init user-data snippet
runcmd:
  - echo "ansible ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ansible-bootstrap
  - chmod 440 /etc/sudoers.d/ansible-bootstrap

Verify it's actually working before you trust it:

ansible -m command -a "sudo -n true" host
# exit code 0 = passwordless sudo works
# exit code 1 = misconfigured, fix before continuing

In the playbook, fail fast with a clear message instead of letting every subsequent task die with a cryptic error:

- name: Verify passwordless sudo works
  ansible.builtin.command: sudo -n true
  register: sudo_check
  changed_when: false
  failed_when: false

- name: Fail with clear message if sudo not configured
  ansible.builtin.fail:
    msg: "NOPASSWD sudo not configured for {{ ansible_user }} — fix cloud-init user-data"
  when: sudo_check.rc != 0

Watch out: if password-based sudo is mandatory in your environment, store ansible_become_pass with ansible-vault encrypt_string. I've seen more than one repo with a plaintext sudo password sitting in group_vars/all.yml, committed and pushed. That's a five-minute fix that prevents a very bad day. Also — scope that NOPASSWD entry to the bootstrap user only, and rotate or remove it once initial provisioning is done. Leaving broad NOPASSWD access around is a privilege-escalation risk if that account is ever compromised later.

Fix #3 — Pin the Python interpreter and handle apt lock contention

Stop relying on auto discovery. Pin it explicitly in group_vars, especially on Debian 11/12 and Ubuntu 22.04/24.04 images where the discovery logic has burned people before:

ansible_python_interpreter: /usr/bin/python3

ansible_python_interpreter: auto_silent just hides the warning — it doesn't fix a wrong-binary selection. If you're not sure what's happening, run ansible-playbook -vvv and check the exact interpreter path in the SSH command output. Fastest way to confirm a mismatch.

For the dpkg lock, the cleanest fix is the native lock_timeout parameter on the apt module (Ansible 2.10+), combined with a retry loop:

- name: Update apt cache with retry
  ansible.builtin.apt:
    update_cache: true
    cache_valid_time: 3600
    lock_timeout: 60      # native lock handling, no manual fuser loop needed
  become: true
  register: apt_result
  retries: 5
  delay: 10
  until: apt_result is succeeded

If you're on an older Ansible core without lock_timeout, fall back to a manual wait — check both lock paths since dpkg versions differ on which one they use:

- name: Wait for dpkg lock to release
  ansible.builtin.shell: |
    while fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 1; done
  changed_when: false
  become: true

Here's the full assembled playbook, tested against Ansible core 2.16.3 with a Python 3.11 control node, targeting Ubuntu 22.04 and Debian 12:

# playbook: bootstrap.yml
---
- name: Bootstrap fresh cloud servers
  hosts: new_servers
  gather_facts: false
  become: false

  vars:
    ansible_python_interpreter: /usr/bin/python3

  tasks:
    - name: Wait for SSH to become available
      ansible.builtin.wait_for_connection:
        delay: 10
        timeout: 300
        sleep: 5

    - name: Wait for cloud-init to finish
      ansible.builtin.wait_for:
        path: /var/lib/cloud/instance/boot-finished
        timeout: 180

    - name: Now gather facts safely
      ansible.builtin.setup:

    - name: Verify passwordless sudo works
      ansible.builtin.command: sudo -n true
      register: sudo_check
      changed_when: false
      failed_when: false

    - name: Fail with clear message if sudo not configured
      ansible.builtin.fail:
        msg: "NOPASSWD sudo not configured for {{ ansible_user }} — fix cloud-init user-data"
      when: sudo_check.rc != 0

    - name: Wait for dpkg lock to release
      ansible.builtin.shell: |
        while fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 1; done
      changed_when: false
      become: true

    - name: Update apt cache with retry
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600
        lock_timeout: 60
      become: true
      register: apt_result
      retries: 5
      delay: 10
      until: apt_result is succeeded

And the matching ansible.cfg — this is what actually makes the fixes above stable across a fleet instead of one lucky instance:

# ansible.cfg — tuned for cold-boot cloud instances
[defaults]
timeout = 30
host_key_checking = False
retry_files_enabled = False

[ssh_connection]
retries = 5
timeout = 30
pipelining = True

Note the fact-gathering timeout lives under [defaults] timeout, not some separate fact-caching setting. I've watched engineers dig through cache config for twenty minutes when the real knob was sitting right there.

Prevention

The real fix isn't in the playbook at all — it's upstream, in the base image or user-data script. Bake readiness (SSH config, sudoers, package manager state) into the image so Ansible never has to fight a race condition in the first place. That's the difference between patching symptoms and removing the root cause.

Practical checklist for next time:

Run the bootstrap playbook through Molecule with a Docker driver (molecule-plugins[docker] v23.x) in CI before it ever touches a real cloud API.
Pin ansible-core in your requirements file and lock collection versions in requirements.yml — interpreter and module behavior drifts silently between minor releases.
Log bootstrap run duration and failure type per host. If cloud-init suddenly takes 90 seconds longer after an image update, you want a graph showing that, not a angry Slack message at 2am.
Watch out for unbounded wait_for_connection timeouts — 600s+ per host across a 200-host inventory can silently tack on 30+ minutes and inflate your CI runner bill if hosts are failing slow instead of failing fast.
If boot-finished never appears, check /var/log/cloud-init-output.log on the target first — that's where the actual cloud-init error usually lives.

We cover more of these provisioning edge cases and CI hardening patterns over on kuryzhev.cloud — worth a look if you're running Ansible against autoscaling groups or spot fleets where this race shows up constantly. Reference the official wait_for_connection module docs for the full parameter list, and cloud-init's own boot stages documentation if you need to understand exactly when the boot-finished marker gets written on your specific provider's image.