Ansible server bootstrap locks you out of new hosts: fix

Originally published on kuryzhev.cloud

---

Failure scenario: an Ansible server bootstrap that locks you out

A typical Ansible server bootstrap goes wrong like this. A team writes one playbook that runs against a fresh cloud VM as root. It creates a deploy user, turns on the firewall, and disables root and password logins in sshd. On the first host it appears to work. On the next one, the play stops halfway with UNREACHABLE, and every later attempt fails with Permission denied (publickey).

The first attempt on a new host behaves differently from every re-run. The root session that started the play is already authenticated, so it survives the sshd restart. A fresh connection is rejected, because root is now disabled and the new user's key may never have been installed correctly.

The playbook below is the shape that produces this failure. It is a composite for illustration, not a recording of a real incident.

---
# ANTI-PATTERN: one play, one connection user, hardening mixed with account setup
- name: Bootstrap server
  hosts: new_servers
  remote_user: root          # inventory also says ansible_user=root, forever
  tasks:
    - name: Create deploy user
      ansible.builtin.user:
        name: deploy
        groups: sudo

    - name: Disable root and password login
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: "^#?{{ item.key }}"
        line: "{{ item.key }} {{ item.value }}"
      loop:
        - { key: PermitRootLogin, value: "no" }
        - { key: PasswordAuthentication, value: "no" }
      notify: restart ssh

    - name: Enable firewall
      community.general.ufw:
        state: enabled          # default policy may deny port 22 if no allow rule exists yet

  handlers:
    - name: restart ssh
      ansible.builtin.service:
        name: sshd
        state: restarted

Nothing in this file is syntactically wrong, which is why it passes review. The problems are in the ordering and in what is never checked.

Why it happens

Several independent causes combine here, and any one of them is enough to lock you out.

The authorized key is never installed. The deploy user is created, but no task places a public key in its ~/.ssh/authorized_keys. Once root login and passwords are off, no valid credential remains.

The firewall can cut off port 22. With the UFW module, enabling the firewall with a default deny incoming policy and no allow rule for SSH drops new connections. The module's documented behavior is to apply exactly what you ask, so a missing allow rule is your mistake, not a bug.

Watch out for drop-in precedence. Current Debian and Ubuntu images ship an Include /etc/ssh/sshd_config.d/*.conf line at the top of sshd_config. For most sshd keywords the first value read wins. A cloud-init file such as 50-cloud-init.conf that sets PasswordAuthentication yes can therefore override your edit at the bottom of the main file. You may think you hardened the host when you did not. Verify the effective values with sshd -T.

There are also three smaller problems:

• Handlers run at the end of the play. If any later task fails, the restart is skipped and the host sits in an inconsistent state until you rerun.
• Minimal images may have no Python, so fact gathering fails before your first task.
• The service is named ssh on Debian-family systems and sshd on RHEL-family ones. Some recent Ubuntu releases also use socket activation, so check how your target restarts the daemon.

The fix: split the bootstrap into verified phases

The principle is simple: never remove an access path until you have proven the replacement works. The playbook below uses one play to create access. A second play connects as the new user and does the hardening.

Phase 1 connects with the image's initial credentials and creates the admin user, its key, and sudo rights.

---
- name: Phase 1 - create admin access
  hosts: new_servers
  gather_facts: false
  vars:
    ansible_user: root                      # play vars override inventory for this play only
  tasks:
    - name: Ensure Python exists (minimal Debian/Ubuntu images; use dnf on RHEL-family)
      ansible.builtin.raw: test -x /usr/bin/python3 || (apt-get update && apt-get install -y python3)
      changed_when: false

    - name: Gather facts now that Python is present
      ansible.builtin.setup:

    - name: Create deploy user
      ansible.builtin.user:
        name: deploy
        shell: /bin/bash
        create_home: true

    - name: Install public key
      ansible.posix.authorized_key:
        user: deploy
        key: "{{ lookup('ansible.builtin.file', 'files/deploy.pub') }}"
        exclusive: true                     # removes stray keys; remove this line if you manage keys elsewhere

    - name: Passwordless sudo, validated before it lands
      ansible.builtin.copy:
        dest: /etc/sudoers.d/90-deploy
        content: "deploy ALL=(ALL) NOPASSWD:ALL\n"
        mode: "0440"
        validate: /usr/sbin/visudo -cf %s   # a broken sudoers file is its own lockout

Phase 2 connects as deploy. If the key or sudo is wrong, the play fails here, while root access still works.

- name: Phase 2 - verify access, then harden
  hosts: new_servers
  become: true
  vars:
    ansible_user: deploy
  tasks:
    - name: Prove key login and sudo work (fails before anything is locked down)
      ansible.builtin.command: id -u
      register: uid
      changed_when: false
      failed_when: uid.stdout != "0"

    - name: Allow SSH BEFORE enabling the firewall
      community.general.ufw:
        rule: allow
        port: "22"
        proto: tcp

    - name: Enable firewall
      community.general.ufw:
        state: enabled
        policy: deny
        direction: incoming

    - name: SSH hardening drop-in (00- sorts before 50-cloud-init.conf)
      ansible.builtin.copy:
        dest: /etc/ssh/sshd_config.d/00-hardening.conf
        content: |
          PermitRootLogin no
          PasswordAuthentication no
        mode: "0644"
      notify: reload ssh

    - name: Test full sshd config before any reload
      ansible.builtin.command: /usr/sbin/sshd -t
      changed_when: false

    - name: Apply now, not at end of play
      ansible.builtin.meta: flush_handlers

    - name: Drop the multiplexed session so the next task opens a fresh SSH login
      ansible.builtin.meta: reset_connection

    - name: Confirm a new key-based connection still works
      ansible.builtin.wait_for_connection:
        timeout: 30

  handlers:
    - name: reload ssh
      ansible.builtin.service:
        name: "{{ 'ssh' if ansible_os_family == 'Debian' else 'sshd' }}"
        state: reloaded

The reset_connection step matters. Ansible normally reuses a persistent SSH control connection, so without it wait_for_connection could succeed over the old session and prove nothing. Also note that if sshd -t fails, the play stops before the reload, but the bad drop-in stays on disk. Fix or remove it before anything restarts sshd or reboots the host.

Once the host is hardened, rerunning is safe if you target only the second play, for example with a tag or a separate inventory group. Keep the root-based play restricted to hosts that have never been bootstrapped. The module names above are from the current ansible.builtin, ansible.posix, and community.general collections. Check the Ansible collections documentation for your installed versions and parameter details.

Prevention checklist

Before the bootstrap playbook goes near a real fleet, run through the points below. Most of them are cheap to automate in CI against a throwaway VM.

[ ] Access first, hardening second: separate plays, separate connection users
[ ] A task proves key login AND sudo work as the new user before hardening
[ ] Firewall: allow 22/tcp (or your SSH port) before enabling; default deny after
[ ] sshd changes live in a drop-in named to sort before cloud-init files
[ ] sshd -t runs before every reload; sudoers uses validate: visudo -cf %s
[ ] Handlers flushed explicitly (meta: flush_handlers) when order matters
[ ] meta: reset_connection before verifying reconnection, so the check uses a fresh login
[ ] Service name handled per OS family (ssh vs sshd); socket activation checked
[ ] Python bootstrap via raw for minimal images, with changed_when: false
[ ] Effective config verified with: sshd -T | grep -Ei 'permitrootlogin|passwordauth'
[ ] Provider console / serial access or a rescue snapshot confirmed before the first run
[ ] Tested on a disposable host: real run first, then a second run (--check --diff is only
    meaningful after the first real run, since Phase 2 needs the deploy user to exist)

Watch out for the second run. A bootstrap playbook that only succeeds once is a script, not automation. Run it twice against the same disposable host. The second pass of the hardening play should report no changes. If it cannot even connect, the inventory still assumes root access that no longer exists.

Keep one out-of-band recovery route, such as the provider's web console or a rescue mode. Confirm it works before you automate anything that touches SSH or the firewall. It will not prevent the mistake, but it turns a rebuild into a quick repair. For more infrastructure automation patterns, browse the rest of kuryzhev.cloud.

A reliable Ansible server bootstrap is mostly about ordering and proof: create access, test it from a fresh connection, and only then close the old doors.

Related

• Cisco Config Backups with Ansible: ios_config vs ios_command
• Terraform Modules for AWS VPC, IAM, and S3: Structure That Scales
• WordPress performance profiling with php-fpm status and slow logs