Molecule Ansible Testing: 3 Docker Pitfalls That Fooled Us

#docker #devops

Originally published on kuryzhev.cloud

Testing Ansible roles with Molecule felt bulletproof for months — every pull request passed, every scenario converged clean, every merge shipped without drama. Then a role that patched sysctl and firewalld rules sailed through CI and took down a production web tier an hour later. That's when we found out how much our "passing" test suite had been lying to us.

Context

Before Molecule, our testing workflow was embarrassingly simple: apply the role to a shared staging VM, watch it converge, and if nothing exploded, ship it. "If it converges once, ship it" was basically our unofficial CI policy. No idempotence checks, no OS matrix, no isolation between test runs — just one long-lived VM that had drifted from prod in ways nobody had audited in over a year.

The wake-up call came when a role designed to harden sysctl parameters and firewalld zones passed cleanly on staging but broke prod outright. Staging was running a newer kernel and a different firewalld backend than half our production fleet. The role assumed capabilities that simply weren't there on older nodes. That incident is what pushed us toward Molecule and Docker-based scenario testing.

We landed on molecule 6.0.3 with ansible-core 2.15, molecule-plugins[docker] installed separately (the driver split happened back in Molecule 4.x, so this trips up anyone following older guides), and Docker Engine 24.x both locally and on our GitLab CI runners. On paper, this setup looked like exactly what we needed: isolated, reproducible, fast. In practice, we made three mistakes that gave us false confidence for longer than I'd like to admit.

Mistake 1: Treating a Docker container as a drop-in replacement for a VM

This one stung because it seemed so obviously fine at first. We spun up scenarios against a plain ubuntu:22.04 image, roles converged, tests went green, and we assumed that was equivalent to running against a real VM. It wasn't. Containers don't ship with a working systemd init, real cgroups, or loaded kernel modules by default. Any role touching systemctl, sysctl, or firewalld was effectively testing against a fake target.

The error that finally made us pay attention was this one, staring back at us in CI logs:

Failed to connect to bus: No such file or directory

That's the classic signature of a container missing a proper init system. The fix people skip — and we skipped it too, initially — is setting privileged: true and cgroupns_mode: host on the platform definition, and swapping bare distro images for ones like geerlingguy/docker-ubuntu2204-ansible or geerlingguy/docker-rockylinux9-ansible, which ship with systemd pre-configured to actually run inside a container. Watch out for this if you're testing anything that assumes a live init system — a green Molecule run against a bare image tells you almost nothing.

Mistake 2: Not matching the test image to the actual production OS/version

Even after fixing the systemd problem, we had a second, quieter issue: our molecule.yml platforms list only tested ubuntu2204, while roughly half of our production fleet was still running ubuntu2004 with an older Python and different apt behavior. Everything passed because we were only ever testing the newest, fastest-to-pull image — not the actual OS matrix we deployed against.

This masked real problems. Package manager quirks between apt and dnf, differences in default Python versions, subtle behavior changes in module defaults across Ansible collections — none of it surfaced because we were validating one distro and calling it done. I stopped trusting "molecule test passed" as a meaningful signal the day I realized our platforms list had quietly drifted away from our inventory groups.

The lesson was blunt: the platforms section in molecule.yml has to mirror your real inventory_group/os_family matrix, not whatever image happens to be cached locally and pulls fastest. If you support Ubuntu 20.04, 22.04, and Rocky 9 in production, your test matrix needs all three, every time, not just on days you remember to check.

Mistake 3: Skipping or misconfiguring the idempotence check

This was the most humbling mistake of the three. Molecule's full test sequence is create → prepare → converge → idempotence → verify → destroy, and idempotence is arguably the whole point — it re-runs the role and fails if anything reports as changed the second time. We were running molecule converge locally during development and calling it good enough, which quietly skipped that stage entirely.

A template task written using command/shell instead of the proper template module reported changed=1 on every single run. Nobody noticed for weeks because nobody was actually running the idempotence stage in CI either — we had wired an old verifier config referencing testinfra, which Molecule 5.x removed in favor of native Ansible-based verification. Old tutorials still reference testinfra.py scaffolding, and if you copy that pattern into a current Molecule setup, it silently no-ops instead of failing loudly.

Here's what the failure actually looks like once idempotence is wired correctly:

# What idempotence failure actually looks like in CI logs
$ molecule test

...
PLAY RECAP (idempotence run) *************************************************
instance-ubuntu2204 : ok=8    changed=1    unreachable=0    failed=0

--> Idempotence test failed because of the following tasks:
    * generate nginx config from raw command instead of template module

# The offending task (before fix):
- name: Render nginx config
  command: envsubst < /tmp/nginx.conf.tpl > /etc/nginx/nginx.conf
  # always reports "changed" — no idempotence awareness

# After fix — using the template module:
- name: Render nginx config
  template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    mode: "0644"
  # second run correctly reports changed=0

Once we enforced the idempotence stage in CI, roles that had "passed" for months started failing immediately — which was uncomfortable, but exactly what we needed.

What we do differently now

After the retro, we rebuilt the whole scenario config from scratch instead of patching around it. The molecule.yml platforms matrix is now generated from the same group_vars used for real inventory, with a dedicated CI matrix job per OS — Ubuntu 20.04, 22.04, Rocky 9, and Debian 12 all run in parallel rather than us hoping one image is representative.

Here's the scenario config we ended up with, with systemd handling and the idempotence stage explicit rather than implied:

# molecule/default/molecule.yml
# Matches production OS matrix instead of a single "fast to pull" image
---
driver:
  name: docker

platforms:
  - name: instance-ubuntu2204
    image: geerlingguy/docker-ubuntu2204-ansible:latest
    privileged: true                # required for systemd tasks (sysctl, firewalld)
    cgroupns_mode: host              # avoids "Failed to connect to bus" errors
    pre_build_image: true
    command: ""                      # let the base image's own init run

  - name: instance-rocky9
    image: geerlingguy/docker-rockylinux9-ansible:latest
    privileged: true
    cgroupns_mode: host
    pre_build_image: true
    command: ""

provisioner:
  name: ansible
  env:
    ANSIBLE_ROLES_PATH: ../../../
  inventory:
    group_vars:
      all:
        # mirrors real inventory vars — do NOT diverge from prod defaults
        env_name: molecule_test
        firewall_enabled: true
  lint:
    name: ansible-lint

verifier:
  name: ansible   # native verifier — testinfra is deprecated since Molecule 5

scenario:
  test_sequence:
    - dependency
    - lint
    - syntax
    - create
    - prepare
    - converge
    - idempotence   # <-- this stage caught our "changed=1 on rerun" bug
    - verify
    - destroy

molecule test — the full sequence, idempotence included — is now a required CI gate before merge. Nobody is allowed to substitute molecule converge as a stand-in anymore, no matter how tight the deadline is. We also pinned ansible-lint 6.22 and matching yamllint versions in requirements.txt after finding that version drift between local pre-commit hooks and CI was producing false "passing" commits locally that failed the moment they hit the pipeline.

On the security side, we stopped putting anything sensitive in the provisioner env block. Vault passwords are mounted via ANSIBLE_VAULT_PASSWORD_FILE at runtime through CI secret injection, never committed alongside molecule.yml. And because running a full four-OS matrix multiplies pipeline minutes roughly 4x, we cache Docker layers aggressively and reuse base images across scenarios to keep runtime and cost sane — see the official Docker build cache docs if you're tuning this yourself.

One small habit that's saved us real disk space: running docker system prune on a schedule, and never walking away from a failed molecule test run without also running molecule destroy. Orphaned containers pile up fast, and docker system df creeping upward over a few weeks is a genuinely annoying problem to debug on a dev laptop. If you're setting up CI infrastructure around this kind of testing, our DevOps notes at kuryzhev.cloud cover a few adjacent patterns we use for pipeline hygiene.

Testing Ansible roles with Molecule is still the right call — it caught real bugs the moment we configured it honestly. The mistake was never the tool. It was trusting a green checkmark without asking what it was actually validating.