What a fake production stack taught me about CI/CD

I recently built jenkins-ansible-sample, a self-contained sandbox that simulates a real deployment pipeline: Jenkins builds and tests an app, Ansible rolls it out across a fake dev/staging/prod fleet over SSH, and a health-check-driven rollback gets exercised by a deliberately broken deploy. Everything runs as Docker containers on a single machine — no cloud account, no real servers.

The point of the project wasn't the app (a two-endpoint Flask service). It was to force myself through the parts of CI/CD that tutorials usually skip: real host topology, secrets that have to flow through three different execution contexts, and a rollback path that has to actually fire, not just exist in a diagram. This post is less "here's how to configure Jenkins" and more "here's what I'd tell someone before they start" — the general principles, with the code that fell out of them.

Simulate the real topology, not a shortcut

It's tempting, when building a demo, to have your "deploy" step just be a docker exec into a sibling container. It's fast and it works. It also teaches you nothing about the failure modes of a real deploy, because docker exec doesn't have host keys, network partitions, or a login shell.

So the fleet here is plain containers running sshd as PID 1, reachable only over real SSH:

fleet-dev:
  build: docker/ssh-fleet
  hostname: dev
  privileged: true
  networks:
    fleet_net:
      aliases: [dev]
  ports:
    - "2201:22"

Ansible talks to dev exactly the way it would talk to a real VM — inventory groups, an SSH key, become: true for privilege escalation. The theoretical point: the fidelity of your test environment should match the fidelity of the failure modes you actually want to catch. A shortcut that skips SSH also skips every SSH-shaped problem (host key changes, key permissions, connection timeouts) you'll eventually hit for real.

The same idea shows up in docker_engine, a role that installs an actual Docker daemon onto each of these simulated hosts (privileged: true, its own nested dockerd) rather than assuming Docker is already there. It's more setup, but it means the role is exercised the same way it would run against a freshly provisioned VM, not against an environment that already has half the prerequisites baked in.

Docker-outside-of-Docker: a tradeoff, not a hack

Any CI step that needs to build or run containers has to reach a Docker daemon somehow. There are two common answers: run a nested dind (Docker-in-Docker) daemon inside the CI container, or mount the host's docker.sock into it (Docker-outside-of-Docker, DooD). This project uses DooD everywhere — the Jenkins agent, the ansible-control container, Molecule's test runner:

ansible-control:
  build: docker/ansible-control
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

DooD is simpler and has none of dind's storage-driver quirks, but the tradeoff is real and worth stating plainly: a container with the host's docker.sock mounted has root-equivalent access to that host. Any process inside it can launch a privileged container and walk straight out of its own sandbox. That's an acceptable tradeoff for a single-tenant CI box you control; it would not be an acceptable default for, say, a shared multi-tenant build service running arbitrary user code.

The lesson generalizes: infrastructure shortcuts aren't inherently bad, but they need to be a named decision with a stated blast radius, not an implementation detail nobody thought about. If you can't articulate what a shortcut costs you, you haven't finished evaluating it.

One consequence of DooD worth knowing about: when a container spawned via docker.sock is a sibling of the container that launched it, not something nested inside it, cleanup can't rely on the parent's lifecycle to take the child down with it. Molecule's test container in this project is launched exactly that way, which is why the CI stage runs it with --destroy=always — without that, a failed step earlier in the test sequence would skip teardown, and the test container would happily outlive the short-lived agent that spawned it.

Rollback is a state machine, not an if statement

The centerpiece of the project is the deploy role's rollback logic, and it's worth describing as what it actually is: a small state machine with three states (deploying, healthy, rolling back), not a single try/catch.

- name: Wait for the new container to pass its post-deploy health check
  ansible.builtin.uri:
    url: "http://localhost:{{ app_port }}/health"
    status_code: 200
  register: health_result
  until: health_result.status | default(0) == 200
  retries: "{{ health_check_retries }}"
  delay: "{{ health_check_delay }}"
  failed_when: false

- name: Roll back to the last-known-good image if the health check failed
  when: health_result.status | default(0) != 200
  block:
    - name: Start the last-known-good image in its place
      ansible.builtin.command: "docker run -d --name {{ app_name }} ... {{ rollback_image }}"
      when: rollback_image | length > 0
    - name: Wait for the rolled-back container to pass its health check
      ansible.builtin.uri:
        url: "http://localhost:{{ app_port }}/health"
        status_code: 200
      ...

- name: Fail the play if the post-deploy health check never passed
  ansible.builtin.fail:
    msg: "..."
  when: health_result.status | default(0) != 200

A few decisions here are more important than the YAML makes them look:

• The "last known good" reference is captured before touching anything, not reconstructed after a failure. By the time you know a deploy is bad, it's too late to reliably ask "what was running a minute ago" — you have to have written it down while it was still true.
• A rolled-back deploy still fails the pipeline. The host ends up healthy, but the build is still reported red. Silently "fixing itself" and reporting green would hide a real regression behind a rollback that happened to work — the whole point of the signal is to make someone look at it.
• Rollback failure is a distinct, more serious outcome than deploy failure, and the code actually distinguishes it (a rollback that itself fails its health check gets a different message than a clean rollback). Collapsing both into "deploy failed" throws away the information an on-call engineer needs most: is this a bad release, or is the host actually down?
• any_errors_fatal: true plus per-environment plays means a bad dev deploy never even attempts staging or prod. Promotion gates only work if a failure upstream actually stops the pipeline instead of merely being logged.

One secret, one indirection, three consumers

Secrets management often turns into a different bespoke mechanism per environment: a .env file locally, a different secret store in CI, yet another one in the deployment tool. This project instead puts a single layer of indirection in front of one password and lets every consumer go through it.

ansible.cfg points at a script instead of a static file:

[defaults]
vault_password_file = vault_pass.sh

#!/bin/sh
set -eu
: "${ANSIBLE_VAULT_PASSWORD:?ANSIBLE_VAULT_PASSWORD must be set}"
echo "$ANSIBLE_VAULT_PASSWORD"

Locally, that's an exported shell variable. In GitHub Actions, it's a repository secret injected as an env var on the steps that need it. In Jenkins, it's a credential bound via withCredentials in the pipeline. All three paths terminate at the exact same script reading the exact same variable name — nothing about how Ansible reads its vault password changes depending on who's driving it. The theoretical payoff: the number of places that need to know "how do I get the secret" should be one, regardless of how many places need the secret.

That same discipline shows up in how the Jenkinsfile passes the SSH key through to a shell command:

sh "ansible-playbook deploy.yml --limit dev -e app_deploy_app_image_tag=${IMAGE_TAG} " +
   "-e fleet_ssh_private_key_file=" + '$FLEET_SSH_KEY'

$FLEET_SSH_KEY is deliberately left single-quoted so the shell expands it at runtime, not Groovy's string interpolation at pipeline-compile time. Interpolating a credential straight into the script text via a Groovy GString bypasses Jenkins' log-masking, because masking works by pattern-matching the credential's value in the process's own output, not by tracking where a variable came from — if the value never appears as a literal in the generated shell command, masking has nothing to catch. It's a small, easy-to-miss distinction with a real consequence: get it backwards and your vault password shows up in plaintext in a build log.

If it's not codified, it doesn't count as tested

Every config surface in this project is provisioned by code, not clicked into place: Jenkins' users, credentials, and seed job are all defined in jenkins.yaml (JCasC); Grafana's datasource and dashboard are provisioned from files, not built by hand in the UI; even which services start at all is driven by Compose profiles (--profile tools, --profile observability) rather than commenting lines in and out.

jenkins:
  systemMessage: "jenkins-ansible-sample — configured via JCasC, do not click around by hand"
  numExecutors: 0   # forces every build onto a Docker Cloud agent, never the controller itself

The reason this matters more than it sounds: a manually clicked configuration is a fact about one running instance, not a fact about the system. It can't be code-reviewed, it can't be diffed between two setups that disagree, and it silently stops being true the moment someone reinstalls. Config-as-code turns "how is this configured" from a question you answer by inspecting a live system into one you answer by reading a file.

Different artifacts fail differently — lint each one on its own terms

It's tempting to reach for one linter and call it done. This project runs four completely different checks, because a Dockerfile, a YAML playbook, a Python app, and an Ansible role's actual runtime behavior fail in four different ways:

• hadolint for Dockerfiles (image-layer hygiene, not application logic)
• yamllint + ansible-lint for the Ansible layer (syntax and Ansible-specific anti-patterns)
• pytest, run as a build stage the final image can't skip:

  FROM base AS test
  RUN uv sync --frozen --group test --no-install-project
  COPY app.py .
  COPY tests/ tests/
  RUN python -m pytest -q

  FROM base AS final
  COPY --from=test /app/app.py .   # only reachable if the test stage above succeeded

• molecule for the common role, which spins up a real container, applies the role, and asserts on the actual resulting state of the machine — not the playbook's syntax, but what it did:

  - name: /etc/timezone should be UTC
    ansible.builtin.command: cat /etc/timezone
    register: tz
    changed_when: false
    failed_when: tz.stdout != "UTC"

A linter can tell you a playbook is well-formed; only actually converging it against a host and asserting on the result tells you it does what you think it does. Static analysis and behavioral testing catch disjoint sets of bugs — treating a passing lint as proof of correctness is a mistake regardless of language or tool.

Make the failure mode visible, not just handled

Deploy success/failure is worth almost nothing as a signal if nobody sees it in context. Rather than invent a new metric for "a deploy happened," this project posts deploy attempts as annotations onto a dashboard that already exists:

- name: Record this deploy as a Grafana annotation
  ansible.builtin.uri:
    url: "{{ grafana_url }}/api/annotations"
    method: POST
    body: {text: "{{ deploy_event_text }}", tags: [deploy, "{{ environment_name }}", "{{ deploy_event_tag }}"]}
  when: app_deploy_grafana_annotate | default(true)
  failed_when: false   # an unreachable dashboard must never fail a deploy over it

Two design choices here matter beyond the specific tool: annotating an existing CPU/memory dashboard means a rollback shows up as a visible marker on the same graph where its effects would be seen, instead of living in a separate "deploys" panel nobody checks. And failed_when: false encodes an explicit priority order — observability is a consumer of the deploy, not a dependency of it. A monitoring outage should never be able to fail a deploy.

CI failures aren't always about your code

Not every red build is a bug in the thing being tested — sometimes it's drift between two steps in the pipeline that only look the same. While working on this project, one CI job's lint step failed consistently, and the cause turned out to be nothing about the YAML being linted: ansible-lint shells out to ansible-playbook --syntax-check internally, which needs the vault password just to parse a group_vars file — even though nothing in a syntax check touches the vaulted values. The pipeline step running the linter simply hadn't been given that credential, while every later step that actually deployed had.

The general takeaway: when a CI step fails in a way that doesn't match the tool's stated job, check what else that tool does under the hood before assuming the input is wrong. And when you're debugging a step that runs inside a container with pinned tool versions, reproducing it with the exact same versions locally turns a guessing game into a repeatable test — version drift between "works on my machine" and CI is its own separate failure mode, easy to rule out early if you pin first and debug second.

What's deliberately not here

Kubernetes, multi-cloud provisioning, and blue-green deploys are all out of scope on purpose. The goal was depth on one deploy pattern (rolling deploy + health-check-gated rollback, over SSH, onto host-like targets) rather than breadth across every pattern that exists. That's its own lesson: a project that tries to demonstrate everything ends up demonstrating nothing well — better to pick one path all the way through, including its failure modes, than to sketch three and finish none.

---

The full project, architecture diagram, and README are at github.com/ykpraveen/jenkins-ansible-sample.