Most of the container incidents I've helped clean up didn't come from anything exotic. They came from small shortcuts that felt reasonable on a Tuesday and turned into a bad Friday. A latest tag here, a manual build there, one image doing three jobs, and eventually something drifts or breaks and nobody can say for certain what's actually running.
The teams that stay calm in production aren't the ones with the fanciest container tooling. They're the ones with a handful of dull, non-negotiable habits. None of these are clever. That's the point — clever doesn't survive an on-call rotation, but boring discipline does. Here's the set I've settled on, and for each one, the specific failure it's meant to prevent.
1. Pin everything
This is the first habit because it prevents the most failures. Every input to your build should be a specific, named version — the base image, the packages, the dependencies.
# Not this
FROM node:latest
RUN apt-get install -y curl
# This
FROM node:20.11.1-slim
RUN apt-get install -y --no-install-recommends curl=7.88.1-10+deb12u5
The failure it prevents: a build that passed last week and fails today, or worse, succeeds today with different contents. latest means "whatever happened to be current when this ran," which quietly reintroduces the exact "works on my machine" drift containers are supposed to kill. If you pin nothing else, pin your base image.
Where practical, pin the base image by digest (FROM node:20.11.1-slim@sha256:...) so even a re-tagged upstream image can't change what you build.
2. Build in CI, never on a laptop
The artifact that goes to production should be built by an automated pipeline from a clean checkout, not on someone's machine.
The failure it prevents: the unreproducible image. When an engineer builds locally and pushes, that image carries whatever was on their laptop — a cached layer, an uncommitted file, an environment variable, a different toolchain version. Six months later nobody can rebuild it, and you can't debug what you can't reproduce.
A minimal CI build stanza looks about like this:
# Runs in CI, from a clean checkout, on every merge
docker build \
--tag registry.example.com/myapp:${GIT_SHA} \
--file Dockerfile .
docker push registry.example.com/myapp:${GIT_SHA}
The rule I hold to: if it wasn't built by CI, it doesn't go to production. Local builds are for development only.
3. Tag with the git SHA
Tag images with the commit they were built from. Human version tags like 1.4.2 are fine for humans, but the SHA is the one that never lies.
docker build -t registry.example.com/myapp:$(git rev-parse --short HEAD) .
The failure it prevents: the "which version is actually running?" scramble during an incident. With a SHA tag, you take what's deployed, git checkout that exact commit, and you're looking at the precise code that's live. No guessing, no "I think it was the Thursday build." You can keep a friendly 1.4.2 tag and an immutable SHA tag pointing at the same image — the SHA is your ground truth.
Avoid reusing or moving tags. A tag should point at one image forever. If myapp:1.4.2 means something different this month than last month, you've thrown away your ability to reason about history.
4. One process per container
Each container should do one job — the app, or the worker, or the proxy. Not all three wrapped in a shell script and a process supervisor.
The failure it prevents: tangled failure modes and impossible scaling. When one container runs three things, a crash in one takes down the others, your logs are interleaved into mush, and you can't scale the busy component without dragging the idle ones along. One process per container means the orchestrator can restart, scale, and reason about each piece independently.
This is less about a rule and more about a boundary. If two things have different failure characteristics or different scaling needs, they want to be different containers. When you've run enough incidents, you start to really value being able to point at a single unit and say "that one, and only that one, is the problem."
5. Keep images small and single-purpose
Start from a minimal base and add only what the app genuinely needs at runtime. Build tools, compilers, and debug utilities don't belong in the shipped image.
# Multi-stage build: compile in one stage, ship a lean final image
FROM golang:1.22 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/myapp .
FROM gcr.io/distroless/static-debian12
COPY --from=build /out/myapp /myapp
USER 10001
ENTRYPOINT ["/myapp"]
The failure it prevents: slow pulls, wasted storage, and a large attack surface. Every package you don't ship is one that can't have a CVE, can't be used by an attacker who gets a foothold, and doesn't slow down every deploy. A multi-stage build lets you compile with a full toolchain and then throw all of it away, shipping only the binary.
The blast-radius framing helps here: a smaller image is a smaller thing that can go wrong.
6. Scan and validate before shipping
Put a structural check and a vulnerability scan in the pipeline, before the image is allowed near production.
# In CI, fail the build on high-severity findings
trivy image --exit-code 1 --severity HIGH,CRITICAL \
registry.example.com/myapp:${GIT_SHA}
The failure it prevents: shipping a known-vulnerable or structurally broken image because a human forgot to look. The two checks catch different things. A scanner like Trivy finds known CVEs in your OS packages and dependencies. A structural validator catches the Dockerfile-level mistakes — running as root, unpinned bases, missing USER, the footguns that scanners don't flag.
I lean on a free Dockerfile validator for that second check because it's the kind of thing I'd rather automate than trust myself to remember at the end of a long day. Wire both into CI and the check happens whether anyone remembers or not — which is the whole point of a check.
7. Treat images as immutable
Once an image is built and tagged, it's frozen. You don't docker exec into a running production container to patch a file, and you don't rebuild a tag with new contents.
The failure it prevents: configuration drift and unexplainable state. The moment you hand-edit a running container, that container no longer matches its image, and you've lost the thread. Need a change? Build a new image, give it a new tag, roll it out. Need to roll back? The previous image still exists, byte-for-byte unchanged, ready to run.
This one is a mindset more than a command. The container is cattle, not a pet. If it's misbehaving, you replace it; you don't nurse it back to health. Anything that needs to persist — data, state, logs you care about — lives in a volume or an external service, never in the container's writable layer.
Putting it together
None of these habits is impressive in isolation. Pin your versions. Build in CI. Tag with the SHA. One process per container. Small images. Scan and validate. Never mutate a running image. Read them back and they sound almost too obvious to write down.
But that's exactly why they hold up. In my experience, the difference between a calm production environment and a stressful one is rarely a single brilliant decision — it's a stack of unglamorous defaults that quietly remove whole categories of failure. Each habit here is tied to a specific 3 a.m. page it prevents, and together they mean that when something does break, you're debugging one small, known, reproducible thing instead of a mystery.
If you want the fuller Docker walkthrough these habits sit on top of — build, run, registries, the mental model — I keep a maintained version at devopsaitoolkit.com/stacks/docker. But honestly, the habits matter more than any single guide. Pick the two you're not doing yet and wire them into CI this week. Future-you, holding the pager, will be grateful.
Top comments (0)