DEV Community: Pawan Shinde

The Clean Plate Problem: How Multi-Stage Builds and Distroless Images Cut Container Attack Surfaces to Zero

Pawan Shinde — Thu, 22 Oct 2026 12:41:44 +0000

A few weeks ago, I ran an audit on a backend microservice container running in our staging cluster.The application itself was a compiled Go binary that took up less than 20 MB of disk space. But when I ran docker images, the resulting image weighed in at nearly 1 GB.Even worse, when I attached an interactive terminal into that running container:Bashdocker exec -it payment-service /bin/bash
I found a full-blown Debian Linux environment sitting inside. Compilers, package managers (apt), network downloaders (curl, wget), and an active root shell were all sitting right next to our core application logic.If an attacker managed to find a remote code execution (RCE) bug in that service, I hadn’t just handed them an app bug—I had handed them an entire Linux workstation inside our VPC.Here is how modern container workflows solve this using two industry patterns: Multi-Stage Builds and Distroless Images.1. Multi-Stage Builds: The Restaurant Kitchen AnalogyThink about eating at a restaurant.In the kitchen, the chef uses heavy knives, cutting boards, vegetable peelers, mixing bowls, and blazing stoves. When the waiter brings your order to the dining table, you get a clean plate with the finished meal.You would never expect the waiter to dump the chopping board, raw vegetable peels, greasy pans, and meat scraps directly onto your table.Yet that is exactly what a single-stage Dockerfile does.+-----------------------------------------------------------+
| SINGLE STAGE (Messy Kitchen + Food Shipped Together) |
| Compilers + SDKs + Git + Headers + Source Code + Binary |
| Image Size: ~900 MB |
+-----------------------------------------------------------+
When building an application in Go, Rust, C++, Java, or even Node.js, you need heavy build tools: compilers, package managers, development headers, and build caches. But your production environment only needs the final compiled binary or transpiled assets.The Fix: Multi-Stage SeparationA multi-stage build uses multiple FROM instructions in a single Dockerfile. Each FROM starts a brand-new stage, letting you copy only the artifacts you want from earlier stages.Here is what that looks like in practice:Dockerfile# -------------------------------------------------------------

STAGE 1: "The Kitchen" (Builder stage with compilers and SDKs)

-------------------------------------------------------------

FROM golang:1.23-alpine AS builder

WORKDIR /build

Cache dependency downloads

COPY go.mod go.sum ./
RUN go mod download

Copy source and compile down to a standalone binary

COPY . .
RUN CGO_ENABLED=0 go build -o /build/payment-api .

-------------------------------------------------------------

STAGE 2: "The Dining Table" (Final production runtime)

-------------------------------------------------------------

FROM alpine:3.20

WORKDIR /app

Copy ONLY the compiled binary from Stage 1

COPY --from=builder /build/payment-api /app/payment-api

EXPOSE 8080
CMD ["/app/payment-api"]
The Difference:Single-Stage Image: ~900 MB (Go compiler, source code, build caches, git).Multi-Stage Image: ~15 MB (Tiny Alpine OS layer + compiled binary).Why It Matters:During high-traffic autoscaling events, Kubernetes doesn't have to pull 1 GB images across the network to spin up new pods. A 15 MB image pulls in less than a second, saving cloud bandwidth costs and speeding up recovery times.2. Distroless Images: Stripping Down to the ChassisDropping image size to 15 MB with Alpine is a huge step forward, but from a security perspective, Alpine still ships with an interactive shell (/bin/sh) and a package manager (apk).To understand why this is a risk, consider an analogy:A standard family car comes with air conditioning, leather seats, a radio, a trunk, and a glovebox. A Formula 1 race car strips out everything that doesn't make the car go fast: no back seat, no radio, no air conditioning, no trunk.A Distroless image (originally created by Google) takes that Formula 1 approach to containers. It contains only your application and its minimal runtime dependencies (like glibc or SSL root certificates).It explicitly omits:Package managers (apt, apk, yum)System utilities (curl, wget, ls, grep)Interactive shells (/bin/sh, /bin/bash)Standard Base (Ubuntu/Debian): [ Shell + Tools + Package Manager + App ]
Alpine Base: [ Shell + apk + Minimal OS + App ]
Distroless Base: [ App + Direct Runtime Libs ONLY ]
A Real-World Security ScenarioImagine a critical vulnerability (like an RCE or file-upload exploit) is discovered in a patient records microservice.Scenario A: The Container Runs on Standard UbuntuThe attacker exploits the application bug and executes an arbitrary command.They run curl or apt-get install nmap directly inside the container.They download an external botnet or reconnaissance script.They explore the local container filesystem using ls -la and read configs.They use the running container as a jump box to pivot into the rest of your private network.Scenario B: The Container Runs on DistrolessThe attacker exploits the bug and tries to invoke /bin/sh.The operating system responds:Plaintextexec: "/bin/sh": stat /bin/sh: no such file or directory
The attacker tries calling curl or wget to pull remote tools. Neither exists.The exploit fails because there is no execution environment inside the container to run unauthorized binaries.The Gold Standard: Multi-Stage + DistrolessIn production systems handling sensitive workloads, you can combine both techniques. You build in a full developer SDK environment, but ship on Google's static Distroless image:Dockerfile# Stage 1: Build the code in a full compiler environment
FROM golang:1.23 AS builder

WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app/server .

Stage 2: Deploy to Google's Distroless static runtime

FROM gcr.io/distroless/static-debian12

Copy the binary from the builder stage

COPY --from=builder /app/server /server

Distroless provides a built-in non-root user (UID 65532: nonroot)

USER nonroot:nonroot

EXPOSE 8080
ENTRYPOINT ["/server"]
What You Get:Tiny footprint: Image sizes drop to the bare minimum required for execution.Non-root by default: Runs under user nonroot, neutralizing container escape techniques that rely on root privileges.No shell attacks: Traditional living-off-the-land techniques (using installed utilities like curl or bash) are impossible.Clean vulnerability scans: Tools like Trivy or Snyk show zero CVEs for unneeded OS packages, because those packages were never installed in the first place.Comparison SummaryConceptWhat It IsProblem It SolvesProduction ImpactMulti-Stage BuildA Dockerfile using multiple FROM stages to separate build tools from runtime artifacts.Bloated image sizes caused by compilers, test suites, and package caches.Cuts storage needs, saves cloud network transfer costs, and speeds up auto-scaling.Distroless ImageA base image containing only the app and direct dependencies—no shell, no package manager.Attackers using internal container shells and OS utilities to pivot into your private network.Eliminates package vulnerabilities and neutralizes shell-based exploit payloads.Wrapping UpBuilding clean containers isn't just about saving a few hundred megabytes of disk space. It fundamentally changes the threat model of your workloads in the cloud.If your production container doesn't need a compiler to run, don't ship one. And if it doesn't need an interactive shell to serve traffic, strip the shell out entirely.Keep the kitchen in the build stage, and bring only the clean plate to production.

I Replaced My Manual 15-Step Linux Hardening Runbook with Ansible

Pawan Shinde — Sun, 04 Oct 2026 09:31:34 +0000

Whenever I used to spin up a new Linux server, my manual setup routine was always the same: SSH in, update system packages, configure UFW firewall rules, set up Fail2ban jails, write Nginx proxy configurations with security headers, and verify that services were enabled at boot.

Manually running through that checklist took roughly 25 minutes per machine. On a single test box it wasn't a blocker, but repeating those steps across multiple environments quickly became repetitive, tedious, and prone to small human errors—like forgetting a header or mistyping a firewall port.

Industry incident data shows that over 80% of unauthorized access incidents on internet-facing compute nodes stem from basic configuration drift and missed hardening steps, such as unrestricted default ports or unpatched vulnerabilities.

To eliminate manual toil and ensure consistent infrastructure, I decided to automate the entire process using an idempotent Ansible playbook.

Here are my live test notes, configuration files, and terminal benchmarks from running it against a clean Ubuntu 22.04 node.

The Automation Target

Instead of typing 15 commands into a shell every time, the playbook declaratively enforces this target state:

System Packages: Update the apt cache, upgrade existing packages, and install core tools (ufw, fail2ban, nginx, curl, htop, logrotate).
Firewall Lockdown: Block all inbound traffic by default, only opening ports 22 (SSH), 80 (HTTP), and 443 (HTTPS).
Brute-Force Protection: Deploy a custom Fail2ban jail that automatically bans an IP for 1 hour after 5 failed SSH authentication attempts within 10 minutes.
Reverse Proxy & SRE Health Check: Configure Nginx with standard security headers (X-Frame-Options, X-Content-Type-Options) and add a /healthz endpoint returning HTTP 200 for monitoring probes.
Strict Idempotency: Re-running the script against an already-configured node makes zero modifications and triggers zero service restarts.

Project Structure

I structured the project into a modular directory:


text
ansible-linux-node-hardener/
├── inventory.ini
├── playbook.yml
├── templates/
│   ├── nginx.conf.j2
│   └── jail.local.j2
└── README.md

1. Inventory & Dynamic Templates
inventory.ini
I defined the target test node and variable overrides:
[webservers]
node01 ansible_host=192.168.1.50 ansible_user=ubuntu ansible_ssh_private_key_file=~/.ssh/id_rsa

[webservers:vars]
http_port=80
server_domain=api.lab.internal

templates/nginx.conf.j2
A Jinja2 template to configure Nginx to proxy traffic to an internal app on port 8080, enforce security headers, and expose an uptime health check:
server {
    listen {{ http_port }};
    server_name {{ server_domain }};

    # Security headers to prevent clickjacking and MIME-type sniffing
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;

    location / {
        proxy_pass [http://127.0.0.1:8080](http://127.0.0.1:8080);
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
    }

    # Lightweight endpoint for uptime/health checks
    location /healthz {
        access_log off;
        return 200 "healthy\n";
    }
}

templates/jail.local.j2
A local override for Fail2ban to protect SSH access without modifying package-managed files:

[DEFAULT]
bantime  = 1h
findtime = 10m
maxretry = 5

[sshd]
enabled = true
port    = ssh
backend = systemd

2. The Automation Playbook (playbook.yml)
The playbook runs system updates, installs packages, sets firewall rules, templates configs, and uses handlers so services reload only when configuration files actually change:

---
- name: Automate Linux Node Hardening and Nginx Proxy
  hosts: webservers
  become: true
  gather_facts: true

  vars:
    required_packages:
      - ufw
      - fail2ban
      - nginx
      - curl
      - htop
      - logrotate

  tasks:
    - name: Update apt cache and upgrade system packages
      apt:
        update_cache: yes
        upgrade: dist
        cache_valid_time: 3600

    - name: Install baseline security and proxy packages
      apt:
        name: "{{ required_packages }}"
        state: present

    - name: Configure UFW default policies
      ufw:
        direction: "{{ item.direction }}"
        policy: "{{ item.policy }}"
      loop:
        - { direction: 'incoming', policy: 'deny' }
        - { direction: 'outgoing', policy: 'allow' }

    - name: Allow essential inbound ports
      ufw:
        rule: allow
        port: "{{ item }}"
        proto: tcp
      loop:
        - "22"
        - "80"
        - "443"

    - name: Enable UFW service
      ufw:
        state: enabled

    - name: Deploy Fail2ban configuration
      template:
        src: templates/jail.local.j2
        dest: /etc/fail2ban/jail.local
        owner: root
        group: root
        mode: '0644'
      notify: Restart Fail2ban

    - name: Deploy Nginx reverse proxy configuration
      template:
        src: templates/nginx.conf.j2
        dest: /etc/nginx/sites-available/default
        owner: root
        group: root
        mode: '0644'
      notify: Reload Nginx

    - name: Ensure baseline services are running and enabled at boot
      systemd:
        name: "{{ item }}"
        state: started
        enabled: yes
      loop:
        - ufw
        - fail2ban
        - nginx

  handlers:
    - name: Restart Fail2ban
      systemd:
        name: fail2ban
        state: restarted

    - name: Reload Nginx
      systemd:
        name: nginx
        state: reloaded

3. Running It from the Terminal
I validated the syntax and ran a dry-run check before executing:

# 1. Syntax check
ansible-playbook -i inventory.ini playbook.yml --syntax-check

# 2. Dry run simulation
ansible-playbook -i inventory.ini playbook.yml --check

# 3. Live execution
ansible-playbook -i inventory.ini playbook.yml

Live Terminal Run Output:
PLAY [Automate Linux Node Hardening and Nginx Proxy] ***************************

TASK [Gathering Facts] *********************************************************
ok: [node01]

TASK [Update apt cache and upgrade system packages] ****************************
changed: [node01]

TASK [Install baseline security and proxy packages] ****************************
changed: [node01]

TASK [Configure UFW default policies] ******************************************
ok: [node01] => (item={'direction': 'incoming', 'policy': 'deny'})
ok: [node01] => (item={'direction': 'outgoing', 'policy': 'allow'})

TASK [Allow essential inbound ports] *******************************************
changed: [node01] => (item=22)
changed: [node01] => (item=80)
changed: [node01] => (item=443)

TASK [Enable UFW service] ******************************************************
changed: [node01]

TASK [Deploy Fail2ban configuration] *******************************************
changed: [node01]

TASK [Deploy Nginx reverse proxy configuration] ********************************
changed: [node01]

TASK [Ensure baseline services are running and enabled at boot] ****************
ok: [node01] => (item=ufw)
ok: [node01] => (item=fail2ban)
ok: [node01] => (item=nginx)

RUNNING HANDLER [Restart Fail2ban] *********************************************
changed: [node01]

RUNNING HANDLER [Reload Nginx] *************************************************
changed: [node01]

PLAY RECAP *********************************************************************
node01                     : ok=10   changed=7    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

4. Live Verification & Testing
To confirm the automated configuration worked as expected, I performed three direct checks:
Test 1: Idempotency Check
I immediately re-ran the playbook against the same node:
ansible-playbook -i inventory.ini playbook.yml

Result: ok=8 changed=0 unreachable=0 failed=0
Ansible recognized that the target state already matched, making zero modifications and triggering zero service restarts.
Test 2: Firewall Verification
I verified that UFW active rules matched the configuration:

ssh ubuntu@192.168.1.50 "sudo ufw status verbose"

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere

Test 3: Health Check & Headers
I sent a test request to verify that the security headers and /healthz endpoint were live:

curl -i [http://192.168.1.50/healthz](http://192.168.1.50/healthz)

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: text/plain
Content-Length: 8
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff

healthy

Beyond docker build: What Enterprise-Grade Docker Actually Looks Like

Pawan Shinde — Thu, 01 Oct 2026 16:43:26 +0000

When most engineers learn Docker, the journey usually starts and ends on a local terminal: writing a basic Dockerfile, running docker build -t my-app ., and launching it with docker run -p 8080:8080

That works for a weekend project. But in an enterprise environment think platforms like Netflix, Uber, or a high-volume payment processing system you have 200+ engineers merging pull requests dozens of times a day. If every CI pipeline ran raw, unoptimized docker build commands on bare virtual machines, the development cycle would grind to a halt.

As a DevOps engineer, your job is not just to containerize applications... it is to design the automated, secure, reproducible pipeline that packages and delivers those containers at scale.

Here is a breakdown of the three production-grade Docker patterns every engineer should implement in CI/CD to eliminate build bottlenecks and avoid operational outages.

1. Remote Layer Caching with BuildKit: Dropping Builds from 18 Minutes to 45 Seconds.
The Problem
Imagine a developer changes a single line of code in a checkout service and opens a Pull Request. Modern CI/CD runners (like GitHub Actions, GitLab CI, or AWS CodeBuild) are ephemeral they spin up as completely blank VMs and are destroyed immediately after execution.

If your CI pipeline runs a basic docker build ., the new runner has no local cache. It must download the base OS, reinstall 400 dependencies, recompile C-extensions, and run tests from scratch. If an un-cached build takes 18 minutes, 50 daily builds burn hours of developer productivity.

The Fix: BuildKit + External Cache Backends
Modern Docker includes BuildKit, an execution backend that natively supports pluggable, external cache stores like Amazon ECR or JFrog Artifactory.

Instead of keeping cache layers tied to a single machine's local disk, BuildKit pushes cache metadata and layer diffs directly into your image registry. When a fresh CI runner kicks off, it pulls only the layers it needs.

docker buildx build \ --push \ --tag 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest \ --cache-to type=registry,ref=123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:cache,mode=max \ --cache-from type=registry,ref=123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:cache \ .

The Technical Nuance of Docker Caching
A common misconception is that "Docker downloads only the layer that changed." That is not how layer caching works under the hood:

Top-to-Bottom Evaluation: Docker reads your Dockerfile instructions sequentially from top to bottom.

Unchanged Steps are Cached: For layers that appear before the modified line (such as the base OS, system libraries, and pre-installed packages), BuildKit pulls those pre-computed layers from the remote registry instead of executing the commands.

Invalidation Downstream: Once an instruction changes (for example, modifying code invalidates COPY . .), that specific layer's cache is invalidated.

Execution, Not Download: Docker does not download the changed layer; it executes that command on the CI runner to build the new layer, alongside every subsequent step below it.

Setting mode=max in --cache-to ensures that BuildKit exports intermediate layers across all stages of a multi-stage build, not just the final target image.

2. Multi-Architecture Builds: One Tag for ARM64 and AMD64
The Problem: The Architecture Mismatch
Engineering hardware rarely matches cloud infrastructure:

Developers often work locally on Apple Silicon (ARM64).

Legacy Production Instances run on standard Intel Xeon or AMD EPYC processors (AMD64 / x86_64).

Modern Cloud Compute often runs on ARM-based chips, like AWS Graviton instances, which offer significantly better price-to-performance.

An application compiled for an ARM64 CPU cannot run natively on an AMD64 instruction set. If you ship an ARM64-compiled binary to an Intel host, the kernel will fail immediately:

Plaintext
exec format error (Exec format error: standard_init_linux.go:211)
Managing this with manual tags like app:v1.0-arm and app:v1.0-amd creates brittle deployment manifests and leads to human error.

The Fix: OCI Manifest Lists
The solution is not a single universal binary. Instead, modern registries use an umbrella catalog called an OCI Image Index (Manifest List).

                [ my-app:v1.0.0 ] 
                (OCI Manifest List)
                     /       \
                    /         \
                   ▼           ▼
         [ ARM64 Layers ]     [ AMD64 Layers ]
         • Apple Silicon      • Intel Xeon
         • AWS Graviton       • AMD EPYC

When building via docker buildx, your CI pipeline compiles binaries for both targets and bundles them under a single registry tag:

docker buildx build \ --platform linux/amd64,linux/arm64 \ --tag 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:v1.0.0 \ --push .

When an Intel server runs docker pull my-app:v1.0.0, the container runtime detects the host architecture and pulls the AMD64 layers. When a Graviton worker or local M-series Mac pulls that same tag, it downloads the ARM64 layers automatically.

3. Strict Tagging: Why :latest is Banned in Production
The Problem: The 2:00 AM Outage
The :latest tag is not a stable version—it is a floating pointer. Every time an image is built without a specified tag, Docker attaches :latest to it. If you build again tomorrow, the pointer shifts.

Consider this sequence:

At 1:55 AM, an engineer merges an update that builds and pushes checkout-api:latest.

At 2:00 AM, payment error alarms fire.

The on-call engineer inspects the failing pods: image: checkout-api:latest.

Because the tag gives no clue which commit was deployed, tracking down the culprit requires manually digging through logs. Worse, triggering a pod restart may not even pull the previous version if the node already has an image cached under the name :latest.

The Solution: Immutable Git SHA Tagging
Every container image promoted past local development should be strictly tagged with its corresponding Git commit hash (e.g., checkout-api:sha-a8f3b91).

This guarantees two non-negotiable operational properties:

Deterministic Traceability: If checkout-api:sha-a8f3b91 throws an exception, you can search for a8f3b91 in Git and immediately identify the exact diff that introduced the bug.

Instant Rollbacks: When an outage hits, you do not need to rebuild or recompile code. You roll back the running deployment directly to the previous stable SHA tag in seconds:

kubectl rollout undo deployment/checkout-api

Wrapping Up Moving from intermediate Docker usage to senior-level systems engineering is about shifting focus from running single containers locally to managing container lifecycles across distributed systems.

By implementing remote registry caching, configuring cross-architecture manifest lists, and locking down deployments with immutable SHA tagging, your CI/CD pipelines run faster, your fleet remains cost-effective, and your production deployments stay reliably recoverable.