Trimming Container Bloat with Docker History and Dive

Why this matters: Large container images increase pull times during CI/CD, consume unnecessary storage in container registries, and expand the attack surface by including extraneous binary dependencies. Managing image size is a fundamental practice for maintaining lean, reproducible data pipelines that scale predictably across environments.

A Docker image is not a single file; it is a stack of filesystem changes. A common misconception is that deleting a file in a later RUN command removes it from the final image. It does not. The file persists in the layer where it was created, and subsequent layers simply mark it as "hidden" or "deleted," meaning your image size remains bloated by the original data.

If you are shipping data processing agents or heavy analytics environments, you are likely carrying "ghost" data—temporary caches, source code, or build artifacts—that provide no runtime value. Finding these hidden bytes requires more than just docker images. You need to peel back the layers.

Prerequisites

To follow this tutorial, you need the following installed:

• Docker Engine (version 20.10 or higher).
• Dive (a tool for exploring a docker image, layer by layer). See the official repository for installation instructions.
• A terminal environment with permissions to build images.

Photo by Sasun Bughdaryan on Unsplash

Step 1: Inspecting layers with docker history

Before using specialized tools, understand what Docker reports natively. The docker history command displays the commands used to create each layer and the resulting size of that layer.

Create a dummy Dockerfile to visualize the problem:

FROM python:3.11-slim
RUN apt-get update && apt-get install -y gcc g++
COPY ./data_processing_script.py /app/script.py
RUN pip install pandas==2.2.0
RUN rm -rf /root/.cache/pip

Build the image:
docker build -t bloat-test:latest .

Now, run the history command:
docker history --no-trunc bloat-test:latest

The output shows every instruction. Look for layers with large sizes. You will notice that the apt-get install layer is significant. Even though you might think you are cleaning up in a later layer, the initial download and extraction remain permanently in the image history.

Step 2: Using Dive to visualize the bloat

docker history is text-based and difficult to parse for large images. Dive provides an interactive TUI (Text User Interface) that allows you to navigate each layer and see exactly which files were added, modified, or removed.

Run Dive on your image:
dive bloat-test:latest

Navigate the interface using the arrow keys:

1. Left Panel: Lists the Docker layers.
2. Right Panel: Shows the contents of the filesystem for the selected layer.

Watch for the "Image Efficiency Score" in the top right. If you see a low score, it usually indicates that you are creating files in one layer and deleting them in another. Dive highlights these changes in red or green. If a file appears in a layer and is later deleted, it is still consuming space in your final build.

Step 3: Implementing multi-stage builds

The most effective way to eliminate bloat is by using multi-stage builds. This approach ensures that build-time dependencies (like compilers or header files) never make it into the final runtime image.

Modify your Dockerfile to use a build stage:

# Stage 1: Build dependencies
FROM python:3.11-slim AS builder
RUN apt-get update && apt-get install -y gcc g++
COPY requirements.txt .
RUN pip install --user -r requirements.txt

# Stage 2: Runtime image
FROM python:3.11-slim
WORKDIR /app
# Copy only the installed packages from the builder
COPY --from=builder /root/.local /root/.local
COPY . .
ENV PATH=/root/.local/bin:$PATH
CMD ["python", "script.py"]

Build the new image:
docker build -t lean-image:latest .

Run dive again on lean-image:latest. You will observe a significant reduction in the "Efficiency Score" and total size, as the gcc and g++ compilers were discarded at the end of the builder stage.

Step 4: Minimizing layers with build techniques

Every RUN command in a Dockerfile creates a new layer. You can reduce the number of layers by chaining commands. This is particularly useful for package management.

Instead of:

RUN apt-get update
RUN apt-get install -y git
RUN apt-get clean

Use:

RUN apt-get update && \
    apt-get install -y git && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Combining these into a single RUN instruction ensures that the temporary files created by apt-get are removed within the same layer, preventing them from being committed to the final image.

Photo by Anne Nygård on Unsplash

Common pitfalls

• The .dockerignore oversight: If you don't use a .dockerignore file, you are likely copying local .git directories, virtual environments, or __pycache__ folders into your image. Always add these to a .dockerignore file in your root directory to keep the build context clean.
• Over-reliance on apt-get clean: While cleaning package lists is good practice, it is secondary to avoiding the installation of unnecessary tools entirely. Always prefer distroless or alpine base images if your application dependencies allow it.
• Ignoring cache mounts: If you are using BuildKit (enabled by default in modern Docker), use --mount=type=cache for your package manager (e.g., pip or apt). This caches files during the build but does not include them in the resulting image layer, significantly speeding up subsequent builds without bloating the output.

Conclusion

Bloated containers are a silent tax on your data engineering velocity. By inspecting layers with dive and enforcing multi-stage builds, you ensure that your production artifacts contain only what is strictly necessary to execute your code. This not only speeds up deployment cycles but also improves the security posture of your infrastructure by reducing the number of binaries and libraries present in your environment.

Try it: Take a current production Dockerfile that you suspect is oversized, run dive on it today, and identify the single largest layer. Then, apply a multi-stage build pattern to isolate that layer's dependencies and compare the final image size.

Cover photo by Rinson Chory on Unsplash.

---

This article was drafted with AI assistance and is intended for technical review and fact-checking before publication.