DEV Community: Atharva Unde

Docker: Layers, Caching, Multi-Stage Explained

Atharva Unde — Sat, 15 Feb 2025 12:55:45 +0000

Docker's efficiency is one of its biggest draws. But what makes Docker builds so fast? The secret lies in its layer-based architecture and clever caching mechanism. Let's dive in and see how it all works.

Dockerfiles: A Layered Cake

Every line in your Dockerfile is an instruction, and Docker treats each of these instructions as a distinct layer. But what is a layer, exactly?

Think of it like this: a layer is an intermediate snapshot of your container image during the build process. Each instruction in the Dockerfile creates a new layer, building upon the previous one.

For instance, consider this common Node.js Dockerfile:

FROM node:18
WORKDIR /app
COPY package.json .
RUN npm install
COPY . .
CMD ["node", "server.js"]

This simple Dockerfile translates into five distinct layers:

Base Image Layer: FROM node:18 (The foundation upon which everything else is built)
Working Directory Layer: WORKDIR /app (Sets the working directory inside the container)
Dependency Definition Layer: COPY package.json . (Copies the package.json file)
Dependency Installation Layer: RUN npm install (Installs the project dependencies)
Application Code Layer: COPY . . (Copies the entire application code)

Each of these instructions results in a distinct layer that's stored in the image.

Docker's Caching Superpower

Here's where the magic happens: Docker caches each of these layers during the build process. This means that if a layer hasn't changed, Docker can reuse the cached version instead of rebuilding it from scratch. This dramatically speeds up subsequent builds.

Cache Hit: If an instruction and its inputs haven't changed, Docker pulls the existing layer from the cache.
Cache Miss: If an instruction or its inputs have changed, Docker invalidates the cache for that layer and all subsequent layers. This means it needs to rebuild not only the changed layer but also every layer that comes after it.

Cache Invalidation: When Things Go Wrong

The cache invalidation behavior is crucial to understand. Imagine you have a Dockerfile with eight instructions. If instruction #2 changes, Docker invalidates the cache for instruction #2 and all instructions that follow (3 through 8). They will all need to be rebuilt. This can lead to longer build times if not managed correctly.

A Real-World Example (Multi-Stage Build and Labels)

Let's examine a more complex scenario involving a multi-stage Dockerfile, which is a best practice for creating smaller and more secure images:

FROM node:20-alpine AS build-env
WORKDIR /app
COPY package.json yarn.lock ./
ENV NODE_ENV=production
RUN yarn install --frozen-lockfile --production
COPY index.js ./

FROM gcr.io/distroless/nodejs20-debian12
WORKDIR /app
LABEL org.opencontainers.image.authors="authoremail@example.com"
LABEL "com.example.vendor"="Example LLC"
LABEL version="1.0.0"
LABEL description="This image is used to run hello world backend written in Express Framework"
COPY --from=build-env /app /app
CMD ["index.js"]

In this Dockerfile, we have two stages:

build-env Stage: This stage uses a Node.js Alpine image to install dependencies and prepare the application for production.
Final Stage: This stage uses a distroless image (gcr.io/distroless/nodejs20-debian12), which contains only the necessary runtime dependencies.

Here's how caching works in this multi-stage context:

Independent Caches: Each stage has its own separate cache. Changes in one stage don't automatically invalidate the cache of other stages, unless they affect the COPY --from instruction (which we'll discuss below).
build-env Stage Changes: If you modify package.json or yarn.lock in the build-env stage, the RUN yarn install instruction will be invalidated, and all subsequent instructions in that stage will need to be rebuilt.
COPY --from Interaction: The COPY --from=build-env /app /app instruction is crucial. If the contents of /app in the build-env stage change (due to a rebuild triggered by a change in package.json, for example), the COPY instruction will also produce a different result in the final stage, invalidating the final stage's cache from that point onward.
Label Invalidation: The LABEL instructions, while important for adding metadata, do not directly influence the caching mechanism. Changing label values will always cause the layer containing the LABEL instruction to be rebuilt, but it doesn't impact any previous layers.
Code Changes: If you simply modify code in the index.js file, only the COPY index.js ./ instruction within build-env, and the subsequent COPY --from instruction in the final stage will be affected. The dependency installation stage (RUN yarn install) can still be pulled from the cache, speeding up the build significantly.

Docker Caching and Multi-Stage Builds: Scenario Table

This table outlines how different changes to your Dockerfile or application code impact the caching mechanism in a multi-stage build.

Scenario	Changed File/Instruction	Impact on `build-env` Stage Cache	Impact on Final Stage Cache	Rebuilt Layers
Dependency Change:	`package.json` or `yarn.lock`	`RUN yarn install` and subsequent instructions are invalidated.	`COPY --from=build-env /app /app` and subsequent instructions invalidated.	All layers from `RUN yarn install` in `build-env`, and from `COPY --from` in the final stage
Code Change Only:	`index.js`	Only `COPY index.js ./` is invalidated.	`COPY --from=build-env /app /app` and subsequent instructions invalidated.	`COPY index.js ./` in `build-env`, and from `COPY --from` in the final stage
Dockerfile Change (build-env, Before COPY package.json)`:	(e.g., adding a new `ENV` variable before COPY)	All instructions after and including the changed instruction are invalidated.	If the content of /app does not change, the final stage stays cached	All layers from that step to end of `build-env`
Dockerfile Change (build-env, After COPY package.json)	(e.g., adding an RUN after copy)	All instructions after and including the changed instruction are invalidated.	`COPY --from=build-env /app /app` and subsequent instructions invalidated.	All layers from changed instruction till end of `build-env` and onwards.
Label Value Change:	(Change in LABEL instruction in the final stage)	No impact.	Only the layer with the modified `LABEL` is invalidated.	Layer containing the `LABEL` instruction in the final stage
No Changes	N/A	All layers are pulled from cache.	All layers are pulled from cache.	None

Explanation:

Scenario: Describes the type of change made.
Changed File/Instruction: Specifies the file or instruction that was modified.
Impact on build-env Stage Cache: Explains which layers in the build-env stage are invalidated.
Impact on Final Stage Cache: Explains which layers in the final stage are invalidated.
Rebuilt Layers: Lists the layers that will be rebuilt during the Docker build process.

The Takeaway: Order and Multi-Stage Considerations

With multi-stage builds, you need to consider caching within each stage, as well as how changes in one stage affect subsequent stages through COPY --from instructions. Strategic placement of instructions and careful management of dependencies are key to maximizing build performance.

In the next section, we will explore best practices to optimize caching and reduce unnecessary rebuilds. Stay tuned!

Dockerfile Labels: A Comprehensive Guide

Atharva Unde — Sat, 08 Feb 2025 08:02:10 +0000

In our previous post, we explored how to significantly optimize Docker image size. Now, let's dive deeper and enhance our images with valuable metadata using Docker labels.

Remember this Dockerfile from last time?

FROM node:20-alpine AS build-env
WORKDIR /app

COPY package.json yarn.lock ./
ENV NODE_ENV=production
RUN yarn install --frozen-lockfile --production
COPY index.js ./

FROM gcr.io/distroless/nodejs20-debian12
WORKDIR /app

COPY --from=build-env /app /app
CMD ["index.js"]

If we inspect this image and check its labels using the command:

docker image inspect --format='{{json .Config.Labels}}' atharvaunde/dockerexamples:distroless

We'll see... null. So, what's the point of Docker labels, and why should we care?

Why Use Docker Labels?

Docker labels are essentially metadata tags that you can add to your Docker images. Think of them as key-value pairs that provide extra information about the image.

Use Cases for Docker Labels

Attribution: Who built this image? Include the author's name and email.
Organization: Which company or team created this image?
Version Tracking: What version of the application is contained within? This is especially useful if you consistently tag images as latest.
Description: Provide a short description of the image's purpose.
Custom Metadata: Add any other relevant information, such as dependencies, build dates, or license details.

By embedding this information directly into the image, you provide valuable context to users who consume your images.

Adding Labels to Our Dockerfile

Let's enhance our Dockerfile with labels to include the author, company, version, and a description:

FROM node:20-alpine AS build-env
WORKDIR /app
COPY package.json yarn.lock ./
ENV NODE_ENV=production
RUN yarn install --frozen-lockfile --production
COPY index.js ./

FROM gcr.io/distroless/nodejs20-debian12
WORKDIR /app
LABEL org.opencontainers.image.authors="authoremail@example.com"
LABEL "com.example.vendor"="Example LLC"
LABEL version="1.0.0"
LABEL description="This image is used to run hello world backend written in Express Framework"
COPY --from=build-env /app /app
CMD ["index.js"]

Now, when we run the same inspection command:

docker image inspect --format='{{json .Config.Labels}}' atharvaunde/dockerexamples:distroless

We'll see the labels we added:

{
  "com.example.vendor": "Example LLC",
  "description": "This image is used to run hello world backend written in Express Framework",
  "org.opencontainers.image.authors": "authoremail@example.com",
  "version": "1.0.0"
}

Important Placement in Multi-Stage Builds

When using multi-stage Dockerfiles like ours, it's crucial to place the LABEL instructions in the final stage (i.e., the one that creates the actual image you'll be distributing). If we had placed the LABEL instructions before the FROM gcr.io/distroless/nodejs20-debian12 line, they would be lost in the final image. The final image is created in a separate stage with separate context, so it wouldn't inherit labels from any previous stage.

Docker Label Inheritance and Overriding

This table summarizes how labels are handled when a Dockerfile adds a label that either exists or doesn't exist in the base image.

Scenario	Base Image Label	Dockerfile Label	Resulting Image Label
Inheritance: Label Exists in Base Image	Exists	Absent	Retained (from base)
Overriding: Label Exists in Base Image	Exists	Present	Overridden (Dockerfile value)
Addition: Label Not in Base Image	Absent	Present	Retained (Dockerfile value)

In Conclusion:

Docker labels are a simple yet powerful way to add metadata to your Docker images, providing valuable context and improving discoverability. Remember to place your LABEL instructions in the correct stage of a multi-stage Dockerfile and be aware of how label inheritance works. Happy containerizing!

Base Images: The Secret to Smaller Docker Images

Atharva Unde — Sun, 02 Feb 2025 12:30:00 +0000

In our previous post, we walked through creating a basic Dockerfile. However, we noticed a significant issue: the resulting image for our simple "Hello World" app was a hefty 1.62GB (uncompressed)! That’s not ideal for efficient deployment and resource utilization.

Today, we're diving into how to dramatically reduce your Docker image size by carefully choosing the right base image. You might be surprised by how much impact this single decision can have.

The Impact of Your Base Image

When you build a Docker image, you're essentially layering instructions on top of a foundational image – the base image. All your commands, dependencies, and application code get added on to this base. Consequently, the size and contents of your base image have a direct impact on the final size of your container image.

As an example, in our previous attempt, using FROM node:latest resulted in a 1.62GB (uncompressed) image. That's a lot of bloat for a tiny Node.js application!

Just by switching our base image to FROM node:alpine, we saw the size drop to 244MB (uncompressed). That's a huge improvement, but can we do better? Absolutely!

Understanding Different Base Image Types

Let's explore the common types of base images and when to consider using them:

Standard Images: These are the full-fledged OS-based images like Ubuntu, Debian, or others. They come packed with a wide range of libraries and tools. While convenient, these images tend to be large due to all the extra baggage they carry. Unless you're unsure about your application's OS dependencies or are in a real rush, it's best to avoid them for production containers due to their size and resource consumption.
Alpine Images: These images are based on the super lightweight Alpine Linux distribution. They are much smaller than standard images because they contain only the bare minimum packages needed to run your application. They are ideal for most use-cases and are one of the best choices when starting out with Docker optimization. However, be sure to test your application thoroughly when first switching to Alpine images, as they might lack OS-level dependencies that your application unexpectedly relies on.
Slim Images: While the name might suggest otherwise, slim images can sometimes be larger than Alpine images, but still much smaller than standard ones. They often include only the packages and dependencies required to run specific applications, so it's worth exploring these images if their package set fits your use case. They can be based on various distributions like Alpine, CentOS, or Debian.
Distroless Images: These are specially designed for multi-stage Docker builds. They contain only your application and its runtime dependencies, excluding package managers, shells, and other common Linux utilities. This makes them incredibly small and helps improve the security posture of your containers. Distroless images are perfect for production deployments after you understand the entire application stack and dependencies.

As Google, the creator of distroless images, puts it, "Distroless images contain only your application and its runtime dependencies." You can read more about them here.

Choosing the Right Image: A Practical Approach

Deciding which image to use might seem daunting, but a simple approach helps here:

Start with Alpine: Begin with an Alpine-based image like node:alpine for Node.js applications.
Inspect and Verify: Examine the image's layers and included packages on Docker Hub. This can give you insights into the image's composition. For example, you can check the layers of a specific image tag like node:current-alpine3.20 here.
Add Dependencies Manually: If an Alpine image lacks required dependencies, you can install them manually within your Dockerfile. You can even create your own custom base image from scratch if needed.
Advance to Distroless: For production, after thorough testing, and when the full application dependencies are well-known, consider using distroless images for maximum size reduction and security.

Let's Compare Image Sizes

To illustrate the point, let's take a look at the image sizes we saw in our example, using different base images. We'll show both uncompressed and compressed sizes for comparison:

Base Image & Build Setup	Image Tag	Uncompressed Docker Image	Compressed Docker Image
`node:20-alpine` (build) & `gcr.io/distroless/nodejs20-debian12`	`mycontainer:distroless`	191 MB	49.61 MB
`node:slim`	`mycontainer:slim`	364 MB	78.52 MB
`node:alpine`	`mycontainer:alpine`	244 MB	56.68 MB
`node:latest`	`mycontainer:default`	1.62 GB	381.73 MB

Note: The uncompressed size can be checked by using the command docker image ls | grep mycontainer. The compressed size can be seen when the image is pushed to a container registry or by using docker save mycontainer:distroless | gzip -c | wc -c to see the compressed file size in bytes.

Whats the difference in compressed and uncompressed size?

Optimization through Distroless (Multi-stage Build)

Now, let's see how to optimize it even further using a Distroless image. Here's an updated Dockerfile:

FROM node:20-alpine AS build-env
WORKDIR /app

COPY package.json yarn.lock ./
ENV NODE_ENV=production
RUN yarn install --frozen-lockfile --production
COPY index.js ./

FROM gcr.io/distroless/nodejs20-debian12
WORKDIR /app

COPY --from=build-env /app /app
CMD ["index.js"]

In this approach:

We use node:20-alpine as a builder image to install our dependencies, copy the source, and prepare the application for production.
We then copy all necessary artifacts (/app) into a distroless image gcr.io/distroless/nodejs20-debian12 which only contains the absolute runtime requirements.

This method creates a final image with a minimal footprint, as demonstrated in the table.

Conclusion

Choosing the right base image is a critical step in optimizing your Docker images. Start with Alpine images, manually add needed dependencies, and ultimately aim for distroless images in production. By understanding the trade-offs of each image type, you can greatly reduce your image size, improving efficiency, performance, and resource consumption. Stay tuned for more Docker tips and tricks in future blog posts!

The table clearly highlights the dramatic size difference when using different base images and build strategies. The multi-stage distroless approach yields the smallest final image, making it ideal for production deployments.

Understanding the Difference: Compressed vs. Uncompressed Docker Image Sizes

Atharva Unde — Sat, 01 Feb 2025 13:40:36 +0000

When you build a Docker image, you're essentially creating a series of layers, each representing a step or instruction in your Dockerfile. These layers build on top of each other, forming the final image. Let's break down what the compressed and uncompressed sizes mean in this context:

Uncompressed Image Size

What it is: The uncompressed size represents the total size of all the layers in your Docker image as they exist on your local machine. This includes all the intermediate layers created during the build process, as well as any duplicate data that may exist across layers. Think of it as the raw, unoptimized size of your image.
Why it's larger: This size is usually much larger because:
- Intermediate Layers: Docker caches intermediate layers during the build process. These layers are kept for efficiency if you rebuild the image later, but they all contribute to the total uncompressed size.
- Duplicate Data: When you copy files into your image, Docker might create new layers even if some data is repeated from previous layers. For example, if you copy a directory in an early step and copy it again later with minor changes.
- Unoptimized Storage: The uncompressed size doesn't take advantage of compression or any optimized storage techniques.

Compressed Image Size

What it is: The compressed size is the size of your Docker image after it has been processed and optimized by Docker for storage and transfer. This is the size you'll see when the image is pushed to a registry like Docker Hub or when you manually save it using docker save with compression.
How it's smaller: Docker applies several optimizations during this process:
- Gzip Compression: The most significant optimization is that Docker uses gzip to compress the individual layers of your image. This significantly reduces the amount of storage space required.
- Layer Deduplication: Docker identifies and removes duplicate data across layers. If two layers contain the same file, only one copy of the data will be stored, and layers can reference that single source.
- Optimized Storage: Docker takes advantage of optimized storage mechanisms which only takes up space required for differences between layers and ensures it is stored efficiently.

Still Confused?

Imagine you're packing for an outing!

Uncompressed: This is like throwing all your clothes, toiletries, and shoes into a big suitcase without any organization. It takes up a lot of space.
Compressed: This is like carefully rolling up your clothes, using packing cubes to organize items, and removing any unnecessary duplicate items. The whole suitcase is now much smaller.

TL DR;

Local vs. Registry: The uncompressed size is what you see locally while the compressed size is what's stored and transferred to/from a container registry.
Transfer Efficiency: Docker optimizes images during upload to a container registry, and because of compression, downloads will also take less time.
True Size: The compressed size gives you a more accurate idea of the actual space your image will occupy on a registry or when you save it as a compressed archive.
Size Optimization Goal: When trying to optimize your docker image size, the goal is to minimize the compressed size, because that affects download and upload times, and the cost of storage in the registry.

In Summary:

The uncompressed size is a useful metric to observe the effects of your changes during the image building process. However, the compressed size is the true indicator of how large your image is when stored and transferred. It's this compressed size that you should focus on when optimizing your Docker images for better performance and efficiency.

So how to check the size?

You can check the uncompressed size by running docker image ls | grep <your_image_name> command. To check the compressed size you can either push it to DockerHub, or use command docker save <your_image_name> | gzip -c | wc -c to see compressed size in bytes.

Still confused or having doubts?
Write down in comments section!

Excited to share my first blog post: "Practical Docker: Step-by-Step Container Creation and Execution"! In this guide, I walk through the essential steps of building and running a simple Docker container, with a focus on practical commands and concepts.

Atharva Unde — Sat, 01 Feb 2025 11:27:11 +0000

Practical Docker: Step-by-Step Container Creation and Execution

Atharva Unde ・ Feb 1

#devops #docker #dockerforbeginners #javascript

Practical Docker: Step-by-Step Container Creation and Execution

Atharva Unde — Sat, 01 Feb 2025 11:18:30 +0000

Running your software in containers, whether it's a website, an app, or even your database, is becoming increasingly important. Docker is a powerful tool for this, and a crucial part of it is the Dockerfile.

Think of a Dockerfile as a recipe for building a Docker image. It's a simple text file that lists all the steps needed to create a complete, self-contained package (the image) of your application. This includes things like installing necessary software, setting up your environment, and copying over your code. Docker reads this recipe and automatically builds the image, making sure everything is prepared exactly the way you want it.

Why all this effort?
Repeatable results are critical in DevOps. A Dockerfile ensures that every time you build your image, you get the exact same, reliable outcome.
It eliminates the "it works on my machine" problem by providing a consistent environment.

Let's get started with the technical details of creating a Dockerfile. This will give you the foundation for building your own containerized applications.

FROM node:latest
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["node", "index.js"]

This example showcases a fundamental Dockerfile structure. In future articles, we'll explore more techniques for writing Dockerfiles that improve efficiency, security, and maintainability.

Breakdown of the Dockerfile

FROM node:latest
This specifies the base image to use. node:latest pulls the latest official Node.js image from Docker Hub. This image already has Node.js and npm pre-installed.

WORKDIR /app
This sets the working directory inside the container to /app. This is crucial for organization and makes the Dockerfile more readable. Subsequent instructions will operate within this directory.

COPY package*.json ./
This copies the package.json and package-lock.json files from the build context (your local project directory) into the /app directory inside the container.

RUN yarn
This crucial step runs the yarn command inside the container. It installs all the dependencies listed in package.json. This ensures the container has all the necessary packages.

COPY . .
This copies all the remaining files and directories from the build context to the /app directory. This effectively copies the entire application code to the container.

EXPOSE 3000
This declares that the container will listen on port 3000. While it's important for visibility, it does not automatically map this port to your host.

CMD ["node", "index.js"]
This is the command that runs when the container starts. It executes the Node.js file index.js. This is how your application is launched within the container. There can be only be one CMD instruction in a Dockerfile.

Now that we've created the Dockerfile, let's build the actual container image. This involves using the docker build command.

To build your image, open your terminal and navigate to the directory containing your Dockerfile and the application code. Then, run the following command.

docker build -t mycontainer:latest .

This builds a docker image with the name as myContainer and gets tagged as latest. . denotes that Dockerfile is supposed to take the current directory from which the command is being run as the build context for source code and other files.

If the build process is successful, you'll see output showing the image ID. You can verify the image was built by running

docker image ls
REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
mycontainer   latest    e50c98928825   7 seconds ago   1.62GB

Let's worry about the size of the image in the next chapter of the blog, where we will discuss how to write an optimized Dockerfile

Lets run the image and see if the container is responding to our requests on http://localhost:3000

docker run -d -p 3000:3000 --name myapp mycontainer:latest

This command creates a detached container named myapp from the mycontainer image using the latest tag, maps port 3000 on the host to port 3000 inside the container, and starts the application defined in your CMD instruction inside the container.

After running this command, you can access your application by navigating to http://localhost:3000 in your browser on the host machine!

Source Code: GitHub

Do let me know in comments in case you face any issues,