Whether you are a software developer, a devops engineer or a linux sys admin, everyone has seen how ubiquitous the containers have become. There is no denying the fact that they are quite convenient to run the applications. You have your application ready, all you need is write a neat little Dockerfile and run the docker build command and there you have your application containerized. Declarative nature of writing a Dockerfile is so easy that we often dont even think about optmising it and the end result is ridiculously larger image size. Although there are a ton of ways to analyse your dockerfile and optimise it but in this post today I would like to focus on one specific scenario that I recently faced in my organisation.
An example setup
As this blog post that I am writing is outside the organisation I am working with so I will try to imagine a situation which is close to the scenario I faced within my org and try to build up a solution from there.
Lets say we have the below dockerfile with us:
FROM ubuntu:latest
RUN apt update && apt install make
RUN mkdir /workdir
WORKDIR /workdir
COPY Makefile .
RUN make
RUN chmod 755 mybinary
RUN mv mybinary /usr/bin/mybinary
What this will do is compile an application of size 2GB and then install it to the /usr/bin. Lets trigger the build of this image and see what the image size is after the end.
satyam@wakanda ~/demo $ docker build -t mydemo:latest .
[+] Building 9.1s (13/13) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 215B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:latest 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/8] FROM docker.io/library/ubuntu:latest 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 30B 0.0s
=> CACHED [2/8] RUN apt update && apt install make 0.0s
=> CACHED [3/8] RUN mkdir /workdir 0.0s
=> CACHED [4/8] WORKDIR /workdir 0.0s
=> CACHED [5/8] COPY Makefile . 0.0s
=> CACHED [6/8] RUN make 0.0s
=> CACHED [7/8] RUN chmod 755 mybinary 0.0s
=> [8/8] RUN mv mybinary /usr/bin/mybinary 2.0s
=> exporting to image 7.0s
=> => exporting layers 7.0s
=> => writing image sha256:0047f0b0f1bb5e2ffbf178f5f0cddbad64aaa71d37d28eb0c6c0aecf1def9963 0.0s
=> => naming to docker.io/library/mydemo:latest 0.0s
satyam@wakanda ~/demo $ docker image ls mydemo:latest
REPOSITORY TAG IMAGE ID CREATED SIZE
mydemo latest 0047f0b0f1bb 4 minutes ago 6.57GB
What!!!
How did the image end up being 6.5 GB when the application is just 2GB !!
Where it all went wrong ?
Now surely if my container image is ending up being 3 times my application size then I am not doing a very good job at containerizing it. Large containers take large disk space and are time taking to build and export.
So how can we analyse how our image size ballooned to this size?
We will use an open source tool called dive.
Let's pass our image tag as input to this command
satyam@wakanda ~demo $ dive mydemo:latest
Image Source: docker://mydemo:latest
Extracting image from docker-engine... (this can take a while for large images)
Analyzing image...
This will take some time to run and then you will be presented with a screen containing very detailed view of each layer of the image and also the filesystem view of the last layer. We will focus on the layer view. This is what you are expected to get:
In the output we can clearly see how much size each of our dockerfile commands have contributed to the overall image size. In Dockerfile, each dockerfile command you write corresponds to one layer of container image and each new layer is added upon the previous layer. Thus at the end the container image is the sum of all the layers that was created by each of the dockerfile commands.
In the image above, we can clearly see that the last three layers are the ones adding to the bulk of image size. So lets start by analysing each of them one by one.
1. The third last layer
2.1 GB RUN /bin/sh -c make # buildkit
In this output we can see that our RUN make command created our application of 2.1GB in size which is ok because we know that the app was supposed to be 2GB. So far so good !!
2. The second last layer - The first mistake
2.1 GB RUN /bin/sh -c chmod 755 mybinary # buildkit
Here we ran chmod 755 mybinary to modify the permission of our binary file. Now on its own this looks like a very innocent command so why is this creating an image layer equivalent to the previous layer in size?
The answer lies in the docker's copy on write approach of creating the image layer. When the container is being built, each of the layers that are created are immutable. They can never be changed by the commands of the next layer. So when we do RUN chmod ..., this will copy the binary onto the new layer and there it will modify its permissions. Hence this simple chmod ends up contributing a size equal to its previous layer. Now this make up total size of ~4GB. What about the rest 2GB?
3. The final layer - The second mistake
2.1 GB RUN /bin/sh -c mv mybinary /usr/bin/mybinary # buildkit
Here we ran mv mybinary /usr/bin/mybinary. Now again a very innocent command. Technically we are just moving files so this should not be adding to the size. But remember our observation for the previous layer. Every previously built layer is immutable. So even though we are using an mv commad, docker cannot perform this operation on the previous layer. What it will end up doing is again copy the file from previous layer to the new layer and then perform the mv operation here. Thus it will again end up adding 2GB.
Hence we have now 6Gb of the image...
How could this have been avoided ?
We can easily fix this problem by using docker's multi stage build. What this means is that we build out the application and fix its permission in one stage and copy the final application onto the next stage. The layers of one stage are not added to the layers of the other stage.
Here is our fixed dockerfile.
# Stage 1
FROM ubuntu:latest
RUN apt update && apt install make
RUN mkdir /workdir
WORKDIR /workdir
COPY Makefile .
RUN make
RUN chmod 755 mybinary
# Stage 2
FROM ubuntu:latest
COPY --from=0 /workdir/mybinary /usr/bin/mybinary
Each stage in a multi stage build is identified by a seperate FROM command. In the first stage, we build out the application and fix the permissions of the application and then in the second stage, we copy the final application from the first stage to the required destination in the final image.
Lets build the image and see the image size now.
satyam@wakanda ~demo $ docker build -t mydemo:fixed .
[+] Building 10.6s (13/13) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 251B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:latest 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [stage-0 1/7] FROM docker.io/library/ubuntu:latest 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 30B 0.0s
=> CACHED [stage-0 2/7] RUN apt update && apt install make 0.0s
=> CACHED [stage-0 3/7] RUN mkdir /workdir 0.0s
=> CACHED [stage-0 4/7] WORKDIR /workdir 0.0s
=> CACHED [stage-0 5/7] COPY Makefile . 0.0s
=> CACHED [stage-0 6/7] RUN make 0.0s
=> CACHED [stage-0 7/7] RUN chmod 755 mybinary 0.0s
=> [stage-1 2/2] COPY --from=0 /workdir/mybinary /usr/bin/mybinary 2.1s
=> exporting to image 7.0s
=> => exporting layers 7.0s
=> => writing image sha256:ec007c5a4c19fd0481b7f41fff27024a3ac25ba0e9e41f25a7fa70e2cae98d46 0.0s
=> => naming to docker.io/library/mydemo:fixed
satyam@wakanda ~demo $ docker image ls mydemo:fixed
REPOSITORY TAG IMAGE ID CREATED SIZE
mydemo fixed ec007c5a4c19 16 seconds ago 2.23GB
Behold!! we reduced our image size from 6GB to ~2GB. We can also visualize it with dive tool
Just two layers!!! Remember I mentioned that in multi stage builds, the layers from previous stage are not added to the final stage. Since in the second stage all we did was copy the application from first stage so we only have two layers and both of them are of expected size.
So this is how we used dive tool to analyse our image layers and used multi stage builds to reduce the image size.
Let me know your thoughts on this in the comments below and how you approached and optimised the image size in your projects.
For more reading:
Multi Stage Builds
Dive tool
Top comments (0)