Why your Docker Build time is more than it should ?

If you have ever worked with Docker & written any Dockerfile for any backend service that includes some language & its packages & an image. chances are you have faced this problem of high Dockerfile build time already.

You might not have noticed it because for a self-paced project, these delays are in seconds which don't matter much to us. But when our hands are messing with 100s of containers using kubernets pods. These delay becomes the reason of large containers which takes more time to rebuild.

Hence it becomes necessary to grab the depth of this Idea early stage.

Today we will see with live example from one of my production style project where I tried to containerize a Go service. & understand ->

• why certain steps come in top & why some in bottom of the Dockerfile

• Difference between a Good Order & Bad Order Dockerfile

• Thumb Rule of Writing efficient Dockerfiles.

Setup

1) go service from otel-demo (an open source project)

2) Two Dockerfiles that differ by three moved lines

---

Dockerfile_v1(Good Order)

In this Docker file, {1,2,3} lines are important libraries & checksum files needed to run any go file. This step of downloading the library into the working directory of our image is slow step & takes longer time than the rest of the layers.

Therefore, we have something called Layer caching in Docker. In the first cold build, every step runs one time, but in the next run, every step or layer that is unchanged gets cached, & docker uses that cached data instead of rebuilding the whole layer.

If that layer is changed, then every layer following that layer gets rebuilt, including itself.

FROM golang:1.27.1-bookworm AS builder
WORKDIR /app

COPY go.mod go.mod   (1)
COPY go.sum go.sum   (2)
RUN go mod download  (3)
 # ~ 30 sec build time, slow step:downloads-libraries

COPY genproto/oteldemo genproto/oteldemo  (4)
 # source code comes AFTER the download
COPY flags flags                          (5)    
COPY *.go ./                              (6)

RUN CGO_ENABLED=0 GOOS=linux go build -o /plain-linux-binary
EXPOSE 3550
CMD ["/plain-linux-binary"]

If you don't have any Idea what this code block is about, Don't worry - today we are not here to understand what these statement means but to understand what do we get to see if we change the order of these Instructions.

Are you ready :)

The Experiment

In this experiment we would change the order of good order dockerfile & will see what change we get to see in our build time. Using those numbers we would discern & find out how should we optimize our Dockerfile for minimum build time.

we are going to swap lines {4,5,6} which copy and downloads the source code and put them just above the {3} & will Observe the changes

Dockerfile__v2(Bad Order):
FROM golang:1.27.1-bookworm AS builder
WORKDIR /app

COPY go.mod go.mod   (1)
COPY go.sum go.sum   (2)
COPY genproto/oteldemo genproto/oteldemo  (4)
 # source code comes Before the download
COPY flags flags                          (5)    
COPY *.go ./                              (6)
RUN go mod download  (3)
 # slow step:downloads-libraries

RUN CGO_ENABLED=0 GOOS=linux go build -o /plain-linux-binary
EXPOSE 3550
CMD ["/plain-linux-binary"]

what's happening under the hood:

In Good order ->

go.mod is copied and stored in /app work directory of an image by named as go.mod

go.sum is copied and stored in /app work directory of an image by named as go.sum

run go mod download downloads library into the image.

Then {4,5,6} steps are there to copy & download the source code.

Time it took for the Good Order build:

1) Cold : 102.6 seconds ; as everything ran for the first time.

time docker build --no-cache --progress=plain -f docker/product-catalog/Dockerfile -t fincomm/product-catalog:dev application/product-catalog  

2) Unchange: 3.8 seconds; as nothing in the code changed, so every layer is cached and got reused.

time docker build --progress=plain -f docker/product-catalog/Dockerfile -t fincomm/product-catalog:dev application/product-catalog

3) Change: 28 seconds; Now we change the source code a little by adding a comment in a main.go file in the last . so for the good order, the only step which re-ran were {4,,5,6} because they were copying & downloading source code into our image work directory.

echo "// cache test" >> application/product-catalog/main.go

time docker build --progress=plain -f docker/product-catalog/Dockerfile -t fincomm/product-catalog:dev application/product-catalog

Time it took for the Bad Order build:

Swapped: 55 second.

Noticed, How swapped version of bad order took more time than changed version of good order ? Any Idea why is that ? Try to build your own hypothesis & that way you can see if we match :)

Apparently that happened because of just one mistake we did - shifted the run go mod download after source code build.

There is a Thumb rule of a Dockerfile. It says something like

Any layer which is changed, all layers following it including itself will get rebuilt. Docker would cached invalidate every layer after it including that layer itself. And that's why we place layers that rarely change (e.g., downloading dependencies/libraries) near the top (beginning) of the Dockerfile, and layers that change frequently (e.g., application source code) near the bottom (end) of the Dockerfile.

Now what happened in our bad order dockerfile:

Docker invalidates the cache of "run go mod download" layer as it comes after the source code which got changed and rebuild & so as a default rule, this layer will get rebuild too.

And this will increase the build time from what the good order Dockerfile normally take.

What you have to do right now:

• You have to Interact with this live artifact to solidify your learning

• Draft a summary of what you have understand from this article & comment it under this Article & I will pinned down the best summary.

Immediately try to replicate this experiment with any Dockerfile for any language you could get.

you just need to change order of source code & libraries or package to replicate the result, you will have your own numbers & your own experiment :) Use "time docker build ..."

Banner Image Credit: Gemini