Art of building small containers

In this article, we will learn how to build small docker containers by understanding builder pattern and multistage builds in detail and go over what benefits they provide.

Spoilers we will reduce our golang container size from over 850mb to just under 12mb!

I've also made a video, if you'd like to follow along. Slides from the video can be found here

Problem

Docker images are often much bigger than they have to be which ends up impacting our deployments, security and dev experience.

Optimizing a build can be complex as it’s hard to keep your image clean and eventually, it gets messy, and hard to follow.

We also end up shipping unnecessary assets like tooling, dev dependencies, runtime or compiler in our releases.

Let's assume we have a simple hello world in Go and we'd like to dockerize it and deploy on Kubernetes

Here's our very simple hello world project

├── Dockerfile
└── main.go

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", helloHandler)
    http.ListenAndServe(":8080", nil)
}

func helloHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World!")
}

and our Dockerfile

FROM golang:1.16.5
WORKDIR /app
COPY main.go .
RUN go build main.go
CMD ./main

$ docker build -t default .
$ docker images

default     latest    afac261974d0   32 seconds ago     868MB

Woah, why is our hello world image over ~850mb!

Solutions

Let's look at some possible solutions for reducing our image size

Smaller base images

One simple solution is to just use a smaller base image when we are building our containers.

Example

GO

1.16.5              862MB
1.16.5-alpine       302MB  <---

Node

16                  907MB
16-slim             176MB
16-alpine           112MB  <---

To do this we can update the FROM statement in our Dockerfile

FROM golang:1.16-alpine

Builder pattern with multistage builds

Builder pattern simply describes a way to build your docker containers by splitting the build process into two or multiple stages to reduce any unnecessary assets from the production image.

The first image is a builder image, which basically builds our code by taking advantage of having all the necessary build utilities available.

The second image is our runtime or release image in which we will just copy over the built binary and deploy it, hence the size reduction!

Multistage builds just allow us to define all our stages in a single Dockerfile as opposed to splitting into multiple Dockerfiles like we had to do before multistage feature was available

Here's how it reflects in our Dockerfile

FROM golang:1.16.5 as builder
WORKDIR /app
COPY main.go .
RUN go build main.go

FROM alpine as production
COPY --from=builder /app/main .
EXPOSE 8080
CMD ./main

$ docker build -t multistage .
$ docker images

multistage     latest    iucs2934758r   18 seconds ago     12.5MB

Note: we can also use scratch or my new favourite distroless containers from Google

TLDR

• Derive from a base image with the whole runtime or SDK
• Copy our source code
• Install dependencies
• Produce build artifact
• Copy the built artifact to a much smaller release image

Benefits

Here are the benefits we get from building small containers

Performance

Some of the benefits of building and deploying small docker containers are:

• Faster push and pull from the container registry
• Small and optimized builds

Cost effective

We can now push our new docker image with a fraction of the cost and space required for the original.

Here's an interesting example from one from my client running around 18 microservices on Kubernetes

Default

18 microservices x ~800mb x 5 deploys cycles month x 12 months

~864GB/year

Optimized

18 microservices x ~25mb x 5 deploys cycles month x 12 months

~27GB/year

Security

Security is an essential part of any application especially if you're working in a highly regulated industry such as healthcare, finance, etc.

Smaller images reduces a lot of attack surface for vulnerabilities, here's a quick scan from AWS ECR

Next steps

Now we can deploy our tiny docker containers on Kubernetes/OpenShift fast and efficiently.

Feel free to reach out to me on Twitter if you have any questions.

Comments

[Thomas Nicollet]

You could use FROM scratch instead of FROM alpine as your base image since scratch doesn't add anything to your image size, and because go binary are self contained. Depending on your codex this can range from 1 to 25mb per image.

I also use UPX as a way to compress my binaries by getting rid of a lot of debug symbols and dead code in my binary

[Karan Pratap Singh]

yes, I've mentioned about scratch and distroless under Note: ..., didn't know about binary compression with UPX, thank you

[Sergey Kislyakov]

Your build instructions for Go does not produce a static linked binary, so if any of your dependencies use some system libraries (e.g. a wrapper library around ImageMagick) you'd have to install these libraries in your base image as well. I've done a bunch of scratch images, and I'd rather have some overhead of alpine (shell, package manager, etc.) because it's much easier to debug the app inside the container that way. And also I don't have to link everything statically, which will increase the complexity of the Dockerfile instructions. Alpine + deps is the way to go if you care about disk usage, but NOT that extremely.

[javad asadi]

Imagine a PaaS multi-tenant environment with single functions in their own containers

[naneri]

In this second stage of the Docker build:

FROM alpine as production
COPY --from=builder /app/main .
EXPOSE 8080
CMD ./main

Is the COPY line correct? Because I though the folder should be just /app as there is no /main folder inside it or am I wrong?

[Karan Pratap Singh]

Hey, so I am copying the binary called main from the build container main is not a dir