After building a Docker image faster, I wanted to build it for the K8s cluster. Running the container on
the local machine isn't the same as running it on a cluster. I'm packaging a Go application in my example. But the same principles apply to any other language.
Starting Dockerfile
I'm starting with the following Dockerfile(Dockerfile_1):
ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder
WORKDIR /app
COPY go.mod go.sum /app/
RUN go mod download -x
COPY . /app/
RUN go build -o app
FROM debian:buster
WORKDIR /app
COPY --from=builder /app/app /app/
ENTRYPOINT [ "/app/app" ]
And I build it with the following command:
docker buildx build -t rnemet/echo:0.0.1 . -f Dockerfile_1 --cache-to type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:main --load
I'm using the --cache-to and --cache-from flags to cache the build process and the --load to load the image into the local Docker daemon. Also, I'm using BuildKit to build an image.
Let's see what I got:
❯ docker images rnemet/echo
REPOSITORY TAG IMAGE ID CREATED SIZE
rnemet/echo 0.0.1 6fc43a3d85eb 4 minutes ago 133MB
If I run the container as a Pod in a K8s cluster or a Docker container on my local machine:
❯ docker exec -it echo bash
root@30fa7aa78401:/app# whoami
root
I managed:
- to open a shell in the container
- found out that I'm running as root
- I run the
whoamicommand, which is not a command I need when I run the container
The whoami is not the problem, but the fact that it is present. I don't need it, and it's not part of my application. There must be others as well. I need to
clean up the image.
Then I learned that my app is run as root. This isn't good. My app needs to run as a non-root user. Why? There are many reasons. For example, the app can access the
whole system with root privileges. Do you remember? The container runtime isolates the process from the host system using namespaces. But the process can still access the host system.
As the cherry on top, I can open a shell in the container and run commands as root. I do not even want to discuss why this is not good.
Smaller, Rootless, and Non-Shell Image
The first thing I have to do is to change the target image. I'm using debian:buster as a target image. That needs to be changed. I decided to use scratch as a
target image. It's a minimal image. It's not a Linux distribution. It's not a shell. It's not a root user. It's a blank slate. So my next Dockerfile(Dockerfile_2):
ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder
WORKDIR /app
COPY go.mod go.sum /app/
RUN go mod download -x
COPY . /app/
RUN go build -o app
FROM scratch
WORKDIR /app
COPY --from=builder /app/app /app/
ENTRYPOINT [ "/app/app" ]
And build it:
docker buildx build -t rnemet/echo:0.0.2 . -f Dockerfile_2 --cache-to type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:main --load
And the result:
❯ docker images rnemet/echo
REPOSITORY TAG IMAGE ID CREATED SIZE
rnemet/echo 0.0.2 188f310d88e2 42 seconds ago 19.1MB
rnemet/echo 0.0.1 6fc43a3d85eb 59 minutes ago 133MB
Wow, the image is smaller. But it fails when I try to run it as a Pod in a K8s cluster or a Docker container on my local machine. Argh...
The problem is how I build the application. I'm building it on a particular Linux image with Go installed. That means all the libs and tools that Go needs to build and
run the application are part of the image. So, I must build the app with all its dependencies packed together. When it comes to Go, it's easy(Dockerfile_3):
...
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o app
...
I'm using the CGO_ENABLED=0 flag to disable using Cgo. I'm using the -ldflags="-s -w" flag to strip the debug information from the binary. And results are:
❯ docker images rnemet/echo
REPOSITORY TAG IMAGE ID CREATED SIZE
rnemet/echo 0.0.3 81e4098f0e76 15 seconds ago 13MB
rnemet/echo 0.0.2 188f310d88e2 25 minutes ago 19.1MB
rnemet/echo 0.0.1 6fc43a3d85eb About an hour ago 133MB
The image is smaller, and it works. But I'm still running it as root. I need to change that, too. But when working with scratch image, I need to create the user in
the builder image and then copy the /etc/password file to the scratch image(Dockerfile_3):
ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder
WORKDIR /app
COPY go.mod go.sum /app/
RUN go mod download -x
COPY . /app/
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o app
RUN useradd -u 10001 appuser
FROM scratch
WORKDIR /app
COPY --from=builder /etc/passwd /etc/passwd
USER appuser
COPY --chown=appuser:appuser --from=builder /app/app /app/
ENTRYPOINT [ "/app/app" ]
I'm running as a non-root user with no shell, and the image is smaller. And I should be happy, but no. Why? Let's go a few steps back.
Why Is Image Smaller?
In this case, the image is smaller because I'm using a scratch image as a target image, which produces an image size of around 19MB. Plus, I'm using stripping
debugging information from the binary. That's why the image is so small. That is nice, right?
Cgo Or No Cgo?
I'm using the CGO_ENABLED=0 flag to disable using Cgo. Why? Because I'm using a scratch image as a target image.
If I use debian:buster as a target image, I don't need to disable Cgo. Why? Because debian:buster image has all the libs and tools that Go needs to build and run
the application. So I don't need to pack all the libs and tools with the application. I can use the libs and tools that are part of the image.
The cgo tool is a tool that allows Go programs to call C code. It enables your Go app to call OS's native libraries dynamically. Using it leads to smaller and faster
builds. But, as you can see, it is not always possible to use it. So, in this case, I have to disable it and create a static binary. The static binary will be bigger
but will have all the libs the app needs to run. It will be more portable. It is common to disable cgo when creating multi-architecture images.
Unless your app depends on C libraries, you can generally disable cgo. Then you have to leave cgo enabled. It's always some trade-off.
Do I Always Need To Scratch When It Itches?
While the scratch image is minimal, it uses the root user, and I can not use the USER instruction to change it. I had to do a workaround. See Dockerfile_3. I had
to add the user to the builder image and then copy /etc/password file to the scratch image. Not a big deal, but it's a workaround. And I'm not too fond of
workarounds. I wonder if it is the only workaround I will do when using the scratch image.
Small detour. You need a base image that you can trust and use easily. When choosing a base image, you should trust the creator of the base image, and you don't want
to make any changes or workarounds. In most cases, you'll see Alpine Linux because it is small, and people trust it. But it has its own problems. My personal choice is Distroless images. They are small, secure, and easy to use. It is not as
small as a scratch image, but still small. And they are secure. They are based on Debian, and they use non-root user. So you don't need to do any workarounds.
Let's check it out. I'm using distroless/static-debian11:nonroot as a target image(Dockerfile_4):
ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder
WORKDIR /app
COPY go.mod go.sum /app/
RUN go mod download -x
COPY . /app/
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o app
FROM gcr.io/distroless/static-debian11:nonroot
WORKDIR /app
COPY --from=builder /app/app /app/
ENTRYPOINT [ "/app/app" ]
And results are:
❯ docker images rnemet/echo
REPOSITORY TAG IMAGE ID CREATED SIZE
rnemet/echo 0.0.4 f04110959f3c About a minute ago 15.5MB
rnemet/echo 0.0.3 0280e6cfb546 5 hours ago 13MB
rnemet/echo 0.0.2 188f310d88e2 6 hours ago 19.1MB
rnemet/echo 0.0.1 6fc43a3d85eb 7 hours ago 133MB
Using distroless/static-debian11:nonroot as a target image, I'm getting an image size of around 15.5MB. And I'm running as a non-root user. I do not have any
workarounds. As well, I'm having USER instruction. My Dockerfile is clean and easy to read. I like it.
One last try to use cgo in Dockerfile_5:
ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder
WORKDIR /app
COPY go.mod go.sum /app/
RUN go mod download -x
COPY . /app/
RUN go build -ldflags="-s -w" -o app
FROM gcr.io/distroless/base-debian11:nonroot
WORKDIR /app
COPY --from=builder /app/app /app/
ENTRYPOINT [ "/app/app" ]
After building it, the results are:
❯ docker images rnemet/echo
REPOSITORY TAG IMAGE ID CREATED SIZE
rnemet/echo 0.0.5 030cf87dd36e 3 minutes ago 33.5MB
rnemet/echo 0.0.4 f04110959f3c 15 minutes ago 15.5MB
rnemet/echo 0.0.3 0280e6cfb546 5 hours ago 13MB
rnemet/echo 0.0.2 188f310d88e2 6 hours ago 19.1MB
rnemet/echo 0.0.1 6fc43a3d85eb 7 hours ago 133MB
So, the smallest image is when we use a scratch image as a target image. But we have to do some workarounds. The second smallest image is when we use
distroless/static-debian11:nonroot as a target image. And we don't have to do any workarounds.
When you compare Distrolees images with a scratch image, you can see that the second is smaller. But, I trust Google and do not have to do any workarounds if I use
Distrolees images. In the case of the Go app, I had only one workaround. If I need to pack another app based on Java or NodeJS, I could do more work using a scratch
image. Distrolees images already come prepackaged with some tools. So I don't have to do any workarounds.
Base Image Conclusion
Let's summarize the results:
| image:tag | base image | size | non-user | no-shell | comments |
|---|---|---|---|---|---|
| rnemet/echo:0.0.1 | debian:buster | 33.5MB | no | no | - |
| rnemet/echo:0.0.2 | scratch | 19.1MB | no | no | do not work |
| rnemet/echo:0.0.3 | scratch | 13MB | yes | yes | static build, strip debug information, requires workaround for non-root user |
| rnemet/echo:0.0.4 | distroless/static-debian11:nonroot | 15.5MB | yes | yes | static build, strip debug information |
| rnemet/echo:0.0.5 | distroless/base-debian11:nonroot | 33.5MB | no | no | - |
In the end, I got what I wanted, a fast build, a small image, a non-root user, no shell access, and no workarounds. I'm happy with the results.
I'm not telling you to use Distroless images without any doubts. Maybe for your case, Alpine Linux](https://hub.docker.com/_/alpine) is better as it is tiny.
Or you prefer Wolfi. Or start from scratch. As you can see, there will be some trade-offs. You need to consider your apps needs, the effort willing to put into your Dockerfile and your trust in the creator of the base image.
Ah, one more thing please try to avoid the latest tags. Use versioned base images whenever possible. You'll have much more control and less headache.
Nice to Have: Labels
Add some metadata to your images. They can be handy later on. Like:
LABEL version="0.0.1-beta"
LABEL vendor="rnemet.dev"
LABEL release-date="2024-02-12"
You could also use ARG to set labels. Like:
ARG VERSION=0.0.1-beta
ARG VENDOR=rnemet.dev
ARG RELEASE_DATE=2024-02-12
LABEL version="0.0.1-beta"
LABEL vendor="rnemet.dev"
LABEL release-date="2024-02-12"
Conclusion
I hope you enjoyed this post and learned something new. If you found this helpful post, please share it with your friends. You can subscribe to my newsletter at the
bottom of this post and/or share this post. I would appreciate it.
If you like what I write check my newsletter and blog.
Top comments (0)