DEV Community: Martin Wimpress

Creating Production-Ready Containers - Advanced Techniques

Martin Wimpress — Fri, 18 Jun 2021 14:00:05 +0000

Creating production-ready containers for use in commercial-grade apps can be a far cry from the "get started with Node.js and Docker"-type of tutorials that are common around the Internet. Those guides are great for introducing all the advantages of Docker containers in modern cloud-native development, but creating a container that passes muster in a large-scale application in production is a different story.

For production-ready containers, there are three key things you want to optimise for when creating a container:

Image Size 📦
Build Speed 🐢
Security 🔐

Image size and build speed ensure that your containers can move through CI/CD and test pipelines easily and efficiently. Security is obviously critical in today's software supply chain, and containers have their own set of security issues. Thankfully, reducing container image size actually can alleviate some security issues in containers.

In my Basics article, I showed you some easy techniques to improve your Dockerfile using a sample "Hello World" Node.js application.

Creating Production-Ready Containers - The Basics

Martin Wimpress ・ Jun 3 '21

#docker #beginners #node #devops

These basics address all three optimisations, though they only scratch the surface.

Let's look at some more advanced techniques for Container Optimisation.

Keen readers who make it to the end of the article will unlock a bonus! 🎁

Alpine Images

The very first thing you'll encounter when looking for techniques to create smaller containers is Alpine Linux. Alpine Linux is an open source project whose goal is to create a bare-bones 🦴 version of Linux that lets developers "build from the ground up."

Pros: Transitioning to Alpine can be an easy way to get a smaller container

Reducing image size with Alpine can be incredibly simple - under the right circumstances. For some apps, it's as easy as changing the base image in your Dockerfile:

FROM

FROM node:16.2.0

TO

FROM node:16.2.0-alpine

When we build the new image, we see that the old image was 856MB and the new one is 114MB 🎉

REPOSITORY          TAG     IMAGE ID        CREATED         SIZE
cotw-node-alpine    latest  2cc7b4a7b09c    2 minutes ago   114MB
cotw-node           latest  873fb9fca53a    3 days ago      856MB

Easy, right? Not so fast.

Cons: Using Alpine images can lead to build problems, now and in the future

There are some not so obvious gotchas with using Alpine images that don't crop up in our super simple example application, such as:

You have to install everything

Those tiny base images have to sacrifice something, right? Alpine users will be installing everything they need, right down to time-zone data or development tools. You won't need your development tools for your production image, most likely, but for most developers, the thought of a server without curl or vim is a bridge too far.

Different compilers and package managers

You'll also be installing any dependencies with the Alpine Package Keeper tool (apk) instead of the more familiar apt or rpm. The differences are small, but can trip up unsuspecting developers.

Fewer examples; less documentation

Finally, while Alpine has been around for nine-plus years, it is and likely always will be a smaller and more specialised user base than established Linux distributions such as Ubuntu and Debian. To wit, at the time of this writing the alpine tag on StackOverflow has just 1,280 questions, compared with over 54,000 for Ubuntu.

Multi-stage builds

The next tactic you are likely to encounter when searching for ways to reduce Docker image sizes is multi-stage 🏗 builds. This tactic, recommended by Docker and many in the Docker community, is essentially building the image twice. The first set of commands builds your base application image, all things included. The second set of commands builds an image off of that base image, taking only what's needed and leaving out anything that's not.

With a multi-stage build, our Dockerfile would look like this. Notice the two FROM statements. The first builds the application image; the second copies the necessary files from that image into the second, more production-ready version.

FROM node:16.2.0-alpine as builder
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm ci
COPY app.js ./

FROM node:16.2.0-alpine
WORKDIR /usr/src/app
COPY --from=builder /usr/src/app .
EXPOSE 3000
USER node
CMD ["node","app.js"]

Pros: Dev and Prod images can be built separately

When combined with Docker Compose, this approach gives developers a flexible development environment while reducing bloat in the production images. You can simply use your initial image for dev/test and the final version for productions. Multi-stage builds work especially well for Go containers, significantly reducing image size, but also work well for static Node.js and React-type applications.

Cons: Added complexity; use-case specific

Multi-stage builds are still relatively new 🌱 on the scene. For most developers still new to containers, knowing what to copy over to the final production image and what to leave behind is a major barrier to entry. Further, this pattern can run into challenges.

Since we're already using an Alpine image, the size savings are relatively minor for our "Hello World" example. You'd expect to see greater gains in a full-blown React or Vue application.

REPOSITORY            TAG     IMAGE ID      CREATED        SIZE
cotw-node-multistage  latest  52bc33d14a87  3 minutes ago  114MB
cotw-node-alpine      latest  2cc7b4a7b09c  4 days ago     114MB
cotw-node             latest  873fb9fca53a  7 days ago     856MB

Development tools and Distroless

There are several tools - and new ones emerging every day - that look to bypass or automate Dockerfile authoring to make image creation easier. Buildpacks are the most mature of these technologies, and can be used through tools like Pack or Waypoint.

There are builder options from multiple sources - Heroku, Google, and Paketo are common favourites - and each gives you a slightly different developer experience and final image when used.

$ pack build cotw-node-bp-google --builder gcr.io/buildpacks/builder:v1
$ pack build cotw-node-bp-heroku --builder heroku/buildpacks:18
$ pack build cotw-node-bp-pb-base --builder paketobuildpacks/builder:base
$ pack build cotw-node-bp-pb-full --builder paketobuildpacks/builder:full

Pros: When they work, they work

In certain instances, Buildpacks can take the pain out of Dockerfile authoring and just create container images of your application with no fuss. The pack tool is looking for "app-like" files in your source directory, and automatically figuring out what kind of application is there and how to containerize it. In the case of our Node sample, it sees package.json and correctly assumes we have a Node.js application.

Cons: When they don't…

Given the relative newness of this approach for Docker containers, there are a lot of gotchas with Buildpacks. Non-standard applications or operating systems can struggle, and we've had issues running them successfully on the new Silicon Macbook Pros. The resulting images vary a lot - we saw a range of 200MB to 800MB in our examples - and the results tend to be lower than what you'd get with other techniques.

Automate it with DockerSlim

The DockerSlim open source project was created by Slim.AI CTO Kyle Quest in 2015 as a way to automate container optimisation.

slimtoolkit / slim

Slim(toolkit): Don't change anything in your container image and minify it by up to 30x (and for compiled languages even more) making it secure too! (free and open source)

Optimize Your Experience with Containers. Make Your Containers Better, Smaller, More Secure and Do Less to Get There (free and open source!)

Note that DockerSlim is now just Slim (SlimToolkit is the full name, so it's easier to find it online) to show its growing support for additional container tools and runtimes in the cloud native ecosystem.

Slim is now a CNCF Sandbox project. It was created by Kyle Quest and it's been improved by many contributors. The project is supported by Root.io (formerly known as Slim.AI).

Here's how Slim and Root work together: Slim helps you build optimized containers, while Root.io automatically fixes vulnerabilities without disrupting your workflows. Use Slim's open source toolkit to optimize containers, then keep them secure with Root's automated vulnerability remediation – from optimization to continuous security in one seamless journey. Learn more at www.root.io.

Overview

SlimToolkit allows developers to inspect, optimize…

View on GitHub

Simply download and run docker-slim build <myimage> and DockerSlim will examine the image, rebuild it with only the required dependencies, and give you a new image that can be run just like the original.

Pros: It's automatic

DockerSlim means you can work with whatever base image you'd like (say, Ubuntu or Debian) and let DockerSlim worry about removing unnecessary tools and files en route to production. The best part is that DockerSlim can be used alongside any of these other techniques. Once tested, it can be integrated into your CI/CD pipeline for automatic container minification, and the reduction in size leads to faster build times and better security.

Cons: Steep learning curve

As with any open-source software, DockerSlim can take some time to get working, especially for non-trivial applications. It works best for web-style applications, micro-services and APIs that have defined HTTP/HTTPS ports which the sensor can find and use to observe the container internals.

For best results, spend some time getting to know the various command flags available to tune your image, and take a look at the examples for whatever framework you're using.

slimtoolkit / examples

Minified application examples with different languages and stacks for DockerSlim

Connecting with DockerSlim

There's an active DockerSlim Discord channel full of experts who can help you triage issues as they arise.

Follow me here on dev.to, the SlimDevOps Twitch 📡 channel and SlimDevOps YouTube 📺 channel where I'll be doing more deep-dives into container optimisation using DockerSlim and the Slim.AI platform in future posts and live-streams.

Exclusive Slim.AI early access

And, if you got to the end of this article dear reader, then here is a bonus for you; an early access invite to the Slim.AI private beta 🙂

Image credit: Dominik Lückmann

Creating Production-Ready Containers - The Basics

Martin Wimpress — Thu, 03 Jun 2021 18:52:51 +0000

So you've coded an awesome app and you are ready to deploy it to the cloud. You've heard a lot about Docker and completed a few online tutorials to containerise your app. All set, right? But what do you need to know if you're going to move that app to a production environment on the public Internet? What if you're using it for your job and need to pass security scans and DevOps checks?

In this series, I introduce some basic concepts for making production ready containers. I also introduce the concept of "slimming" a container. Slimming refers to both optimising and minifying your Docker containers, reducing them in size by up to 80-percent while also making them more secure by decreasing the attack surface. Slimming your container is also a great way to implement container best practices without re-engineering your entire workflow.

There are many ways to slim a container, from basic security to fully automated open-source tools like DockerSlim. Full disclosure: I work for Slim.AI, a company founded on the DockerSlim open source project. Let's look at some of the common ways developers create production-ready container images today.

I'll explore each of these in a separate article using a simple "Hello World" Node.js example that can be found in a number of online tutorials.

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
 res.send('Hello World!')
})

app.listen(port, () => {
 console.log(`Example app listening at http://localhost:${port}`)
})

Let's get started by simply improving your Dockerfile to build a better Docker image.

Creating a Better Dockerfile

Most Dockerfile examples you'll find are not "production ready" and they aren't meant to be. They are for instructional purposes to help developers successfully build an image. But when one gets into production scenarios, there are a number of "good-to-know" and a few "have-to-know" techniques that will improve build times, security, and reliability.

Let's look at a typical example that you might run into if you're a Node.js developer looking to get "Hello World" running with Docker. I won't go through building an actual app - there are a lot of great examples out there to show you how to do this - but rather focus on what to do if you were actually going to ship this to production.

The typical Dockerfile in a "Hello World" example might look something like this:

FROM node:latest
WORKDIR /usr/src/app
COPY package*.json app.js ./
RUN npm install
EXPOSE 3000
CMD ["node", "app.js"]

It uses the latest version of the official Node.js image, sets a directory and copies your app into the container image, installs dependencies, exposes port 3000, and runs the app via CMD.

While this will run no problem on your local machine, and is great for learning the ropes, this approach is almost certainly going to run into issues when you ship it to production. Let's take a look at some of these in order of severity.

Major issues

Running as Root

Since this example doesn't set a USER explicitly in the Dockerfile, Docker runs the build and all commands as the root user. While not an issue for local development, your friendly neighbourhood SysAdmin will tell you the myriad of issues that come with running applications as root on a server in production. And with Docker, a new set of attack methods can arise.

Thankfully, most major languages and frameworks have a predefined user for running applications. In Node.js, the user is just node and can be invoked in the Dockerfile explicitly.

FROM node:latest
WORKDIR /usr/src/app
COPY package*.json app.js ./
RUN npm install

USER node

EXPOSE 3000
CMD ["node", "app.js"]

Using `latest` version

Choosing a version number for your container is often called pinning. While many tutorials - and even some experts - will counsel newcomers to pin their images to the latest tag, which means you get whatever the most recently updated version is, using the latest tag can cause issues in production.

Containers are meant to be ephemeral, meaning they can be created, destroyed, started, stopped, and reproduced with ease and reliability. Using the latest tag means there isn't a single source of truth for your container's "bill of materials". A new version or update of a dependency could introduce a breaking change, which may cause the build to fail somewhere in your CI/CD pipeline.

Example Dockerfile

FROM node:latest

Production Dockerfile

FROM node:16.2.0

Other tutorials I've seen pin only the major version. For example, using node:14. This carries the same risks as using latest, as minor versions can change dependencies as well.

Now, pinning a specific major and minor version in your Dockerfile is a trade-off decision - you're choosing to not automatically receive security, fixes or performance improvements that come via new updates - but most DevSecOps teams prefer to employ security scanning and container management software as a way to control updates rather than dealing with the unpredictability that comes with container build failures in production CI/CD pipelines.

Performance improvements

Better layer caching

Docker works on the concept of layer caching. It builds images sequentially. Layering dependencies on top of each other and only rebuilding them when something in the layer has changed.

Layer 0 in a Docker image is often the base operating system, which rarely change significantly; although commercial Linux vendors often publish new base images to incorporate security fixes.

Application code, however, is highly likely to change during the software development cycle, as you iterate on features, refactor, and fix bugs. Dependencies in our core system, installed here by npm install, change more often than the base OS, but less often than the application code.

In our example Dockerfile, we simply need to break the installation of the dependencies into separate instructions on their own lines.

FROM node:16.0.2
WORKDIR /usr/src/app

COPY package*.json ./
RUN npm ci

USER node

COPY app.js ./

EXPOSE 3000
CMD ["node", "app.js"]

We actually end up creating another layer by now having two COPY commands. While adding layers is typically a no-no for build times and image sizes, the tax we pay on this optimisation is going to save us in the long run as we cycle through the QA process, since we aren't reinstalling dependencies if we don’t have to.

We also opt for the npm ci command instead of npm install, which is preferred for automated environments, such as CI/CD, and will help prevent breaking changes from dependencies. Read more about npm ci here.

Use `ENTRYPOINT` instead of `CMD`

At a surface level, there isn't a big difference between using ENTRYPOINT with your app file versus running CMD using the shell plus your app file. However, web- and API-type containers like Node.js applications are often running as executables in production, and there, proper signal handling - such as graceful shutdowns - are important.

CMD provides some flexibility for calling executables with flags or overwriting them, which is common in development. But that generally won't be relevant to production instances and ENTRYPOINT will likely provide better signal processing.

FROM node:16.0.2
WORKDIR /usr/src/app

COPY package*.json ./
RUN npm ci

USER node

COPY app.js ./

EXPOSE 3000
ENTRYPOINT ["node", "app.js"]

Cleaning up cached files

Most package managers have the ability to clean up their own cache. If you don’t do this, you'll just be moving a bunch of unused files into your container for no reason. It might not save a lot of space depending on your application, but think of it as dropping your unused items at the charity shop before you move rather than loading them in the moving van. It's not a lot of effort and it's the right thing to do. We do this by adding && npm cache clean --force to our RUN instruction.

FROM node:16.0.2
WORKDIR /usr/src/app
COPY package*.json ./

RUN npm ci && npm cache clean --force

USER node

COPY app.js ./

EXPOSE 3000
ENTRYPOINT ["node", "app.js"]

Conclusions

Improving your Dockerfile is the first step towards creating a slimmed and optimised container. It closes some major security loopholes that are likely to raise flags with downstream checks and adds baseline optimisations for build time and docker image size.

If this is all you do to improve your containers prior to shipping to production, you won't be in a bad spot, but there's definitely more - way more - that you can do to optimise images. We'll explore those techniques in the next article.

Update

Since publishing this article my colleague and I ran through the techniques presented here in a video.

Image credit: Frank McKenna

DEV Community: Martin Wimpress

Creating Production-Ready Containers - Advanced Techniques

Creating Production-Ready Containers - The Basics

Martin Wimpress ・ Jun 3 '21

Alpine Images

Pros: Transitioning to Alpine can be an easy way to get a smaller container

FROM

TO

Cons: Using Alpine images can lead to build problems, now and in the future

You have to install everything

Different compilers and package managers

Fewer examples; less documentation

Multi-stage builds

Pros: Dev and Prod images can be built separately

Cons: Added complexity; use-case specific

Development tools and Distroless

Pros: When they work, they work

Cons: When they don't…

Automate it with DockerSlim

slimtoolkit / slim

Slim(toolkit): Don't change anything in your container image and minify it by up to 30x (and for compiled languages even more) making it secure too! (free and open source)

Optimize Your Experience with Containers. Make Your Containers Better, Smaller, More Secure and Do Less to Get There (free and open source!)

Overview

Pros: It's automatic

Cons: Steep learning curve

slimtoolkit / examples

Minified application examples with different languages and stacks for DockerSlim

Connecting with DockerSlim

Exclusive Slim.AI early access

Creating Production-Ready Containers - The Basics

Creating a Better Dockerfile

Major issues

Running as Root

Using latest version

Performance improvements

Better layer caching

Use ENTRYPOINT instead of CMD

Cleaning up cached files

Conclusions

Update

Using `latest` version

Use `ENTRYPOINT` instead of `CMD`