12 Things You Might Not Know About Buildpacks

Joe Kutner — Thu, 01 Dec 2022 13:53:16 +0000

For most buildpack users, the only thing you need to know is that running pack build will turn your source code into a Docker image. But the power of buildpacks goes far beyond this initial experience. Buildpacks also provide powerful features that no other image building tool can match. Let's take a look at a few of them.

1. Use metadata from a previous build

A buildpack can analyze information about layers it created during a previous build. For buildpack authors, this can aid in optimizing build performance by skipping steps if a buildpack determines something hasn't changed.

For example, it's common to store a checksum of a dependency in the metadata of a build layer, and then compare it to the latest version in the next build. If they match, the buildpack may skip installing that dependency and instead reuse the exact same layer from the previous build. This avoids the costly overhead of downloading large layers and moving them around unnecessarily.

Effectively, this means a buildpack can construct a new image from both existing layers and new layers in any order.

For more information on buildpack layer caching, see the documentation on buildpack layer types.

2. Update the operating system without re-building

Buildpacks allow you to rapidly update an image when its base image (i.e operating system) has changed. This process is called rebasing, and it avoids the need to fully rebuild the app when a new operating system or base image becomes available.

Consider an image my-app:latest that was originally built using the latest version of a base image, pack/run:latest. Running the following will update those layers of my-app:latest with the latest version of pack/run:latest without rebuilding the app layers above them:

pack rebase my-app:latest

Avoiding rebuilds is a nice performance boost for a single image, but for a fleet of hundreds (or even millions) of images, it becomes a critical operational and security tool.

3. Write Buildpacks in any language

Not everyone needs to write a buildpack. Most people just use the Paketo, Heroku, or Google buildpacks. But for those who need to create one from scratch, they're able to choose any language they want.

The buildpack API defines its entry-points only as executables. That means the executable can be a Python script, a Bash script, or a binary compiled from Golang code.

The Paketo Buildpacks are mostly written in Go, while the Heroku buildpacks are written in Bash and Rust. The libcnb library is a Go language binding of the Cloud Native Buildpacks API. It's supported by the project, but there are also libraries outside of the project like libcnb.bash.

4. Add buildpacks inline with your source code

You can create a buildpack in your source code repo very similarly to how you might drop in a Dockerfile. To do so, create a project.toml file and define your buildpack commands inline.

For example, if you wanted run a Bash script as part of your build, your project.toml would include something like this:

[[build.buildpacks]]
id = "me/setup-tasks"

  [build.buildpacks.script]
  api = "0.8"
  inline = "bash setup.sh"

For more information, see the inline buildpacks documentation.

5. Create images from scratch

Buildpacks can build images that have no underlying base layers, just as if you were using FROM scratch. Most of the off-the-shelf buildpacks from Paketo and Heroku don't do this because they rely on the operating system they run on, but when creating your own buildpacks it is possible with a custom base image. The Dockerfile for such a base image can be as simple as this:

FROM scratch
ENV CNB_USER_ID=1000
ENV CNB_GROUP_ID=1000
LABEL io.buildpacks.stack.id="my-scratch-img"
USER ${CNB_USER_ID}:${CNB_GROUP_ID}

For a more robust but still slim image, try the Paketo "tiny" base image. For more information on using custom base images see the documentation on stacks.

6. Use the Pack CLI as a library

The Pack CLI (i.e. pack) is a great tool for using buildpacks. But many end users want to bake these capabilities directly into some other tooling. Fortunately, pack is also a Golang library. You can include it in other Go based projects by simply running:

go get github.com/buildpacks/pack

Don't worry, this is a completely supported pattern. In fact, many of the organizations contributing to the project, including Salesforce, do this very thing.

7. Compose buildpacks with other buildpacks

Most advanced buildpacks aren't actually a single buildpack, but instead a composite of several buildpacks that each do smaller jobs. For example, the Heroku Node.js buildpack is actually a collection of several NPM and Node engine buildpacks. Here's an excerpt from its buildpack.toml descriptor:

[[order.group]]
id = "heroku/nodejs-engine"
version = "0.8.12"

[[order.group]]
id = "heroku/nodejs-npm"
version = "0.5.2"

[[order.group]]
id = "heroku/procfile"
version = "2.0.0"
optional = true

This composability makes it much easier for buildpack authors to reuse buildpacks, and allows end users to leverage them in unexpected ways.

8. Create a standardized SBOM

A Software-Bill-of-Materials (SBOM) lists all the software components included in an image. Buildpacks support SBOMs in CycloneDX, Syft and SPDX formats.

Each buildpack can populate the SBOM with the information about the dependencies they have provided. And end users can view that metadata with the Pack CLI by running:

pack sbom download your-image-name

For more information see the buildpacks documentation on SBOM.

9. Use buildpacks without Docker installed

Pack uses the Docker API to run containerized builds and create runnable OCI images, which means it works equally well with a secure remote daemon as it does with a local daemon. Using a remote daemon looks something like this:

export DOCKER_HOST=tcp://daemon-hostname:2376
export DOCKER_CERT_PATH=~/.docker/
export DOCKER_TLS_VERIFY=1

pack build my-app --path ./my-source

For more information on how to do this, see our past blog post using Pack with a remote Docker daemon.

10. Create reproducible builds

Given a set of inputs (source code, dependencies, etc) a buildpack can produce an image with the exact same digest (i.e. the same checksum of everything it contains) as an image from a previous build.

Reproducible builds aren't something that happens by default; each buildpack needs to be smart enough to compare the layers it creates to the metadata from previous builds. But the buildpacks API makes it possible for discerning users.

With something like Dockerfile, reproducible builds are effectively impossible because even a change to the timestamp of a file will give you a new image digest. This is why buildpacks produce images that appear to be created 40 years ago (you can use the --creation-time flag if you don't want this feature).

11. Run buildpacks with Github Actions

The Buildpacks project maintains a set of github-actions that can be used for various buildpack related tasks. Among these is the ability to set up a job that is ready with the pack CLI. It's as simple as using this action:

uses: buildpacks/github-actions/setup-pack@v4.1.0

From here, you can run pack to build images, create builders, inspect SBOM metadata, and more.

12. Use buildpacks with a Dockerfile

Ok, I'm cheating here because this might not be ready when I publish this blog post. Very soon, you'll be able to provide a Dockerfile in your buildpack build in order to customize the build process.

For more information, see the finalized RFC on supporting Dockerfiles.

DEV Community: Joe Kutner