Docker Caching with Compose

When working with Docker Compose, enabling caching can significantly speed up your builds by reusing layers from previous builds. This article will guide you through the process of setting up caching for Docker Compose, ensuring your development workflow is as efficient as possible.

Prerequisites

Before we start, make sure you have the following installed:

• Docker CLI
• yq CLI (YAML processor)
• docker-compose.yaml file

Step-by-Step Guide

Skip the Step-by-Step and go straight to the sauce.

Step 1: Check for Necessary Tools

First, we need to ensure that the required tools are installed on your system. Here’s a script that checks for Docker CLI, yq CLI, and the presence of the docker-compose.yaml file.

# Check for Docker CLI
if ! command -v docker &> /dev/null
then
    echo "Docker CLI not found. Please install Docker."
    exit 1
fi

# Check for yq CLI
if ! command -v yq &> /dev/null
then
    echo "yq CLI not found. Please install yq."
    exit 1
fi

# Check for docker-compose.yaml
if [ ! -f ./docker-compose.yaml ]; then
    echo "docker-compose.yaml not found. Please ensure the file exists in the current directory."
    exit 1
fi

Step 2: Set Up Preflight Script

The preflight script prepares the environment for caching. It sets up a temporary file path for the caching configuration.

# Set up necessary environment variables if not already set
: "${DOCKER_IMAGE:=your_default_docker_image}"
: "${DOCKER_REGISTRY:=your_default_docker_registry}"
: "${DOCKER_USER:=your_default_docker_user}"
: "${DOCKER_PASSWORD:=your_default_docker_password}"

# Preflight script for cacher-preflight
working_dir=$(dirname "$(realpath ./docker-compose.yaml)")
temp_file="$working_dir/docker-compose.caching.yaml"
export CACHE_COMPOSE_PATH=$temp_file
export DOCKER_BUILDKIT=1

Step 3: Set Up Cacher Script

The cacher script modifies the Docker Compose file to include cache settings for each service.

# Copy into cache path, and add caching
cp ./docker-compose.yaml $CACHE_COMPOSE_PATH
services=$(yq -r '.services | keys | .[]' ./docker-compose.yaml)
for service in $services; do
  yq eval -i ".services.${service}.build.cache_from[0] = \"type=registry,ref=$DOCKER_IMAGE:${service}-buildcache\"" $CACHE_COMPOSE_PATH
  yq eval -i ".services.${service}.build.cache_to[0] = \"type=registry,ref=$DOCKER_IMAGE:${service}-buildcache,mode=max\"" $CACHE_COMPOSE_PATH
done

Step 4: Build and Push Script

Finally, the build script builds and pushes the Docker images, utilizing the caching configurations set up in the previous steps.

# Build script for build
docker buildx create --use

# After using buildx, you will need to relogin into docker.
echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USER" --password-stdin $DOCKER_REGISTRY

docker compose -f $CACHE_COMPOSE_PATH build
docker compose -f $CACHE_COMPOSE_PATH push

Conclusion

With this I was able to speed up compilation in CI/CD greatly by utilizing remote caching. :)