Fly.io registry for custom base Dockerfiles

Intro

From the moment my friend Tomas recommended me Fly.io, I just in love with it for my side projects. Dubbed as the "Repacked Spirit of OG Heroku" - and from the past 6 months using I can testify to the truth of the statement!

Copy-pasted Dockerfiles taking over my project

This article focuses on how to re-use your custom Docker base across multiple Fly.io apps. My personal use case were deploying multiple batch jobs each having the same Python base but a few tweaks in the end for the entrypoint or config values. Making sure all 4 of those Dockerifles are the same was somewhat annoying to me so first I was like

TOOD(P1, devx): Have a common Docker base image". 
FROM python:3.12-slim

WORKDIR /app/backend

# `git` is required by python package `supawee`
# `postgresql-dev` and `libffi-dev` are for psycopg[binary,pool] (this is always such a pain to install)
RUN apt-get update && apt-get install -y \
    git \
    libpq-dev \
# ... blah blah blah

The moment

But then a sub-package wanted to install tiktoken which for performance reasons requires rustup so now I had to copy-past that snippet to all my Dockerfiles and it takes like 1-2 mins to build I was aaargh if I only had one base image for my project I only have to build it once! So that's how it started.

Result

Pasting two of my Dockerfiles to get you an idea of how re-using the common Dockerfile.base can help you manage your deployment easier:

FROM registry.fly.io/web-scraping-batch-jobs-base-registry:latest

CMD ["python", "-m", "batch_jobs.scraper.daily_scrape"]

and second

FROM registry.fly.io/web-scraping-batch-jobs-base-registry:latest
ENV YOLO=true
CMD ["python", "-m", "batch_jobs.upsert_plugins"]

Solution

Well it took me a few hours to figure this out! :face-palm:
At the high level:

• build the base image locally
• push it to fly.io registry
• fly deploy with app-specific Dockerfile referencing FROM:

Resulting script

Here I employ a new style of blog writing, context-based explaining of each (hopefully) self-describing command.

#!/bin/bash
# TODO(P1, blog): Write down my experience building this for fly.io

# Usage deploy_batch_job.sh scraper/fly.toml
FLY_CONFIG_PATH=$1

# Set variables
BASE_DOCKERFILE="<your-path-to>/Dockerfile.base"
# Here `<my-project>-base-registry` is a dummy app on fly.io which is just used for storing base docker images. 
# You can create such a dummy app by `fly apps create $REGISTRY_DUMMY_APP_NAME`
REGISTRY_DUMMY_APP_NAME="<my-project>-base-registry"
IMAGE_NAME="registry.fly.io/$REGISTRY_DUMMY_APP_NAME"
# we do --platform linux/amd64 to match the one fly.io builders have
#   https://github.com/superfly/rchab/blob/8d37d90dc7d418660b50a10f288715fda4a00b5d/build.sh#L7
PLATFORM="linux/amd64" 

echo "Logging into fly.io registry"
# Authentication successful. You can now tag and push images to registry.fly.io/{your-app}
fly auth docker

echo "Building batch jobs base image $DOCKERFILE ($PLATFORM) to {$IMAGE_NAME}"

# Build the image locally
docker build --platform $PLATFORM -t $IMAGE_NAME:latest -f $DOCKERFILE .

# Get the image ID (content hash), removing the "sha256:" prefix as a 
IMAGE_ID=$(docker inspect --format='{{.Id}}' $IMAGE_NAME:latest | cut -d: -f2)
echo "Image ID: $IMAGE_ID (content hash)"
FULL_IMAGE_NAME="$IMAGE_NAME:$IMAGE_ID"

# Tag the image with its content hash
docker tag $IMAGE_NAME:latest $FULL_IMAGE_NAME

# Check if the image already exists in the registry
if docker manifest inspect $FULL_IMAGE_NAME > /dev/null 2>&1; then
    echo "Image $FULL_IMAGE_NAME"
    echo " -- already exists in the registry."
    echo " -- Skipping push."
else
    echo "Pushing new image $FULL_IMAGE_NAME to the registry."
    docker push $IMAGE_NAME:$IMAGE_ID
    docker push $IMAGE_NAME:latest
fi

echo "Deploying the batch job $FLY_CONFIG_PATH"
# To debug problems with Fly.io app builders you can find them at https://fly.io/dashboard/<your-organization-snake-case>/builders
# or with CLI fly logs -a fly-builder-<assigned-builder-name> (get it from logs)
fly deploy --config $FLY_CONFIG_PATH

Discussion

Was it worth it?

Well, I conclude that this effort to be net positive for humanity I had to share it! Cause truth to be told, continuing doing careful copy-pasting (a synonym to devops from a devops person I really appreciate) would been more effective for me. But hey, we lazy engineers who want to automate our job right?

Also #lesscodelessbugs

Feature: If the image is same, do not push it

It took a few Claude Sonet (I am sick of ChatGPT for programming) I managed to setup a content-hash based push if doesn't yet exist approach to save some deploy time. Cause you know, if deploy takes over 30 seconds you tab-out to some stupid YouTube video or visit MK and then it takes 5mins to deploy.

Some gotchas

• There is the --platform flag to match your local build with the fly.io build. Some versions of Docker Client (like Docker Desktop) had problems with that. I personally use colima which is also much more battery efficient.

Possible improvements

• Would be nice to build the Dockerfile.base using the fly.io workers; certainly possible with I guess fly.toml.
• FROM <your-base-image> takes some while even when both imae

Appendix

The full git commit setting this up in my side-project to see a
"production" version linked here on my github.