Faster builds with VMs than with Google Cloud Build

Google Cloud Build has to pull your Docker image on every build, which can be slow if the image is 1GB+ (e.g. when using cutting edge Wasm tooling that doesn't work on all OSes/architectures yet).

Here's how to speed up your builds using plain VMs instead:

initiate-build.sh creates/starts/resumes the build VM if it's nonexistent/stopped/suspended, copies a Dockerfile and context, then runs build.py on the VM and copies the artifact back to your local machine:

set -xeuo pipefail

cd "$(dirname "${BASH_SOURCE[0]}")"

: "${INSTANCE:=some-build-vm}"

status() {
  gcloud compute instances list --format="value(status)" --filter="name=$INSTANCE" | tr -d '\n'
}

state="$(status)"
case "$state" in
"")
  gcloud compute instances create "$INSTANCE" --machine-type=n2-standard-2 --create-disk=boot=yes,image=projects/ubuntu-os-cloud/global/images/ubuntu-2210-kinetic-amd64-v20221101,size=100
  ;;
"SUSPENDED")
  gcloud compute instances resume "$INSTANCE"
  ;;
"STOPPED")
  gcloud compute instances start "$INSTANCE"
  ;;
"RUNNING") ;;
*)
  echo "Unrecognized instance state $state"
  exit 1
  ;;
esac

rm -rf scratch
mkdir scratch
cp setup.py auto-shutdown.py auto-shutdown.service Dockerfile otherfiles scratch
gcloud compute scp --recurse scratch "$INSTANCE":"~"
gcloud compute ssh "$INSTANCE" -- 'bash -c "( cd scratch && python3 setup.py ); rm -rf scratch"'
gcloud compute scp "$INSTANCE":"~/artifact" artifact

build.py installs Docker if needed, sets up an auto-shutdown after 5mins of inactivity (to mimic the serverless nature of Google Cloud Build), builds the Docker image, then copies the artifact out:

#!/usr/bin/env python3

import subprocess
from subprocess import check_call


def main():
    ensure("jc", "jc")
    ensure("docker", "docker.io")

    check_call("sudo chmod 755 auto-shutdown.py", shell=True)
    check_call("sudo mv auto-shutdown.py /usr/bin", shell=True)
    check_call("sudo mv auto-shutdown.service /etc/systemd/system", shell=True)
    check_call("sudo systemctl daemon-reload", shell=True)
    check_call("sudo systemctl restart auto-shutdown", shell=True)

    check_call("sudo docker build -t img .", shell=True)

    check_call(
        f"sudo docker run --rm --entrypoint cat img /artifact > ~/artifact", shell=True
    )


def ensure(command, pkg):
    try:
        check_call(["which", command])
    except subprocess.CalledProcessError:
        check_call("sudo apt-get update", shell=True)
        check_call(["sudo", "apt-get", "install", "-y", pkg])


if __name__ == "__main__":
    main()