DEV Community

Chris Wendt
Chris Wendt

Posted on • Edited on

3 2

Faster builds with VMs than with Google Cloud Build

Google Cloud Build has to pull your Docker image on every build, which can be slow if the image is 1GB+ (e.g. when using cutting edge Wasm tooling that doesn't work on all OSes/architectures yet).

Here's how to speed up your builds using plain VMs instead:

initiate-build.sh creates/starts/resumes the build VM if it's nonexistent/stopped/suspended, copies a Dockerfile and context, then runs build.py on the VM and copies the artifact back to your local machine:

set -xeuo pipefail

cd "$(dirname "${BASH_SOURCE[0]}")"

: "${INSTANCE:=some-build-vm}"

status() {
  gcloud compute instances list --format="value(status)" --filter="name=$INSTANCE" | tr -d '\n'
}

state="$(status)"
case "$state" in
"")
  gcloud compute instances create "$INSTANCE" --machine-type=n2-standard-2 --create-disk=boot=yes,image=projects/ubuntu-os-cloud/global/images/ubuntu-2210-kinetic-amd64-v20221101,size=100
  ;;
"SUSPENDED")
  gcloud compute instances resume "$INSTANCE"
  ;;
"STOPPED")
  gcloud compute instances start "$INSTANCE"
  ;;
"RUNNING") ;;
*)
  echo "Unrecognized instance state $state"
  exit 1
  ;;
esac

rm -rf scratch
mkdir scratch
cp setup.py auto-shutdown.py auto-shutdown.service Dockerfile otherfiles scratch
gcloud compute scp --recurse scratch "$INSTANCE":"~"
gcloud compute ssh "$INSTANCE" -- 'bash -c "( cd scratch && python3 setup.py ); rm -rf scratch"'
gcloud compute scp "$INSTANCE":"~/artifact" artifact
Enter fullscreen mode Exit fullscreen mode

build.py installs Docker if needed, sets up an auto-shutdown after 5mins of inactivity (to mimic the serverless nature of Google Cloud Build), builds the Docker image, then copies the artifact out:

#!/usr/bin/env python3

import subprocess
from subprocess import check_call


def main():
    ensure("jc", "jc")
    ensure("docker", "docker.io")

    check_call("sudo chmod 755 auto-shutdown.py", shell=True)
    check_call("sudo mv auto-shutdown.py /usr/bin", shell=True)
    check_call("sudo mv auto-shutdown.service /etc/systemd/system", shell=True)
    check_call("sudo systemctl daemon-reload", shell=True)
    check_call("sudo systemctl restart auto-shutdown", shell=True)

    check_call("sudo docker build -t img .", shell=True)

    check_call(
        f"sudo docker run --rm --entrypoint cat img /artifact > ~/artifact", shell=True
    )


def ensure(command, pkg):
    try:
        check_call(["which", command])
    except subprocess.CalledProcessError:
        check_call("sudo apt-get update", shell=True)
        check_call(["sudo", "apt-get", "install", "-y", pkg])


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

See auto-shutdown.

Heroku

Simplify your DevOps and maximize your time.

Since 2007, Heroku has been the go-to platform for developers as it monitors uptime, performance, and infrastructure concerns, allowing you to focus on writing code.

Learn More

Top comments (0)

Heroku

Build apps, not infrastructure.

Dealing with servers, hardware, and infrastructure can take up your valuable time. Discover the benefits of Heroku, the PaaS of choice for developers since 2007.

Visit Site

👋 Kindness is contagious

Explore a sea of insights with this enlightening post, highly esteemed within the nurturing DEV Community. Coders of all stripes are invited to participate and contribute to our shared knowledge.

Expressing gratitude with a simple "thank you" can make a big impact. Leave your thanks in the comments!

On DEV, exchanging ideas smooths our way and strengthens our community bonds. Found this useful? A quick note of thanks to the author can mean a lot.

Okay