Modernizing cloudbuild.yaml for Container Builds

#googlecloud #devops #docker #cicd

tl;dr: Running bash scripts in the Cloud Build documentation tells you use the script property with the automapSubstitutions option and I give this recommendation a 💯.

One of my favorite things to do in writing YAML is minimizing the square brackets and hyphens. The frequency of these things really impacts readability for me, especially when I'm trying to glance at a whole block of markup and assess what it's trying to do.

Over the last couple years, Cloud Build has made improvements in reducing square brackets and enabling more concise configuration for running shell scripts.

I'm going to step through a couple "generations" of simplification I walked through today in overhauling a container image build pipeline.

This is what a lot of minimal Cloud Build configurations look like:

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  args: ['build', '--tag', 'gcr.io/${PROJECT_ID}/${_IMAGE}:latest', '.']

- id: 'Push Container Image to Container Registry'
  name: 'gcr.io/cloud-builders/docker:latest'
  args: ['push', 'gcr.io/${PROJECT_ID}/${_IMAGE}:latest']

substitutions:
  _IMAGE: service

Starting character count: 335

Step #1: Remove square-bracketed [distractions]

Use the script property instead of args.

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build --tag gcr.io/${PROJECT_ID}/${_IMAGE}:latest .

- id: 'Push Container Image to Container Registry'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker push gcr.io/${PROJECT_ID}/${_IMAGE}:latest

substitutions:
  _IMAGE: service

🛑 If you try to use this it won't work.

It turns out switching from args to script means the code no longer has substitutions, only environment variables.

A year ago, the next step would be to manually map the substitutions to environment variables:

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build --tag gcr.io/${PROJECT_ID}/${_IMAGE}:latest .

- id: 'Push Container Image to Container Registry'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker push gcr.io/${PROJECT_ID}/${_IMAGE}:latest

options:
  env:
  - PROJECT_ID=$PROJECT
  - _IMAGE=$_IMAGE

substitutions:
  _IMAGE: service

Updated character count: 390 (sometimes shorter isn't clearer)

Step #2: Remove excess hyphenation

Use the automapSubstitutions global option and the substitutions are injected as environment variables to all step scripts.

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag gcr.io/${PROJECT_ID}/${_IMAGE}:latest

- id: 'Push Container Image to Container Registry'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker push gcr.io/${PROJECT_ID}/${_IMAGE}:latest

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service

Updated character count: 373 (progress, but I'd be more satisfied if we got back to 335 somehow)

Simplify the Image Push

Since this Cloud Build configuration doesn't have any follow-up steps that need to use the image outside the build, I can simplify further, by replacing the step Push Container Image to Container Registry with the images property. (Read more about that in Remember images in your cloudbuild.yaml! by @glasnt).

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag gcr.io/${PROJECT_ID}/${_IMAGE}:latest

images:
- gcr.io/${PROJECT_ID}/${_IMAGE}:latest

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service

This doesn't reduce hyphens any further, but it does reduce two lines of configuration.

Updated character count: 263 (Success!)

Bonus Round: Migrating to Artifact Registry

You may have noticed this configuration ships the container image to a Google Container Registry URL! There's a good chance if you are modernizing your Cloud Build YAML this way you've got some GCR in place, if so, you need to migrate soon.

Check out Transition from Container Registry. There are options to migrate while keeping the gcr.io URL.

Say I've already created a new Artifact Registry instance for my container, what does that YAML look like?

(Let's say it's a multi-regional repository called container in the us location.)

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag "us-docker.pkg.dev/${PROJECT_ID}/container/${_IMAGE}:latest"

images:
- "us-docker.pkg.dev/${PROJECT_ID}/container/${_IMAGE}:latest"

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service

Since an Artifact Repository is a Cloud resource, a project might have more than one. This encourages me to parameterize the location and repository name, so that development & testing are easier.

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag "${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest"

images:
- "${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest"

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service
  _LOCATION: us
  _REPO: container

Updated character count: 358 (A little longer than 335, but Artifact Registry is more verbose than Container Registry, not much we can do about that.)

In a more complicated YAML block, automapSubstitutions would make a much bigger difference. I go out of my way to avoid using args, and this option helps me do that without needing to add a lot of boilerplate config.

Aside Info: Reader, maybe you'll suggest I delete all those pesky curly braces around my variables, getting me a win at 342 characters. This stackoverflow answer explains the value of the curly braces, and I'm one of the people that makes curly braced variables part of my "shell variables in strings" practice.