Building docker images in docker, and dynamic sequential Jenkins stages

A couple of things I thought would be super easy but turned out to require a few hours of research and trial-and-error...

At GOALS we run Jenkins in Kubernetes for various reasons.
Some build jobs generates Docker images as artefacts, for example when we build new versions of some backend service.
However, Kubernetes is deprecating Docker, and we are running all our nodes on containerd.
So, how to build Docker images then?

Turns out that Google has a project for it, named Kaniko. And since we're running GKE, and has Workload Identity properly configured, it should be super easy to build Docker images and push them to our GCP managed docker repo, right?

No. I get this error:

aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "europe-west1-docker.pkg.dev/XXX": creating push check transport for europe-west1-docker.pkg.dev failed: GET https://europe-west1-docker.pkg.dev/v2/token?YYY: UNAUTHORIZED: authentication failed

AWS credentials - wth?

After some testing I found this, which solved the issue :). For an example Jenkinsfile, see below.

Another issue I had was that I wanted to generate dynamic build stages in Jenkins, and execute them sequentially. There are a lot of examples of how to execute dynamically generated stages in parallel, but that's not what I wanted.
Turned out to be super simple in the end ofc, but it aint simple until you've learned.

Here's an example Jenkinsfile that demonstrates both Kaniko and dynamic sequential build steps:

def stageConfigs = [
  [name: "Sara", age: 20, color: "blue"],
  [name: "Mona", age: 10, color: "green"],
  [name: "Lotta", age: 8, color: "red"]
]

def generateBuildStage(stageConfig) {
  return {
    stage("Building ${stageConfig.name}") {
      container('build-container') {
        echo "Building ${stageConfig.name}"
        writeFile(
          file: "${stageConfig.name}.generated",
          text: "${stageConfig.name} is ${stageConfig.age} years old and loves ${stageConfig.color} things")
      }
    }
  }
}

def buildStages = stageConfigs.collectEntries {
    ["${it}" : generateBuildStage(it)]
}

podTemplate(
  inheritFrom: 'linux',
  containers: [
    containerTemplate(name: "build-container", image: "busybox", command: "sleep", args: "infinity"),
    containerTemplate(name: "kaniko", image: "gcr.io/kaniko-project/executor:343f78408c891ef7a85bab1ecbf2dd69367a58bc-debug", command: "sleep", args: "infinity", runAsUser: "0", ttyEnabled: true)])
  {
  node(POD_LABEL) {
    stage("Checkout") {
      checkout(scm)
    }

    stage("Build application") {
      // execute builds in parallel
      // parallel(buildStages)
      // execute builds sequentially
      for (stage in buildStages.values()) {
        stage.call()
      }
      container('build-container') {
        writeFile(
          file: "Dockerfile",
          text: '''
            FROM busybox
            COPY *.generated ./
          ''')
        sh "ls -al"
      }
    }

    stage('Build image') {
      container('kaniko') {
        sh "/kaniko/executor --context `pwd` --dockerfile `pwd`/Dockerfile --destination europe-west1-docker.pkg.dev/XXX"
      }
    }
  }
}