Using GitLab Kubernetes Runners to Build Melange Packages

Motivation

Recently, I came across Chainguard and wrote the article How to build Docker Images with Melange and Apko. As a fervent supporter of Kubernetes and GitLab CI, I was eager to experiment with building images using Melange in this particular setup. GitLab's shared Runners work seamlessly with Bubblewrap, eliminating the need for additional configurations. This post is intended for enthusiasts like myself, interested in hosting their own Kubernetes Runners and leveraging the Kubernetes Runner Type of Melange.

Understanding the Kubernetes Executor of Melange

Melange offers four Runner Types:

• Bubblewrap (default)
• Docker
• Lima
• Kubernetes

Each has its own security implications and drawbacks. I prefer Docker/Lima for local builds and Kubernetes in a CI environment running on a Kubernetes Cluster. Let's explore how to set this up.

When invoking Melange with the --runner Kubernetes flag, Melange will:

• Check if you have sufficient permissions to create pods in Kubernetes (RBAC).
• Build a temporary Docker Image and push it to ttl.sh.
• Run this Docker Image as a Pod in Kubernetes and build the package.

Remember, this is a high-level overview to understand the necessary configurations and not an explanation of the actual Kubernetes Runner.

Configuring the GitLab Runner

Now that we understand how the Melange Runner works, let's configure our GitLab Runner accordingly. Melange doesn't work with the out-of-the-box Role-Based Access Control (RBAC) Policy of the GitLab Runner. When using GitLab CI with Kubernetes, a Pod (from a Deployment) manages creating Pods for each Job in your GitLab CI Pipeline. This Service Account has the permissions to interact with pods, precisely what we need for Melange. However, Jobs started by this Orchestrator Deployment will only have the default ServiceAccount attached to it. This is fine for normal CI work but not for Melange, as it needs to create new pods and interact with them.

You can change this by modifying the config.template.toml Configmap or the runners.config in the values.yaml. Simply set the service_account for your Kubernetes Runner to the ServiceAccount of the Orchestrator. Here's an example if you named your Helm Release and thus the ServiceAccount gitlab-runner:

runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        service_account = "gitlab-runner"
        image = "alpine"

Remember: This has significant security implications. Now, every Job could start background Pods in your Cluster. While this might seem like a significant risk, anyone with access to your repository could do this anyway.

A small extra note: This is the lazy way to do it without the need to deploy any new ServiceAccount and other RBAC Resources. We just need to tweak one little line in our values.yaml file, which I find convenient. If you find a better and more secure way, please let me know and leave a comment!

Configuring Melange

When using the Kubernetes Runner, Melange looks for a config file named .melange.k8s.yaml. Here you can specify the Namespace of the pod, the container Registry to be used, and other Kubernetes Specs. By default, Melange will attempt to start a pod in the default Namespace. However, since no sane person uses the default namespace, we can easily change it to the Namespace of our GitLab Runner.

While we're at it, we can also change the Registry and Repository to the GitLab Registry instead of using ttl.sh. We don't want everyone to have access to our super secure project, do we?

Let's assume our GitLab Runner is running in a namespace called stammkneipe-dev, and our project has the following path stammkneipe.dev/golang-apko-example:

namespace: stammkneipe-dev
repo: registry.gitlab.com/stammkneipe.dev/gitlab-runner-operator

Configuring GitLab CI

Now we are finally ready to use Melange with the Kubernetes Executor in GitLab CI. The only thing left to do is to ensure that Melange has the permission to push images to the GitLab Registry, since we changed the default from ttl.sh to registry.gitlab.com. GitLab has some predefined Variables that can be used to authenticate against our Registry:

• CI_REGISTRY
• CI_REGISTRY_USER
• CI_REGISTRY_PASSWORD

Our final build job would look something like this:

package:
  stage: build
  image:
    name: cgr.dev/chainguard/melange:latest
    entrypoint: [""]
  before_script:
    - cat ${MELANGE_RSA} > melange.rsa # Assuming you have set a Melange RSA Key as a GitLab CI Variable
  script:
    - 'mkdir ~/.docker && echo -n "{\"auths\": {\"${CI_REGISTRY}\": {\"auth\": \"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > ~/.docker/config.json'
    - melange build --workspace-dir $CI_PROJECT_DIR --runner kubernetes --signing-key melange.rsa melange.yml
  artifacts:
    paths:
      - ./packages

Conclusion and TL:DR

To use the Kubernetes Executor of Melange with GitLab CI and the Registry, you need to do the following three things:

• Set the service_account for the runners.config in the [runners.kubernetes] section to the orchestrator Service Account.
• Create .melange.k8s.yaml with your namespace set to the GitLab Runner Namespace and the repo set to the GitLab CI Repo (registry.gitlab.com/<CI_PROJECT_PATH>).
• Authenticate against the GitLab Registry: 'mkdir ~/.docker && echo -n "{\"auths\": {\"${CI_REGISTRY}\": {\"auth\": \"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > ~/.docker/config.json'

Now you should be able to use Melange without the need for any external Registries and using your Kubernetes Cluster to build packages.