CI/CD for Kubernetes using GitHub Actions, and Keel

In this article, the goal is to show how to set up a containerized application in Kubernetes with a very simple CI/CD pipeline to manage deployment using GitHub Actions and Keel.

Before we start:

Kubernetes, also known as K8s, is an open-source container orchestration system for automating deployment, scaling, and management.

Keel is a K8s operator to automate Helm, DaemonSet, Stateful & Deployment updates. It’s open-source, self-hosted with zero requirements of CLI/API, and comes with a beautiful and insightful dashboard.

GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub.

Workflow:

There are basically two steps in the workflow.

• You will push some changes to the GitHub repo. A workflow will be triggered, which will build the docker image of our application and push the image to the Docker registry.
• Keel will get notified of the updated image. Based on the update policy, the deployment will be updated in the configured cluster.

Step 1:

In the first step, we will prepare the GitHub repo to trigger workflows.

The repo, aka-achu/go-kube contains a simple web application written in golang and a Dockerfile, which will be used to build a docker image of the application. You can maintain any number of environments for your application like Production, QnA, Staging, Development, etc. For sake of simplicity, we will be maintaining only two deployment environments of the application.
There are only two branches in the repo.

• main branch (for Production environment)

name: Stable Build
on:
  push:
    tags:
      - "*.*.*"
...
      - name: Set tag in env
        run: echo "TAG=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
...
          tags: runq/go-kube:${{ env.TAG }}, runq/go-kube:latest

The stable workflow will be triggered when a tag is pushed to GitHub. In the workflow, the tag associated with the commit will be used as the docker image tag.

• dev branch (for Development/Staging environment)

name: Development Build
on:
  push:
    branches: [ dev ]
...
      - name: Set short commit hash in env
        run: echo "COMMIT_SHA=$(echo $GITHUB_SHA | cut -c1-7)" >> $GITHUB_ENV
...
          tags: runq/go-kube:dev-${{ env.COMMIT_SHA }}

The dev workflow will be triggered when any changes are pushed to the dev branch. For development builds, instead of git tags, we will use the short commit hash of length 7, which we often see in GitHub. The docker image tag of the development build image will be dev-SHORT_COMMIT_SHA.

Both workflows aim to integrate the code, maybe run some tests, build the docker image, and update the image registry. Till this point, we have completed Continuous Integration and Continuous Delivery.

Step 2:

In this step, we will automate our deployment update. We will be using the K8s LoadBalancer service in our workflow. So, if you're using an on-premise cluster then you can use MetalLB, which is a load-balancer implementation for bare metal Kubernetes clusters.

Install keel:

Keel doesn't need a database. Keel doesn't need any persistent disk. It gets all required information from your cluster

kubectl apply -f https://sunstone.dev/keel?namespace=keel&username=admin&password=admin&tag=latest

The above command will deploy Keel to keel namespace with basic authentication enabled and admin dashboard. You can provide an admin password while applying the manifest or you can download the manifest and replace the default password.

            # Basic auth (to enable UI/API)
            - name: BASIC_AUTH_USER
              value: admin
            - name: BASIC_AUTH_PASSWORD
              value: admin
            - name: AUTHENTICATED_WEBHOOKS
              value: "false"

Keel policies:

In keel, we use policies to define when we want our application/deployment to get updated. Following semver best practices allows you to safely automate application updates. Keel supports many different policies to update resources. For now, we will use only all and glob policies to update our deployment.

• all: update whenever there is a version bump (1.0.0 -> 1.0.1) or a new prerelease created (ie: 1.0.0 -> 1.0.1-rc1)
• glob: use wildcards to match versions (eg: dev-* in our scenario)

How Keel will get notified:

• Webhooks- when a new image is pushed to the registry, keel will get notified by the added webhook in DockerHub, and based on the configured update policy, the deployment will be updated.
• Polling- when an image with a non-semver style tag is used (ie: latest) Keel will monitor SHA digest. If a tag is semver - it will track and notify providers when new versions are available.

Configuring webhook:

First, we need to get the External IP address of the keel service.

kubectl get all -n keel

The output will something like this:

Now, that we have the external address of the service/keel, we will add a webhook for our repository in DockerHub. The URL for the hook will be http://<External-IP>:9300/v1/webhooks/dockerhub. Now pushing a new docker image will trigger an HTTP call-back.

If you don't want to expose your Keel service - the recommended solution is webhookrelay which can deliver webhooks to your internal Keel service through a sidecar container. Here is how you can set up the sidecar.

Configuring deployment manifest:

• To configure our staging deployment, which runs an image having a tag of dev-SHORT_COMMIT_SHA, we will use the glob policy. We will specify our policy/update rule using annotations under the metadata of the deployment manifest. Like-

...
  annotations:
    keel.sh/policy: "glob:dev-*"
...

• To configure our production deployment, which runs an image having a semver tag, we will use the all policy. This will update our production deployment when it encounters any version bump in the tag. We will specify our policy/update rule using annotations under the metadata of the deployment manifest. Like-

...
  annotations:
    keel.sh/policy: all
...

Testing the workflow:

Now, that we are done with the setup, a push to the dev branch, will update the staging deployment and a tagged release will update the production deployment.

• Push changes to the dev branch
• Development workflow will be triggered which will build a docker image
• A Docker image with the tag dev-SHORT_COMMIT_SHA will be pushed to the DockerHub registry.
• Keel will get notified by DockerHub using the webhook
• Keel will validate, whether the new image tag satisfies the policy we have specified (all tags starting with dev- will qualify for the update process. eg: dev-c722d00, dev-0ca740e)
• If the tag qualifies for the update, keel will create a new replicaset with the new image. Once the pods are ready, keel will scale the number of replicas in the old replicaset to 0.

The release of a tag will trigger a similar flow of events.

Visualizing using Keel dashboard:

You can access the dashboard at External-IP:9300 or by using the NodePort. Use the same credentials which you had set while setting up the keel.
You can -

• View resources managed by keel
• Get an audit of the changes done by keel
• Change/Pause update policies of resources
• Approve updates

What's next?

Check out the keel documentation to explore more features.

• Enabling approvals for udpates
• Setting up notification pipelines
• Supported webhook triggers and polling
• Use helm templating for update
• Updating DaemonSets, StatefulSets, etc.

Here are all the workflow and deployment manifests used.