DEV Community: Alugbin Abiodun Olutola

My Simple Github Actions CI/CD Pipeline:

Alugbin Abiodun Olutola — Wed, 08 Jun 2022 00:00:52 +0000

A little story of how I combined 2 repositories with github actions' repository_dispatch webhook to manage my deployment pipeline.

Background.

I have always wanted to deploy a new image to kubernetes once my build is build is successful, but I always need to update the deployment file to achieve this.

I started using ArgoCD recently, and I enjoy the simplicity and the insights I get from the UI. Only problem was that I need to initialize it from my github repo. My repository is private, got some challenges with that, but that's for another time.

Challenge

The challenge here is that I want my deployment repository to be separate from my application (app) repository (repo). Also, because I am using branches in ArgoCD, I don't get to run my tests or build my image before the changes get picked up by ArgoCD. For these reasons, it makes sense to separate the repos and find a way to notify the deployment repo once the app repo has completed all the checks and the docker image is built, we can then proceed to communicate with the deployment repo, sending the new image tag over. The deployment repo then updates the image in kubernetes deployment file, checks out the code and pushes. ArgoCD picks up the new change and syncs automagically.
Enough Talks, here is the actions code.

Application

Application was written in go, but this can be swapped out for other services as well.

app_repo/.github/workflows/ci.yml

name: Test and Build

on:
  push:
    branches: [ master, develop, staging, testing ]
  pull_request:
    branches: [ master ]
jobs:
  lint:
    name: Linting
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2

      - name: Lint and Vet
        uses: golangci/golangci-lint-action@v2
        with:
          version: latest
          args: --timeout=3m
  test:
    name: Test
    runs-on: ubuntu-18.04
    services:
      mysql:
        image: mysql:5.7
        env:
          MYSQL_DATABASE: app_name
          MYSQL_USER: cicd-user
          MYSQL_PASSWORD: password
          MYSQL_ROOT_PASSWORD: password
        ports:
          - 3306:3306
    steps:
      - uses: actions/checkout@v2

      - name: Set up Go
        uses: actions/setup-go@v2
        with:
          go-version: 1.17

      - name: Test
        run: go test ./...
        env:
          DB_HOST: 127.0.0.1
          DB_PORT: 3306
          ENVIRONMENT: ci-cd
          DB_USERNAME: cicd-user
          DB_PASSWORD: password
          DATABASE: app_name
          TOKEN_STRING: "hello hello world"

  build:
    name: Build Image
    runs-on: ubuntu-latest
    needs:
      - test
      - lint
    steps:
      - name: Extract branch name
        shell: bash
        run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/})"
        id: extract_branch

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to Dockerhub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Build and Push
        uses: docker/build-push-action@v2
        with:
          file: .docker/Dockerfile
          push: true
          tags: <<docker_repo>>/<<image_name>>:${{ steps.extract_branch.outputs.branch }}-${{ github.run_id }}-${{ github.run_number }}


  deploy:
    name: Trigger New Deployment
    runs-on: ubuntu-latest
    needs:
      - build
    steps:
      - name: Extract branch name
        shell: bash
        run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/})"
        id: extract_branch

      - name: Deployment
        uses: mvasigh/dispatch-action@main
        with:
          token: ${{ secrets.G_ACCESS_TOKEN }}
          repo: <<deploy_repo>>
          owner: <<owner>>
          event_type: update_image
          message: |
            {
              "branch": "${{ steps.extract_branch.outputs.branch }}",
              "tag": "${{ steps.extract_branch.outputs.branch }}-${{ github.run_id }}-${{ github.run_number }}"
            }

Deployment.

Here, I keep a folder for each environment:

production
staging
testing

There is a stubs folder called stubs with this sample file for each environment:

./stubs/environment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <<name>>
  namespace: <<environment>>
spec:
  replicas: 2
  selector:
    matchLabels:
      tier: backend
      app: <<app_name>>
  template:
    metadata:
      labels:
        tier: backend
        app: <<app_name>>
    spec:
      containers:
        - name: <<app_name>>
          image: <<docker_repo>>/<<image>>:#TAG#
          imagePullPolicy: Always
          envFrom:
            - configMapRef:
                name: <<app_config>>
            - secretRef:
                name: <<app_secret>>

PS: all data between << and >> should be swapped with their actual value. Only #TAG should be the placeholder.

deploy_repo/.github/workflows/deploy.yml

name: Deploy
on:
  repository_dispatch:
    branches: ["master"]
    types: [update_image]

jobs:
  build:
    name: Run API
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
        with:
          ref: ${{github.event.client_payload.message.branch}}

      - name: Update Stub With New Tag
        shell: bash
        run: |
          cp stubs/${{ github.event.client_payload.message.branch }}.yaml ${{ github.event.client_payload.message.branch }}/deployment.yaml.stub
          sed -i "s/#TAG#/${{ github.event.client_payload.message.tag }}/g" ${{ github.event.client_payload.message.branch }}/deployment.yaml.stub
          cp ${{github.event.client_payload.message.branch}}/deployment.yaml.stub ${{github.event.client_payload.message.branch}}/deployment.yaml

      - name: Commit files
        run: |
          git config --local user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git config --local user.name "github-actions[bot]"
          git commit -m "Update Tag ${{ github.event.client_payload.message.tag }}" -a

      - name: Push changes
        uses: ad-m/github-push-action@master
        with:
          github_token: ${{ secrets.G_ACCESS_TOKEN }}
          branch: ${{ github.event.client_payload.message.branch }}        

      - name: All Done
        run: |
          echo "All is Well"

Here is a breakdown of the deploy.yml action:

Checks out the repo using the provided branch
Copies the environment stub into a deployment.yaml.stub file.
Updates the #TAG# placeholder with the new tag from the repository_dispatch
Commit the files using the bot email and using the latest tag as the commit message. This allows us to compare the sync in argocd
Push the changes using ad-m/github-push-action.
Congratulatory message from me :)

Please if there is an easier way to achieve this, please share in the comment. I will love to learn how to make my life easier.

Creating an NFS Server with Vagrant and Archlinux for Kubernetes Cluster

Alugbin Abiodun Olutola — Mon, 06 Apr 2020 02:29:18 +0000

Disclaimer: This post is originally meant to house my process for creating an NFS server (it was a huge toil). As such a lot of things might be omitted or assumed. If you find it useful and you need more insight or questions, please feel free to drop a comment and ask your question.

Also, there are several cloud providers for storage online. Almost if not all major cloud providers have the service available. However in the spirit of curiosity and how is it done, I decided to create mine.
(Also, it's a pet project and might not even see the light of the day so why waste $)

That said, let's move on.

While learning kubernetes recently from here, I began undertaking the task of migrating one of my laravel pet projects to kubernetes.

Everything was fine, until I got to the point where I needed to persist some of my essential files (logs,database,images etc).

First, I provision my k8s cluster using vagrant with the knowledge I learnt from here. Then I added an extra machine to the Vagrantfile to provision an archlinux box as well.

Here is the NFS server box:

#filename=Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |cf|
  # NFS Server
  cf.vm.define "nfs-server" do |nfs|
    nfs.vm.box = "archlinux/archlinux"
    nfs.vm.hostname = "nfs-server.example.com"
    nfs.vm.network "private_network", ip: "172.42.42.99"
    nfs.vm.provider "virtualbox" do |n|
      n.name = "nfs-server"
      n.memory = 1024
      n.cpus = 1
    end
    nfs.vm.provision "shell",path: "bootstrap_nfs.sh"
  end
end

This box defines a machine named nfs-server with the following properties:

OS: archlinux official image from vagrant box collection
Hostname: "nfs-server.example.com", This machine can be reached by this name within the network as well.
IPAddress: The entire cluster runs on a private address, and since I will like this to be available to the cluster alone, I add it to the same address space.
Provider: I use virtualbox, various options exists as defined in official vagrant documentation. Various providers are listed here
Name: uniquely identify the box with the list of machines (I have a lot of them for my k8s cluster)
Memory: Assigned memory to this machine. 1Gb in this case
cpus: Just 1, it can be increased for more performance though.

bootstrap_nfs.sh is the script that is used to provision the machine.

#filename=bootstrap_nfs.sh

# Update hosts file
echo "[TASK 1] Update /etc/hosts file"
cat >>/etc/hosts<<EOF
172.42.42.99 nfs-server.example.com nfs-server
172.42.42.100 lmaster.example.com lmaster
172.42.42.101 lworker1.example.com lworker1
172.42.42.102 lworker2.example.com lworker2
EOF

echo "[TASK 2] Download and install NFS server"
yes| sudo pacman -S nfs-utils

echo "[TASK 3] Create a kubedata directory"
mkdir -p /srv/nfs/kubedata
mkdir -p /srv/nfs/kubedata/db
mkdir -p /srv/nfs/kubedata/storage
mkdir -p /srv/nfs/kubedata/logs

echo "[TASK 4] Update the shared folder access"
chmod -R 777 /srv/nfs/kubedata

echo "[TASK 5] Make the kubedata directory available on the network"
cat >>/etc/exports<<EOF
/srv/nfs/kubedata    *(rw,sync,no_subtree_check,no_root_squash)
EOF

echo "[TASK 6] Export the updates"
sudo exportfs -rav

echo "[TASK 7] Enable NFS Server"
sudo systemctl enable nfs-server

echo "[TASK 8] Start NFS Server"
sudo systemctl start nfs-server

I added the Task number with a description to show what is being done and also as a documentation.

The hosts needs to be updated so that the nfs server can reach the cluster nodes. Each node will also have the corresponding entry, it's just a way of finding each other within the private network.

On to my PersistentVolume definition on the cluster,

#filename=storage_volume.yml
apiVersion: v1
kind: PersistentVolume
metadata:
  namespace: app
  name: app-storage-pv
  labels:
    tier: storage
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
  storageClassName: app-storage-pv
  mountOptions:
    - nfsvers=4.1
  nfs:
    path: "/srv/nfs/kubedata/storage"
    server: nfs-server.example.com
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-storage-pvc
  namespace: app
  labels:
    tier: storage
    app: app-storage
spec:
  storageClassName: app-storage-pv
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

I created a PV that maps to the exposed folder /srv/nfs/kubedata/storage and using the FQDN within the network, I can use nfs-server.example.com for my server name (please ping this from one of your nodes to ensure it is reachable).

#filename=logs_volume.yml
apiVersion: v1
kind: PersistentVolume
metadata:
  namespace: app
  name: app-logs-pv
  labels:
    tier: storage
    app: app-logs
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
  storageClassName: app-logs-pv
  mountOptions:
    - nfsvers=4.1
  nfs:
    path: "/srv/nfs/kubedata/logs"
    server: nfs-server.example.com
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-logs-pvc
  namespace: app
  labels:
    tier: storage
    app: app-logs
spec:
  storageClassName: app-logs-pv
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

This creates a PersistentVolume mapping to path /srv/nfs/kubedata/logs on our NFS server using the same server details nfs-server.example.com.

Now comes our deployment file

#filename=deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-backend
  namespace: app
  labels:
    app: app-backend
    tier: backend

spec:
  selector:
    matchLabels:
      app: app-backend
  template:
    metadata:
      labels:
        app: app-backend
        tier: backend

    spec:
      containers:
        - name: app-backend
          image: image_name
          env:
            - name: DB_PASSWORD
              value: "password"
          ports:
            - containerPort: 80
          livenessProbe:
            tcpSocket:
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 15
          readinessProbe:
            tcpSocket:
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 10
          volumeMounts:
            - mountPath: /app/storage/app
              name: quest-storage
            - mountPath: /app/storage/logs
              name: app-logs

      volumes:
        - name: app-storage
          persistentVolumeClaim:
            claimName: app-storage-pvc
        - name: app-logs
          persistentVolumeClaim:
            claimName: app-logs-pvc
---

apiVersion: v1
kind: Service
metadata:
  name: app-backend
  namespace: app
  labels:
    tier: backend
spec:
  selector:
    app: app-backend
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 80

In Laravel, the storage folder is mostly used to house file uploads and other generated files. official docs

Given pods are mortal you don't want to lose customer's profile pictures , image uploads or any other essential file when your pod(s) go down (they do that often), we mount the nfs /srv/kubedata/storage/ of the nfs server to Laravel storage/app on our pod.

Application logs are also super important, Laravel uses monolog and you don't want your log file to go down with each pod hence we also mount /srv/nfs/kubedata/logs to Laravel's storage/logs/.

Once this mount is successful, you can always be sure that even if your cluster goes down, as long as your NFS server is still up and running, your files will be safe and you can bring your cluster back to life without changing a thing.

PS: There might be a better way to mount the entire storage folder, but I didn't want to add some files that are not needed eg, the view cache files. This might be useful for some other people though; and it might make sense to add them to the mount.

Again, there are various cloud provider options that can be used and kubernetes supports most of them, this is just to get my pet project up and running in kubernetes and to also add another feather to my cap during this lockdown.

Go and explore!!!

Thank you!