DEV Community: Eduardo Lomeli

Kamal: Speed up the image builds using managed third-party builders and GitHub Actions

Eduardo Lomeli — Sun, 10 Nov 2024 08:13:58 +0000

In some cases it is more cost-effective to use a managed third-party builder to speed up your image building process in a Kamal deployment. There are new products that offer powerful optimized builders, such as:

If you are interested in setup and tune your own builder, there are great examples in Kamal docs

In this guide, I'm going to cover how to use and configure Docker Build Cloud inside GitHub Actions and deploy the image using Kamal.

Prerequisites

Create an account in docker.com
Setup Docker Build Cloud in your account

GitHub Action

Note the configuration in docker/setup-buildx-action@v3

name: Workflow - Deployment

on:
  push:
    branches:
      - main

jobs:
  deploy:
    name: kamal deploy
    timeout-minutes: 5
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true

      - uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}

      - name: Log in to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.KAMAL_REGISTRY_USERNAME }}
          password: ${{ secrets.KAMAL_REGISTRY_PASSWORD }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          version: "lab:latest"
          driver: cloud
          endpoint: "your-org/builder-name"
          install: true

      - run: gem install kamal

      - run: kamal lock release

      - run: kamal deploy

Kamal configuration

You need to configure docker builder driver in order to pickup the docker environment builder defaults.

builder:
  arch: amd64
  driver: docker

Kamal: How to use Grafana Loki as a logging driver

Eduardo Lomeli — Sun, 10 Nov 2024 07:14:27 +0000

You can easily configure Grafana Loki as a logging driver in your Kamal deployments.

In this guide I'm using Grafana Cloud and Loki Docker driver client.

Install Loki Docker driver in your Kamal hosts

This step can be automated using Kamal docker-setup hook. Which runs right after Docker is installed, usually when you run kamal setup.

`docker-setup`

This script example is prepared to use multiple destinations.

File location: .kamal/hooks/docker-setup

#!/bin/sh

echo "Docker set up on $KAMAL_HOSTS. Destination $KAMAL_DESTINATION"

# Check if the Loki Docker Driver plug-in is already installed
if ! kamal server exec "docker plugin ls" -d $KAMAL_DESTINATION | grep -q "loki"; then
  echo "Installing Loki Docker Driver plug-in..."
  kamal server exec \
  "docker plugin install grafana/loki-docker-driver:2.9.2 --alias loki --grant-all-permissions" \
  -d $KAMAL_DESTINATION
else
  echo "Loki Docker Driver plug-in is already installed."
fi

Configure Kamal logging driver

`deploy.yaml`

logging:
  driver: loki
  options:
    loki-url: https://<user_id>:<password>@logs-us-west1.grafana.net/loki/api/v1/push
    loki-batch-size: "400"

Kamal: How to integrate with GitHub Actions using multiple destinations

Eduardo Lomeli — Sun, 10 Nov 2024 06:28:53 +0000

The recently introduced Kamal tool is very powerful and easy to use, you can achieve full deployment automation when it is integrated into the CI/CD pipelines. I'm going to walk you through how to run it automatically on GitHub Actions with multiple destinations (eg. staging and production).

Pipeline workflows:

Event	Action
Push to `main` (PR merged)	Desploy to Staging
Publish release (Tag created)	Deploy to Production

Requirements

Setup your app secrets in your repo for both environments (staging and production) in this tutorial they are prefixed by STG_ and PROD_.
Make sure you have created SSH_PRIVATE_KEY, KAMAL_REGISTRY_USERNAME and KAMAL_REGISTRY_PASSWORD secrets.

GitHub Action files

.github/
├─ workflows/
   ├─ job-deploy-kamal.yml
   ├─ workflow-deploy-production.yml
   ├─ workflow-deploy-staging.yml

job-deploy-kamal.yml is the shared job where Kamal runs
workflow-deploy-production.yml acts as a trigger for production
workflow-deploy-staging.yml acts as a trigger for staging

`job-deploy-kamal.yml`

name: Job - Deploy to Kamal

on:
  workflow_call:
    inputs:
      kamal-destination:
        required: true
        type: string
    secrets:
      KAMAL_REGISTRY_USERNAME:
        required: true
      KAMAL_REGISTRY_PASSWORD:
        required: true
      SSH_PRIVATE_KEY:
        required: true
      DATABASE_URL:
        required: true
      QUEUE_DATABASE_URL:
        required: true
      SECRET_KEY_BASE:
        required: true

env:
  DOCKER_BUILDKIT: 1
  KAMAL_REGISTRY_USERNAME: ${{ secrets.KAMAL_REGISTRY_USERNAME }}
  KAMAL_REGISTRY_PASSWORD: ${{ secrets.KAMAL_REGISTRY_PASSWORD }}
  # App secrets
  DATABASE_URL: ${{ secrets.DATABASE_URL }}
  QUEUE_DATABASE_URL: ${{ secrets.QUEUE_DATABASE_URL }}
  SECRET_KEY_BASE: ${{ secrets.SECRET_KEY_BASE }}

jobs:
  deploy:
    name: kamal deploy
    timeout-minutes: 10
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true

      - uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}

      - name: Log in to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.KAMAL_REGISTRY_USERNAME }}
          password: ${{ secrets.KAMAL_REGISTRY_PASSWORD }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - run: gem install kamal

      - run: kamal lock release -d ${{ inputs.kamal-destination }}

      - run: kamal deploy -d ${{ inputs.kamal-destination }}

`workflow-deploy-staging.yml`

name: Workflow - Deployment (staging)

on:
  push:
    branches:
      - main

concurrency:
  group: deployment-staging
  cancel-in-progress: false

jobs:
  deploy:
    uses: ./.github/workflows/job-deploy-kamal.yml
    with:
      kamal-destination: staging
    secrets:
      KAMAL_REGISTRY_USERNAME: ${{ secrets.KAMAL_REGISTRY_USERNAME }}
      KAMAL_REGISTRY_PASSWORD: ${{ secrets.KAMAL_REGISTRY_PASSWORD }}
      SSH_PRIVATE_KEY: ${{ secrets.STG_SSH_PRIVATE_KEY }}
      DATABASE_URL: ${{ secrets.STG_DATABASE_URL }}
      QUEUE_DATABASE_URL: ${{ secrets.STG_QUEUE_DATABASE_URL }}
      SECRET_KEY_BASE: ${{ secrets.STG_SECRET_KEY_BASE }}

`workflow-deploy-production.yml`

name: Workflow - Deployment (production)

on:
  release:
    types: [published]

concurrency:
  group: deployment-production
  cancel-in-progress: false

jobs:
  deploy:
    uses: ./.github/workflows/job-deploy-kamal.yml
    with:
      kamal-destination: production
    secrets:
      KAMAL_REGISTRY_USERNAME: ${{ secrets.KAMAL_REGISTRY_USERNAME }}
      KAMAL_REGISTRY_PASSWORD: ${{ secrets.KAMAL_REGISTRY_PASSWORD }}
      SSH_PRIVATE_KEY: ${{ secrets.PROD_SSH_PRIVATE_KEY }}
      DATABASE_URL: ${{ secrets.PROD_DATABASE_URL }}
      QUEUE_DATABASE_URL: ${{ secrets.PROD_QUEUE_DATABASE_URL }}
      SECRET_KEY_BASE: ${{ secrets.PROD_SECRET_KEY_BASE }}

Kamal files

.kamal/
├─ secrets-common
config/
├─ deploy.production.yml
├─ deploy.staging.yml
├─ deploy.yml

`secrets-common`

KAMAL_REGISTRY_USERNAME=$KAMAL_REGISTRY_USERNAME
KAMAL_REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD
DATABASE_URL=$DATABASE_URL
QUEUE_DATABASE_URL=$QUEUE_DATABASE_URL
SECRET_KEY_BASE=$SECRET_KEY_BASE

`deploy.yml`

service: myapp
image: myorg/myapp

registry:
  username:
    - KAMAL_REGISTRY_USERNAME
  password:
    - KAMAL_REGISTRY_PASSWORD

proxy: 
  ssl: true
  app_port: 3000

builder:
  arch: amd64

volumes:
  - "app_storage:/app/storage"

`deploy.production.yml`

servers:
  web:
    - 192.168.0.1
  job:
    hosts:
      - 192.168.0.1
    cmd: bin/jobs

proxy: 
  host: api.myapp.com

env:
  clear:
    RAILS_LOG_TO_STDOUT: 1
    WEB_CONCURRENCY: 2
    JOB_CONCURRENCY: 1
    SOLID_QUEUE_IN_PUMA: true
  secret:
    - DATABASE_URL
    - QUEUE_DATABASE_URL
    - SECRET_KEY_BASE

`deploy.staging.yml`

servers:
  web:
    - 192.168.0.2
  job:
    hosts:
      - 192.168.0.2
    cmd: bin/jobs

proxy: 
  host: api-staging.myapp.com

env:
  clear:
    RAILS_LOG_TO_STDOUT: 1
    WEB_CONCURRENCY: 0
    JOB_CONCURRENCY: 1
    SOLID_QUEUE_IN_PUMA: true
  secret:
    - DATABASE_URL
    - QUEUE_DATABASE_URL
    - SECRET_KEY_BASE