DEV Community: Yash Thakkar

Setup CI/CD for Serverless monorepos application on AWS using GitHub Actions

Yash Thakkar — Sun, 29 Aug 2021 20:39:54 +0000

There are many ways to write server applications in many languages. There are many frameworks to help us building the server like Golang with mux, Java with Spring Boot, NodeJS with express etc. When it comes to hosting the server application and scaling the system, it requires lots of efforts, planning. Serverless is a cloud-native development model that allows developers to build and run applications without having to manage servers. So using serverless architecture model, we don't need to worry about provisioning-managing-scaling the servers, updating the security patches, someone hacking into our server and much more. These all will be taken care by the cloud provider. So we can say that, we should try to use serverless architecture for API wherever possible.

In this post, we will be talking about running JavaScript functions as an api application.

AWS Lambda

AWS Lambda is a serverless compute service which we will be using to deploy our JS functions. We just upload our code as a ZIP file and Lambda automatically and precisely allocates compute execution power and runs your code based on the incoming request or event, for any scale of traffic.

There are many frameworks available to write NodeJS serverless application like Architect, Up, Middy and many more. We will be using Serverless Framework to write our backend API application because we can use multiple programming languages and we will be using multiple other AWS services like S3, API Gateway and Serverless framework has our of the box support for it.

Serverless Framework

The Serverless Framework helps you develop and deploy your AWS Lambda functions, along with the AWS infrastructure resources they require. It's a CLI that offers structure, automation and best practices out-of-the-box, allowing you to focus on building sophisticated, event-driven, serverless architectures, comprised of Functions and Events.

Setup

Prerequisites

NodeJS: >= 12
Serverless CLI: Install command npm install -g serverless

We will be writing simple NodeJS application with below file/folder structure or you can run serverless command and setup a new project from template as well. You can clone the demo code from https://github.com/thakkaryash94/aws-serverless-ci-cd-demo.

.
├── handler.js
├── package-lock.json
├── package.json
└── serverless.yml

handler.js

'use strict';

module.exports.main = async (event, context) => {
  return {
    statusCode: 200,
    body: JSON.stringify(
      {
        message: 'Hello from new service A!',
        input: event,
      },
      null,
      2
    ),
  };
};

We will be updating our serverless.yml file as below. We will be using AWS S3 to store the function zip and AWS API Gateway service to access our function as an API URL.

serverless.yml

service: servicea

frameworkVersion: "2"

provider:
  name: aws
  runtime: nodejs14.x
  lambdaHashingVersion: 20201221
  region: ${env:AWS_REGION}
  apiGateway:
    restApiId: ${env:AWS_REST_API_ID}
    restApiRootResourceId: ${env:AWS_REST_API_ROOT_ID}
  # delete below section if you don't want to keep the Lambda function zip in a bucket
  deploymentBucket:
    blockPublicAccess: true # Prevents public access via ACLs or bucket policies. Default is false
    skipPolicySetup: false # Prevents creation of default bucket policy when framework creates the deployment bucket. Default is false
    name: ${env:AWS_BUCKET_NAME} # Deployment bucket name. Default is generated by the framework
    maxPreviousDeploymentArtifacts: 10 # On every deployment the framework prunes the bucket to remove artifacts older than this limit. The default is 5

functions:
  main:
    handler: handler.main # Function name
    memorySize: 128
    events:
      - http:
          path: servicea # URL path to access the function
          method: get # Method name for API gateway

Now, you can run below command to execute the function locally. It will give you warning about missing environment variables like AWS_REGION, AWS_BUCKET_NAME etc but we can ignore them or you can comment them as well for the local development purpose.

$ serverless invoke local -f main

It will return the response as below.

{
    "statusCode": 200,
    "body": "{\n  \"message\": \"Hello from new service A!\",\n  \"input\": \"\"\n}"
}

So it means our serverless application is ready and working correctly locally. For the real project, we will need many of these applications. So each application will serve on individual API request.

We keep our code on VCS providers like GitHub, GitLab, BitBucket etc. The problem is it will be very difficult to maintain many repositories like below even for a single project. So to fix that, we can store all these applications in one repository and it will be called as Monorepo. That's the difference between monorepo and multirepo.

Setup Monorepo

To convert our serverless, we just need to create a folder eg. aws-serverless-ci-cd-demo and move the folder services inside it.Now, we can have as many functions as we need for our project and they all will be inside a single repository.

Final overall structure will look like below.

aws-serverless-ci-cd-demo
├── README.md
├── servicea
│   ├── handler.js
│   ├── package-lock.json
│   ├── package.json
│   └── serverless.yml
└── serviceb
    ├── handler.js
    ├── package-lock.json
    ├── package.json
    └── serverless.yml

We will be using AWS Lambda, AWS S3 and API Gateway to deploy and access our services. So let's discuss about the use of AWS S3 and API Gateway.

AWS S3

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

We will store our Lambda function code as a zip in the bucket. The advantage is we can keep the track of Lambda functions with code and date-time. It is a totally optional step. You can skip it by commenting, deleting the deploymentBucket code in serverless.yml file.

AWS API Gateway

Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. APIs act as the "front door" for applications to access data, business logic, or functionality from your backend services.

We will use API Gateway to expose our lambda function to a url so that we can access it. Serverless framework will use the path, method from http section and setup route based on it. In the demo, it will create a /servicea route with GET method. Serverless framework will map this route with our Lambda function.

CI/CD Deployment

The "CI" in CI/CD refers to continuous integration, which is an automation process to built, tested the code when developers push the code. In our application, creating a zip file of the functions.

The "CD" in CI/CD refers to continuous delivery and/or continuous deployment, which means deliver/deploy the code to the development, staging, uat, qa, production environment. In our application, it means deploying the zip to Lambda function and configuring API Gateway.

Fortunetly, Serverless framework has inbuild support for it. So all we need to do is proving aws credentials as environment variables like AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY. We will need few more variables because we are using other services like S3 and API Gateway.

We will be using GitHub actions to build, package and deploy our functions. So let's see how we can implement CI/CD using GitHub actions.

GitHub Actions

GitHub Actions help you automate tasks within your software development life cycle. GitHub Actions are event-driven, meaning that you can run a series of commands after a specified event has occurred. For example, every time someone creates a pull request for a repository, you can automatically run a command that executes a software testing script.

We can configure GitHub actions based on multiple triggers like on push, pull_request etc and as per different branches as well.

There are 2 ways to deploy the serverless functions.

Auto Deployment

Trigger setup

  name: Auto Serverless Deployment
  on: [push, pull_request]

The first job is to detech the changes in repo files and folders and return the list. Here we will use git commit diff between current and last commit and use a filter which will return only added or updated files list. Then we will go through each file, collect only unique folders name and return as output.

  jobs:
    changes:
      name: Changes
      runs-on: ubuntu-latest
      outputs:
        folders: ${{ steps.filter.outputs.folders }}
      steps:
        - uses: actions/checkout@v2
        - name: Check changed files
          id: diff
          run: |
            if [ $GITHUB_BASE_REF ]; then
              # Pull Request
              git fetch origin $GITHUB_BASE_REF --depth=1
              export DIFF=$( git diff --name-only origin/$GITHUB_BASE_REF $GITHUB_SHA )
              echo "Diff between origin/$GITHUB_BASE_REF and $GITHUB_SHA"
            else
              # Push
              git fetch origin ${{ github.event.before }} --depth=1
              export DIFF=$( git diff --diff-filter=d --name-only ${{ github.event.before }} $GITHUB_SHA )
              echo "Diff between ${{ github.event.before }} and $GITHUB_SHA"
            fi
            echo "$DIFF"
            # Escape newlines (replace \n with %0A)
            echo "::set-output name=diff::$( echo "$DIFF" | sed ':a;N;$!ba;s/\n/%0A/g' )"
        - name: Set matrix for build
          id: filter
          run: |
            DIFF="${{ steps.diff.outputs.diff }}"

            if [ -z "$DIFF" ]; then
              echo "::set-output name=folders::[]"
            else
              JSON="["
              # Loop by lines
              while read path; do
                # Set $directory to substring before /
                directory="$( echo $path | cut -d'/' -f1 -s )"

              # ignore .github folder
              if [[ "$directory" != ".github" ]]; then
                # Add build to the matrix only if it is not already included
                JSONline="\"$directory\","
                if [[ "$JSON" != *"$JSONline"* ]]; then
                  JSON="$JSON$JSONline"
                fi
              fi
              done <<< "$DIFF"

              # Remove last "," and add closing brackets
              if [[ $JSON == *, ]]; then
                JSON="${JSON%?}"
              fi
              JSON="$JSON]"
              echo $JSON

              # Set output
              echo "::set-output name=folders::$( echo "$JSON" )"
            fi

After adding a serverless service servicea, the build action will print something like below.

Check changed files

  Diff between 3da227687e19da14062916c6f71cef0c7e3f9033 and 96a8e3a39ab79ccff3a294ea485c4c3854d496c6
  servicea/.gitignore
  servicea/handler.js
  servicea/package-lock.json
  servicea/package.json
  servicea/serverless.yml

Set matrix for build

  ["servicea"]

Next, we will create a job for every folder name using matrix strategy as below.

    deploy:
      needs: changes
      name: Deploy
      if: ${{ needs.changes.outputs.folders != '[]' && needs.changes.outputs.folders != '' }}
      strategy:
        matrix:
          # Parse JSON array containing names of all filters matching any of changed files
          # e.g. ['servicea', 'serviceb'] if both package folders contains changes
          folder: ${{ fromJSON(needs.changes.outputs.folders) }}

Now, it's time to build and deploy the function. Please define all the env variables used in serverless.yml file in env section as below. So here, we will go through every folder and run npx serverless deploy. This command will create a zip, update it to S3, create/update the Lambda and finally configuring it with API Gateway.

      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@v2
        - name: Configure AWS Credentials
          uses: aws-actions/configure-aws-credentials@v1
          with:
            aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
            aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
            aws-region: ${{ secrets.AWS_REGION }}
        - name: deploy
          run: npx serverless deploy
          working-directory: ${{ matrix.folder }}
          env:
            AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
            AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
            AWS_REST_API_ROOT_ID: ${{ secrets.AWS_REST_API_ROOT_ID }}
            AWS_REST_API_ID: ${{ secrets.AWS_REST_API_ID }}
            AWS_BUCKET_NAME: ${{ secrets.AWS_BUCKET_NAME }}

Before we push the code and action start to build and deploy the code, we need to add below secret environment variables. You should never use root account and always create a new user with restricted permissions based on use cases. In this process, our user will need access to Lambda, S3 write access and API Gateway access.

  - AWS_ACCESS_KEY_ID
  - AWS_SECRET_ACCESS_KEY
  - AWS_REGION
  - AWS_REST_API_ROOT_ID
  - AWS_REST_API_ID
  - AWS_BUCKET_NAME: bucket name where we want our zip files to be stored, if you are ignoring `deploymentBucket` from `serverless.yml` file, you can ignore this variable as well.

auto.yml

name: Auto Serverless Deployment
on: [push, pull_request]

jobs:
  changes:
    name: Changes
    runs-on: ubuntu-latest
    outputs:
      folders: ${{ steps.filter.outputs.folders }}
    steps:
      - uses: actions/checkout@v2
      - name: Check changed files
        id: diff
        run: |
          if [ $GITHUB_BASE_REF ]; then
            # Pull Request
            git fetch origin $GITHUB_BASE_REF --depth=1
            export DIFF=$( git diff --name-only origin/$GITHUB_BASE_REF $GITHUB_SHA )
            echo "Diff between origin/$GITHUB_BASE_REF and $GITHUB_SHA"
          else
            # Push
            git fetch origin ${{ github.event.before }} --depth=1
            export DIFF=$( git diff --diff-filter=d --name-only ${{ github.event.before }} $GITHUB_SHA )
            echo "Diff between ${{ github.event.before }} and $GITHUB_SHA"
          fi
          echo "$DIFF"
          # Escape newlines (replace \n with %0A)
          echo "::set-output name=diff::$( echo "$DIFF" | sed ':a;N;$!ba;s/\n/%0A/g' )"
      - name: Set matrix for build
        id: filter
        run: |
          DIFF="${{ steps.diff.outputs.diff }}"

          if [ -z "$DIFF" ]; then
            echo "::set-output name=folders::[]"
          else
            JSON="["
            # Loop by lines
            while read path; do
              # Set $directory to substring before /
              directory="$( echo $path | cut -d'/' -f1 -s )"

            # ignore .github folder
            if [[ "$directory" != ".github" ]]; then
              # Add build to the matrix only if it is not already included
              JSONline="\"$directory\","
              if [[ "$JSON" != *"$JSONline"* ]]; then
                JSON="$JSON$JSONline"
              fi
            fi
            done <<< "$DIFF"

            # Remove last "," and add closing brackets
            if [[ $JSON == *, ]]; then
              JSON="${JSON%?}"
            fi
            JSON="$JSON]"
            echo $JSON

            # Set output
            echo "::set-output name=folders::$( echo "$JSON" )"
          fi
  deploy:
    needs: changes
    name: Deploy
    if: ${{ needs.changes.outputs.folders != '[]' && needs.changes.outputs.folders != '' }}
    strategy:
      matrix:
        # Parse JSON array containing names of all filters matching any of changed files
        # e.g. ['servicea', 'serviceb'] if both package folders contains changes
        folder: ${{ fromJSON(needs.changes.outputs.folders) }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}
      - name: deploy
        run: npx serverless deploy
        working-directory: ${{ matrix.folder }}
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_REST_API_ROOT_ID: ${{ secrets.AWS_REST_API_ROOT_ID }}
          AWS_REST_API_ID: ${{ secrets.AWS_REST_API_ID }}
          AWS_BUCKET_NAME: ${{ secrets.AWS_BUCKET_NAME }}

Manual

Sometimes, we want to deploy a function manually or it may be skipped by above script, we can deploy it manually because deploying it is more important than finding the issue at that time.

Here, we will skip the step to identify the files using git diff and returning the folder. We can directly go inside the function folder name and run the deployment command eg. npx serverless deploy.

For manual deployment, we will take the function name as action input and deploy the specific function rather than deploying manually.

  name: Manual Serverless Deployment
  on:
    push:
      branches:
        - main
    pull_request:
      branches:
        - main
    workflow_dispatch:
      inputs:
        function:
          description: "Function name"
          required: true

After this, we will use it in our job as below.

  jobs:
    deploy:
      if: ${{ github.event_name == 'workflow_dispatch' }}
      name: deploy
      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@master
        - name: deploy
          run: npx serverless deploy
          working-directory: ${{ github.event.inputs.function }}
          env:
            AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
            AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
            AWS_REST_API_ROOT_ID: ${{ secrets.AWS_REST_API_ROOT_ID }}
            AWS_REST_API_ID: ${{ secrets.AWS_REST_API_ID }}
            AWS_BUCKET_NAME: ${{ secrets.AWS_BUCKET_NAME }}

manual.yml

name: Manual Serverless Deployment
on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main
  workflow_dispatch:
    inputs:
      function:
        description: "Function name"
        required: true
jobs:
  deploy:
    if: ${{ github.event_name == 'workflow_dispatch' }}
    name: deploy
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
      - name: deploy
        run: npx serverless deploy
        working-directory: ${{ github.event.inputs.function }}
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_REST_API_ROOT_ID: ${{ secrets.AWS_REST_API_ROOT_ID }}
          AWS_REST_API_ID: ${{ secrets.AWS_REST_API_ID }}
          AWS_BUCKET_NAME: ${{ secrets.AWS_BUCKET_NAME }}

Here, we saw, how we can setup auto, manual deployment for serverless application with CI/CD.

Postgres backup using Docker

Yash Thakkar — Sun, 17 Jan 2021 21:07:28 +0000

Postgres is one of the most popular open-source Relational database. You can read more about it here. The purpose of this blog to explain how you can take a backup of your Postgres database running with and without docker using the docker image and why I have created this image.

Problem

Running Postgres inside the docker is very easy. Docker hub has an official postgres image that we can run. With a single command, we can start using Postgres database. The issue happens when we want to run a backup with cron. There are many ways to do this. We can follow official documentation as well to automate the backup on Linux. The advantage of using docker is flexibility and no platform dependency. The official document only shows us how to run cron job on Linux. This will add the os level of dependency that we wanted to ignore. When searching for the backup docker image, I was not able to find any Docker image that supports Postgres 13 with S3 backup support. Existing images are not up-to-date.

Solution

To tackle the problem, I have created my own docker image. To start the backup, follow the below instruction.

Required Environment Variables

All the environment variables from https://www.postgresql.org/docs/current/libpq-envars.html are supported because we are using native postgres-client binary for backup.

PGHOST: behaves the same as the host connection parameter. eg. postgresql
PGHOSTADDR: behaves the same as the hostaddr connection parameter. This can be set instead of or in addition to PGHOST to avoid DNS lookup overhead.
PGHOST: postgresql
PGPORT: 5432
PGDATABASE: database
PGUSER: postgres
PGPASSWORD: password
S3_HOST: https://storage.googleapis.com || s3.eu-west-1.amazonaws.com || nyc3.digitaloceanspaces.com
S3_BUCKET: BUCKET
S3_ACCESS_KEY: ACCESS_KEY
S3_SECRET_KEY: SECRET_KEY
CRON_SCHEDULE: "* * * * *". Read more https://crontab.guru/

Docker Run Command

docker run -d \
      --name postgres-backup \
      -v $(pwd)/backups:/backups \
      -e PGHOST=postgresql
      -e PGPORT=5432
      -e PGDATABASE=db_name
      -e PGUSER=postgres
      -e PGPASSWORD=password
      -e S3_ACCESS_KEY=ACCESS_KEY
      -e S3_SECRET_KEY=SECRET_KEY
      -e S3_BUCKET=BUCKET
      -e S3_HOST=https://storage.googleapis.com || s3.eu-west-1.amazonaws.com || nyc3.digitaloceanspaces.com
      -e CRON_SCHEDULE="@daily"
      docker.pkg.github.com/thakkaryash94/docker-postgres-backup/docker-postgres-backup:latest

That's it, now the container should be up and running.

Links:

Build NextJS Application Using GitHub Workflow and Docker

Yash Thakkar — Mon, 09 Nov 2020 07:09:24 +0000

NextJS is a JavaScript framework created by vercel. It lets you build serverless API, server-side rendering and static web applications using React. Vercel provides the out of box CI/CD integration with GitHub, GitLab, and BitHub. But sometimes, we want to host our NextJS application on other platforms than vercel, like AWS, GCP, DigitalOcean or Azure. In this blog, we will see how we can build our NextJS application using GitHub Workflow and Docker.

Setup NextJS Application

NextJS recommends using create-next-app, which sets up everything automatically for you. To create a project, run:

npx create-next-app
# or
yarn create next-app

After the installation is completed, follow the instructions to start the development server. Try editing pages/index.js and see the result on your browser.

For more information on how to use create-next-app, you can review the documentation

Setup Dockerfile

We will package our NextJS application in Docker image. The reason for using Docker is we won't need to install any additional packages like nodejs, pm2 etc when we want to run our NextJS server. Docker will bundle everything up and give us the image that we can run anywhere. Below is the sample Dockerfile for our NextJS application.

FROM node:lts-alpine

ENV NODE_ENV production
ENV NPM_CONFIG_LOGLEVEL warn

RUN mkdir /home/node/app/ && chown -R node:node /home/node/app

WORKDIR /home/node/app

COPY package.json package.json
COPY package-lock.json package-lock.json

USER node

RUN npm install --production

COPY --chown=node:node .next .next
COPY --chown=node:node public public

EXPOSE 3000

CMD npm start

Now, let's see what is happening in above Dockerfile step-by-step.

We are using node:lts-alpine as the base image.
Setting environment variable as production.
Setting up an app folder with node user as owner.
Copying package.json and package-lock.json into the image.
Running npm install production to install only production dependencies.
Copying .next and public folder into the container. This is a very interesting step. Why are we copying the folders and not building the the application using next build command? We will discuss this in detail below.
Exposing port 3000, so that our application can be accessible out of the container.
Finally, running npm start command to start our NextJS application server.

We can see, we are not making any changes in the Dockerfile. It's easy to understand and straightforward. The interesting part is we are copying .next and public folder into the container, instead of building inside the container.

Here is the detailed explanation:

In NextJS application, we may need to use NEXT_PUBLIC environment variables. NEXT_PUBLIC variables are required for the build time process. (eg. firebase web client)
If we use a firebase web client, then we need to provide a few required variables like firebase api_key, app_id, auth_domain.
We write these variables in .env or .env.local file when developing our application locally. But we DO NOT, SHOULD NOT and MUST NOT push this file on VCS systems like git.
So when we build our application locally, it will use these variables from the .env and process gets completed without any error. But when we build our application in Docker using RUN next build command, our build command will fail because we are not providing these variables in the docker image.
If we want to build our NextJS application inside docker build process, we need to use --build-args in docker build command to pass the build-time variables. There are 2 ways to do this.
1. We use ci secret variables and pass them into the docker build command
2. We create a .env file, encode tis using base64, pass it as ci secret variable, decode it using base64 inside docker file and then build the docker image.
This will become very difficult to pass and maintain if our public variables list grows in the future.
So to not complicate the build process, we will build our application outside the docker image using ci job and then copy the .next, public folders into a docker image.
To pass environment variables in ci, there are 2 ways.
1. Pass the environment variables as secrets
2. Pass the base64 encoded of .env file, decode it inside ci process, write the file at the root of our project folder, same as local development and build our application.

GtiHub Workflow

A workflow is a configurable automated process made up of one or more jobs. We will configure the workflow with YAML file. You can read more here.

As discussed above, we will use GitHub workflow jobs to build our NextJS application. Below is the workflow file, we will be using the same. Save this file at PROJECT_ROOT_FOLDER/.github/workflows/main.yml, so that GitHub can read the yaml file and setup actions accordingly.

Note: To be able to see the actions in the UI, you need to have the same file available in master or main branch.

Workflow file:

name: Build & Publish

on:
  push:
    branches:
      - "**"             # all branches
      - "!dependabot/**"      # exclude dependbot branches
  workflow_dispatch:      # Manually run the workflow

jobs:
  next-build:
    if: ${{ github.event_name == 'workflow_dispatch' }}       # Run only if triggered manually
    runs-on: ubuntu-latest
    container: node:lts          # Use node LTS container version, same as Dockerfile base image
    steps:
      - name: Checkout
        uses: actions/checkout@v2       # Checkout the code
      - run: npm ci            #install dependencies
      - run: npm run build
        env:
          NEXT_PUBLIC_FIREBASE_API_KEY: ${{secrets.NEXT_PUBLIC_FIREBASE_API_KEY}}
          NEXT_PUBLIC_FIREBASE_APP_ID: ${{secrets.NEXT_PUBLIC_FIREBASE_APP_ID}}
          NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN: ${{secrets.NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN}}
          NEXT_PUBLIC_FIREBASE_PROJECT_ID: ${{secrets.NEXT_PUBLIC_FIREBASE_PROJECT_ID}}
          NEXT_PUBLIC_SENTRY_DSN: ${{secrets.NEXT_PUBLIC_SENTRY_DSN}}
      - name: Upload Next build          # Upload the artifact
        uses: actions/upload-artifact@v2
        with:
          name: build
          path: |
            .next
            public
          retention-days: 7         # artifact retention duration, can be upto 30 days
  docker-push:
    needs: next-build        # Job depends on next-build(above) job
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Download next build       # Download the above uploaded artifact
        uses: actions/download-artifact@v2
        with:
          name: build
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.CR_PAT }}
      - name: Build and Push Docker Images
        run: |
          export CURRENT_BRANCH=${GITHUB_REF#refs/heads/}
          export TAG=$([[ $CURRENT_BRANCH == "main" ]] && echo "latest" || echo $CURRENT_BRANCH)
          export GITHUB_REF_IMAGE=ghcr.io/$GITHUB_REPOSITORY:$GITHUB_SHA
          export GITHUB_BRANCH_IMAGE=ghcr.io/$GITHUB_REPOSITORY:$TAG
          docker build -t $GCR_IMAGE -t $GITHUB_REF_IMAGE -t $GITHUB_BRANCH_IMAGE .
          echo "Pushing Image to GitHub Container Registry"
          docker push $GITHUB_REF_IMAGE
          docker push $GITHUB_BRANCH_IMAGE

Now, let's discuss what is happening in yaml file.

We need to pass the condition on which event we want to trigger our workflow. In our case, we want it on the push event. It can multiple as well like [push, pull_request]. You can read more here.
We can define the branches, we want this workflow to run to watch. ! means want to exclude these branches.
workflow_dispatch to manually run the build process. If we don't write this, our workflow will run every-time we push to any branch of our repository. You can read more here.
We have divided our build process into 2 jobs.
1. next-build:
  - In this job, we are using node:lts as the base image, this has to be the same as Dockerfile base image
  - We are keeping this job manual, as we don't want this job to run everytime we push the code. So we add if: ${{ github.event_name == 'workflow_dispatch' }} condition in step.
  - In env section, we are exporting environment variables from secrets. So we need to add these variables in GitHub project secrets. Read more here on how to do it.
  - In next step, action will checkout the code, run npm ci to install dependencies and npm run build to build the NextJS application using exported environment variables.
  - Finally, after a successful build, CI job will use actions/upload-artifact@v2 action to upload our build folder as an artifact on GitHub with 7 days of retention time, so that ci job can download the same folder in docker-build job and use it to build the image. In the build folder, we are including .next and public folder. .next folder is generated by the build process and we use public folder for assets like, svgs, images etc. So we want to keep that folder as well.
  - You can see the folder in action detail as below.
2. docker-push: To build our docker image
  - This job depends on needs:next-build, which means we will only see this job after a successful next-build job. If we don't write this, then our both the job will parallel and this job will fail because it won't be able to download build artifact. next build will upload the artifact then only, ci job and we will be able to access it. So we need to write this, it will create a sequential job, instead of parallel.
  - CI job will checkout the code, download the build artifact folder using actions/download-artifact@v2 and extract it as well.
  - We want to keep our docker image to be hosted on GitHub packages, for that, we will use docker/login-action@v1 action to login into the GitHub Container Registry server using username and password. We need to pass CR_PAT as well in repository secrets same as NEXT_PUBLIC vars. We can add here other registries as well like GCR, AWS ECR etc.
  - Next, ci job will get the CURRENT_BRANCH and tag our docker build accordingly. Here, we are creating 2 tags, one is with branch name like dev, qa, uat, main and another is with commit SHA.
  - after that, the job will start building our docker images and push it to GitHub packages after a successful build. Here, we can push it to other registries as well like GCR, AWS ECR etc.
  - finally, this job will exit and our workflow will be successfully passed.

To run the job, we have to navigate to repo actions and you will see the workflow with Build & Push on the left sidebar. Click on that link and you will be able to see the screen as below.

With this, We will be able to build and package our NextJS application. You will see the action screen below the screenshot.

Thank you for reading. Have a great day!

Help Links:

Host Static website using AWS CDK for Terraform and CloudFront: Part 2

Yash Thakkar — Fri, 07 Aug 2020 13:21:44 +0000

In part 1, we saw how we can host our website using S3. In this part, we will see how we can configure AWS CloudFront to serve our S3 bucket objects as website. In case, if you have not checked out the Part 1, please read this first.

Let's setup CloudFront distribution using AWS CDK of Terraform. We need to create CloudFront Origin Access Identity(OAI), which we will use for CloudFront and S3 bucket policy.

import { CloudfrontOriginAccessIdentity } from './.gen/providers/aws';

/*
 * Create am Origin Access Identity
 * Doc link: https://aws.amazon.com/premiumsupport/knowledge-center/cloudfront-access-to-amazon-s3/
 * Tutorial link: https://aws.amazon.com/premiumsupport/knowledge-center/cloudfront-access-to-amazon-s3/
 */
const cloudfrontOriginAccessIdentity = new CloudfrontOriginAccessIdentity(this, 'aws_cloudfront_origin_access_identity', {
  comment: 's3-cloudfront-cdk-example'
})

Now, we need to set few required parameters for the CloudFront configuration.

dependsOn: Bucket is required to setup CloudFront.
defaultRootObject: index.html is our default file that we need to serve.
customErrorResponse: We can setup custom rules/response for errors like 400, 404, 500, 501 etc.
origin:
- originId: unique id (should be same as targetOriginId)
- domainName: S3 bucket as domain (eg. thakkaryash94-cdk-dev.s3.amazonaws.com)
defaultCacheBehavior:
- targetOriginId: unique id (should be same as originId)
restrictions: We want our website be to accessible from everywhere, so set it to none.
viewerCertificate: We can use CloudFront default certificate and can also add custom ACM certificate, IAM certificate etc.

import { CloudfrontDistribution } from './.gen/providers/aws';

const originId = `S3-${BUCKET_NAME}`;

const cloudFrontDistribution = new CloudfrontDistribution(this, `aws_cloudfront_${BUCKET_NAME}`, {
  enabled: true,
  dependsOn: [bucket],
  defaultRootObject: 'index.html',
  customErrorResponse: [{
    errorCode: 404,
    responseCode: 200,
    responsePagePath: '/index.html'
  }],
  origin: [{
    originId: originId,
    domainName: bucket.bucketDomainName,
    s3OriginConfig: [{
      originAccessIdentity: cloudfrontOriginAccessIdentity.cloudfrontAccessIdentityPath
    }]
  }],
  defaultCacheBehavior: [{
    allowedMethods: ['GET', 'HEAD'],
    cachedMethods: ['GET', 'HEAD'],
    forwardedValues: [{
      cookies: [{ forward: 'none' }],
      queryString: false
    }],
    targetOriginId: originId,
    viewerProtocolPolicy: 'allow-all'
  }],
  restrictions: [{
    geoRestriction: [{
      restrictionType: 'none'
    }]
  }],
  viewerCertificate: [{
    cloudfrontDefaultCertificate: true
  }],
});

Previously, our bucket was public, it means anyone can access bucket objects using bucket website URL. Now, we have configured CloudFront to serve our website, so it's time to block that access. With this, our website will be only accessible by CloudFront URL only. No-one will be able to access bucket objects using S3 website URL.

bucket.acl = 'private'            // Set bucket ACL as private
bucket.website = []               // Disable website hosting feature
bucket.policy = `{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity ${cloudfrontOriginAccessIdentity.id}"
      },
      "Action": [
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::${BUCKET_NAME}/*"
      ]
    }
  ]
}`

Last, we will print the CloudFront url, which will serve our s3 objects as a website.

// Output the cloudfront url to access the website
new TerraformOutput(this, 'cloudfront_website_endpoint', {
  description: 'CloudFront URL',
  value: `https://${cloudFrontDistribution.domainName}`
});

Now, we follow the same process to deploy the changes as per the Part 1. After successfull deployment, CloudFront public URL will be printed on the console and we will be able to access our website with default https certificate.

So this is how, we can setup CloudFront with AWS S3 using AWS CDK for Terraform.

thakkaryash94 / terraform-cdk-react-example

Host react website using terraform CDK on AWS S3

Host Static website using AWS CDK for Terraform

This repo contains the code for DEV.to blog

View on GitHub

Host Static website using AWS CDK for Terraform: Part 1

Yash Thakkar — Sun, 02 Aug 2020 07:49:14 +0000

There are many ways we can write and host a website like writing plain HTML files, using frameworks like Angular, React, Vue, Gatsby, many more, and hosting it on various services like Netlify, S3, Firebase, Azure, Zeit for free. In this blog, we will see, how we can use AWS CDK and Terraform to host our website on S3 without leaving the terminal.

AWS Cloud Development Kit (AWS CDK)

The AWS-CDK was released and open sourced around May 2018. The AWS Cloud Development Kit (AWS CDK) is an open-source software development framework to define cloud infrastructure in code and provision it through AWS CloudFormation.

Terraform

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc.

CDK for Terraform

Hashicorp published CDK for Terraform with Python and TypeScript support. CDK for Terraform generates Terraform configuration to enable provisioning with Terraform. The adaptor works with any existing provider and modules hosted in the Terraform Registry. The core Terraform workflow remains the same, with the ability to plan changes before applying.

The CDK for Terraform project includes two packages
- “cdktf-cli” - A CLI that allows users to run commands to initialize, import, and synthesize CDK for Terraform applications.
- “cdktf” - A library for defining Terraform resources using programming constructs.

Host Static Website

Getting Started

We will be using Node.js with Typescript to setup the deployment.

install the CDK for Terraform globally.

$ npm i -g cdktf-cli

Initialize a New Project

Create a directory name deployment under project folder and initialize a set of TypeScript templates using cdktf init.

$ mkdir -p project/deployment
$ cd project/deployment
$ cdktf init --template=typescript

Enter details regarding the project including Terraform Cloud for storing the project state. You can use the --local option to continue without using Terraform Cloud for state management.

We will now setup the project. Please enter the details for your project.
If you want to exit, press ^C.

Project Name: (default: 'deployment')
Project Description: (default: 'A simple getting started project for cdktf.')

We will be using a ReactJS project to write our website code for the example. You can use any framework and any programming language you want. Now, let's run below command to create ReactJS project.

$ npm i -g create-react-app
$ create-react-app web

After setting up the React project, run below command to build the React website. Below command will create a build folder, which contains all the files like HTML, CSS, JS, images etc which are required for the website under web folder.

$ cd web
$ npm run build

Now, if we run tree command on terminal or open the project folder in code editor, we will see our folder structure as below.

$ tree
├── deployment
│   ├── cdktf.json
│   ├── cdktf.out
│   ├── help
│   ├── main.d.ts
│   ├── main.js
│   ├── main.ts
│   ├── node_modules
│   ├── package-lock.json
│   ├── package.json
│   ├── terraform.tfstate
│   └── tsconfig.json
└── web
    ├── README.md
    ├── build
    ├── node_modules
    ├── package.json
    ├── public
    ├── src
    └── yarn.lock

Open main.ts file from deployment folder. We will be writing in this file as per our requirements and cdktf will read this file and setup Terraform file based on this.

Let's import AWS Provider class from .gen folder. Next, we need to specify which region we will using for deployment process.

// import required classes from generated folder
import { AwsProvider, S3Bucket, S3BucketObject } from './.gen/providers/aws';

// assign AWS region
new AwsProvider(this, 'aws', {
  region: 'us-west-1'
});

After setting up the region, we will be creating a S3 bucket with policies.

Use public-read for Access Control.
Enable website static website hosting feature. This is same as Use this bucket to host a website from AWS Console.
Update policy to make the objects in your bucket publicly readable.

// Define AWS S3 bucket name
const BUCKET_NAME = '<YOUR-WEBSITE-BUCKET-NAME>';

// Create bucket with public access
const bucket = new S3Bucket(this, 'aws_s3_bucket', {
  acl: 'public-read',
  website: [{
    indexDocument: 'index.html',
    errorDocument: 'index.html',
  }],
  tags: {
    'Terraform': "true",
    "Environment": "dev"
  },
  bucket: BUCKET_NAME,
  policy: `{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "PublicReadGetObject",
        "Effect": "Allow",
        "Principal": "*",
        "Action": [
          "s3:GetObject"
        ],
        "Resource": [
          "arn:aws:s3:::${BUCKET_NAME}/*"
        ]
      }
    ]
  }`,
});

After setting up the Bucket configuration, it's time to write code, which will upload the files to S3.

Read all the files from the build folder
Create a S3 bucket object for each file
1. dependsOn: this step will wait till the bucket is not created. This is very import parameter.
2. key: defines our files and folder structure on S3.
3. source: is the actual file we want to upload to S3.
4. etag: This is very important and tricky, reason is Terraform only creates or deletes the files. If there is already a file uploaded on S3 with a name and we are changing the content of the file then it won't replace the file. With this parameter, it will replace existing files as well because their etags are changed.
5. contentType: This is also very important because this parameter will instruct the browser to open the file as a HTML, image, js, css file. If we don't define this, then when we open our S3 bucket public URL in the browser, the browser will download the index.html file instead of opening it because without content-type, the browser has no idea what to do with this file.

// import necessary packages
import * as path from 'path';
import * as glob from 'glob';
import * as mime from 'mime-types';

// Get all the files from build folder, skip directories
const files = glob.sync('../web/build/**/*', { absolute: false, nodir: true });

// Create bucket object for each file
for (const file of files) {
  new S3BucketObject(this, `aws_s3_bucket_object_${path.basename(file)}`, {
    dependsOn: [bucket],            // Wait untill the bucket is not created
    key: file.replace(`../web/build/`, ''),       // Using relative path for folder structure on S3
    bucket: BUCKET_NAME,
    source: path.resolve(file),          // Using absolute path to upload
    etag: `${Date.now()}`,
    contentType: mime.contentType(path.extname(file)) || undefined       // Set the content-type for each object
  });
}

Last step is to output the bucket public URL. So after successful execution, we will get the public URL on our terminal. We just need to copy and paste it on browser and we will be able to view our website.

// import required class to print the output
import { TerraformOutput } from 'cdktf';

// Output the bucket url to access the website
new TerraformOutput(this, 'website_endpoint', {
  value: `http://${bucket.websiteEndpoint}`
});

So now, out script is ready, it's time to Synthesize TypeScript to Terraform Configuration.

Synthesize TypeScript to Terraform Configuration

Let's synthesize TypeScript to Terraform configuration by running cdktf synth. The command generates Terraform JSON configuration files in the cdktf.out directory.

$ cd deployment
$ cdktf synth
Generated Terraform code in the output directory: cdktf.out

$ tree cdktf.out
cdktf.out
└── cdk.tf.json

Inspect the generated Terraform JSON file by examining cdktf.out/cdk.tf.json. It includes the Terraform configuration for the S3 and S3 bucket objects which looks like below.

"resource": {
    "aws_s3_bucket": {
      "typescriptaws_awss3bucket_D835B1D8": {
        "acl": "public-read",
        "bucket": "thakkaryash94-cdk-dev",
        "policy": "{\n        \"Version\": \"2012-10-17\",\n        \"Statement\": [\n          {\n            \"Sid\": \"PublicReadGetObject\",\n            \"Effect\": \"Allow\",\n            \"Principal\": \"*\",\n            \"Action\": [\n              \"s3:GetObject\"\n            ],\n            \"Resource\": [\n              \"arn:aws:s3:::thakkaryash94-cdk-dev/*\"\n            ]\n          }\n        ]\n      }",
        "tags": {
          "Terraform": "true",
          "Environment": "dev"
        },
        "website": [
          {
            "index_document": "index.html",
            "error_document": "index.html"
          }
        ],
        "//": {
          "metadata": {
            "path": "typescript-aws/aws_s3_bucket",
            "uniqueId": "typescriptaws_awss3bucket_D835B1D8",
            "stackTrace": [
              "new TerraformElement (/Users/yash/github_workspace/typescript-aws/deployment/node_modules/cdktf/lib/terraform-element.js:10:19)",
              "new TerraformResource (/Users/yash/github_workspace/typescript-aws/deployment/node_modules/cdktf/lib/terraform-resource.js:9:9)",
              "new S3Bucket (/Users/yash/github_workspace/typescript-aws/deployment/.gen/providers/aws/s3-bucket.js:13:9)",
              "new MyStack (/Users/yash/github_workspace/typescript-aws/deployment/main.js:18:24)",
              "Object.<anonymous> (/Users/yash/github_workspace/typescript-aws/deployment/main.js:66:1)",
              "Module._compile (internal/modules/cjs/loader.js:1185:30)",
              "Object.Module._extensions..js (internal/modules/cjs/loader.js:1205:10)",
              "Module.load (internal/modules/cjs/loader.js:1034:32)",
              "Function.Module._load (internal/modules/cjs/loader.js:923:14)",
              "Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)",
              "internal/main/run_main_module.js:17:47"
            ]
          }
        }
      }
    },
    "aws_s3_bucket_object": {
      "typescriptaws_awss3bucketobjectindexhtml_E7345193": {
        "bucket": "thakkaryash94-cdk-dev",
        "content_type": "text/html; charset=utf-8",
        "etag": "1596349930423",
        "key": "index.html",
        "source": "/Users/yash/github_workspace/typescript-aws/web/build/index.html",
        "depends_on": [
          "aws_s3_bucket.typescriptaws_awss3bucket_D835B1D8"
        ],
        "//": {
          "metadata": {
            "path": "typescript-aws/aws_s3_bucket_object_index.html",
            "uniqueId": "typescriptaws_awss3bucketobjectindexhtml_E7345193",
            "stackTrace": [
              "new TerraformElement (/Users/yash/github_workspace/typescript-aws/deployment/node_modules/cdktf/lib/terraform-element.js:10:19)",
              "new TerraformResource (/Users/yash/github_workspace/typescript-aws/deployment/node_modules/cdktf/lib/terraform-resource.js:9:9)",
              "new S3BucketObject (/Users/yash/github_workspace/typescript-aws/deployment/.gen/providers/aws/s3-bucket-object.js:13:9)",
              "new MyStack (/Users/yash/github_workspace/typescript-aws/deployment/main.js:50:13)",
              "Object.<anonymous> (/Users/yash/github_workspace/typescript-aws/deployment/main.js:66:1)",
              "Module._compile (internal/modules/cjs/loader.js:1185:30)",
              "Object.Module._extensions..js (internal/modules/cjs/loader.js:1205:10)",
              "Module.load (internal/modules/cjs/loader.js:1034:32)",
              "Function.Module._load (internal/modules/cjs/loader.js:923:14)",
              "Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)",
              "internal/main/run_main_module.js:17:47"
            ]
          }
        }
      }
    }
  }

We can also print Terraform JSON configuration in their terminal using cdktf synth --json command.

After synthesis, we can use the Terraform workflow of initializing, planning, and applying changes within the cdktf.out working directory or use the CDK for Terraform CLI to run cdktf deploy.

Terraform workflow is as follows:

$ cd cdktf.out
$ terraform init
Terraform has been successfully initialized!
$ terraform plan
# omitted for clarity
$ terraform apply
aws_s3_bucket.typescriptaws_awss3bucket_D835B1D8: Creating...
aws_s3_bucket.typescriptaws_awss3bucket_D835B1D8: Creation complete after 25s [id=thakkaryash94-cdk-dev]
aws_s3_bucket_object.typescriptaws_awss3bucketobjectindexhtml_E7345193: Creating...
# omitted for clarity
Apply complete! Resources: 20 added, 0 changed, 0 destroyed.

Outputs:

typescriptaws_websiteendpoint_D74DC454 = http://thakkaryash94-cdk-dev.s3-website-us-west-1.amazonaws.com
# destroy resources
$ terraform destroy
Plan: 0 to add, 0 to change, 20 to destroy.
Destroy complete! Resources: 20 destroyed.

This is how we can deploy our ReactJS website to AWS S3 using AWS CDK for Terraform.Here is the sample repo for the reference.

thakkaryash94 / terraform-cdk-react-example

Host react website using terraform CDK on AWS S3

Host Static website using AWS CDK for Terraform

This repo contains the code for DEV.to blog

View on GitHub

Links:

Many ways to build a container image

Yash Thakkar — Mon, 18 May 2020 17:59:46 +0000

Whenever we think about building or running a container, the first tool that comes to our mind is Docker and it's perfectly fine. Docker has made our life way easier than we can think of. But there are other ways as well, we can build and run the image as well. So now the question comes to the mind is, how is it possible for Kubernetes, OpenStack, ECS, DC/OS to understand the build system of every different way we can build the image. Turns out, they don't need to. Because there is a specification on what should be the Image format of the container. That's why OCI(Open Container Initiative) is formed.

Open Container Initiative (OCI)

The Open Container Initiative (OCI) is a lightweight, open governance structure (project), formed under the auspices of the Linux Foundation, for the express purpose of creating open industry standards around container formats and runtime. The OCI was launched on June 22nd, 2015 by Docker, CoreOS, and other leaders in the container industry. Read more here.
We will be building nodejs-express server using the below tools. Now, Let's get started with the list.

Buildah

Buildah is a command-line tool for building Open Container Initiative-compatible (that means Docker and Kubernetes-compatible, too) images quickly and easily. Buildah is easy to incorporate into scripts and build pipelines, and best of all, it doesn't require a running container daemon to build its image.

buildah build-using-dockerfile -f Dockerfile .

Doc link: https://github.com/containers/buildah/blob/master/docs/buildah-bud.md

BuildKit

BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive, and repeatable manner. BuildKit is composed of the buildkitd daemon and the buildctl client. While the buildctl client is available for Linux, macOS, and Windows, the buildkitd daemon is only available for Linux currently.

buildctl build \
    --frontend=dockerfile.v0 \
    --local context=. \
    --local dockerfile=.

Doc Link: https://github.com/moby/buildkit#building-a-dockerfile-with-buildctl

Buildpacks

Buildpacks were first conceived by Heroku in 2011. Since then, they have been adopted by Cloud Foundry and other PaaS such as GitLab, Knative, Deis, Dokku, and Drie. Buildpacks embrace modern container standards, such as the OCI image format.

express-sample> pack suggest-builders
Suggested builders:
  Cloud Foundry:     cloudfoundry/cnb:bionic         Ubuntu bionic base image with buildpacks for Java, NodeJS and Golang
  Cloud Foundry:     cloudfoundry/cnb:cflinuxfs3     cflinuxfs3 base image with buildpacks for Java, .NET, NodeJS, Golang, PHP, HTTPD and NGINX
  Cloud Foundry:     cloudfoundry/cnb:tiny           Tiny base image (bionic build image, distroless run image) with buildpacks for Golang
  Heroku:            heroku/buildpacks:18            heroku-18 base image with buildpacks for Ruby, Java, Node.js, Python, Golang, & PHP

Tip: Learn more about a specific builder with:
  pack inspect-builder [builder image]

pack build express-pack-app --path . --builder heroku/buildpacks:18

Doc Link: https://buildpacks.io/docs/app-developer-guide/build-an-app/

img

Standalone, daemon-less, unprivileged Dockerfile and OCI compatible container image builder.

img is more cache-efficient than Docker and can also execute multiple build stages concurrently, as it internally uses BuildKit's DAG solver.

The commands/UX are the same as docker {build,tag,push,pull,login,logout,save} so all you have to do is replace docker with img in your scripts, command line, and/or life.

img build -t r.j3ss.co/img .

Doc link: https://github.com/genuinetools/img#build-an-image

Kaniko

Kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster. Kaniko doesn't depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. This enables building container images in environments that can't easily or securely run a Docker daemon, such as a standard Kubernetes cluster.

docker run \
    -v "$HOME"/.config/gcloud:/root/.config/gcloud \
    -v /path/to/context:/workspace \
    gcr.io/kaniko-project/executor:latest \
    --dockerfile /workspace/Dockerfile \
    --destination "gcr.io/$PROJECT_ID/$IMAGE_NAME:$TAG" \
    --context dir:///workspace/

Doc link: https://github.com/GoogleContainerTools/kaniko

Podman

Podman specializes in all of the commands and functions that help you to maintain and modify OCI images, such as pulling and tagging. It also allows you to create, run, and maintain those containers created from those images. For building container images via Dockerfiles, Podman uses Buildah's golang API and can be installed independently from Buildah.

podman build -f Dockerfile .

Doc link: https://docs.podman.io/en/latest/markdown/podman-build.1.html#examples

PouchContainer

PouchContainer is a highly reliable container engine open sourced by Alibaba. It is an excellent software layer to fill up the gap between business applications and underlying infrastructure. The strong-isolation ability and rich container are its representative features.

pouch build [OPTION] PATH
Option:
      --addr string             buildkitd address (default "unix:///run/buildkit/buildkitd.sock")
      --build-arg stringArray   Set build-time variables
  -h, --help                    help for build
  -t, --tag stringArray         Name and optionally a tag in the 'name:tag' format
      --target string           Set the target build stage to build

Doc link: https://pouchcontainer.io/#/pouch/docs/commandline/pouch_build.md

OCIBuilder

The ocibuilder offers a command line tool called the ocictl to build, push and pull OCI compliant images through declarative specifications, allowing you to pick between Buildah or Docker as the container build tool.

ocictl init
ocictl build        # using docker
ocictl build --builder buildah      # using buildah

Doc link: https://ocibuilder.github.io/docs/quickstart/

As we can see here, there are many ways we can build an image and run it practically anywhere we want. Comment down below if you any other ways we can build it.

Help Links:

Docker centralized logging using Fluent Bit, Grafana and Loki

Yash Thakkar — Wed, 13 May 2020 14:26:45 +0000

When running microservices as containers, monitoring becomes very complex and difficult. That's where Prometheus, Grafana come to the rescue. Prometheus collects the metrics data and Grafana helps us to convert those metrics into beautiful visuals. Grafana allows you to query, visualize, and create an alert on metrics, no matter where they are stored. We can visualize metrics like CPU usage, memory usage, containers count, and much more. But there are few things that we can't visualize like container logs, it needs to be in tabular format with text data. For that, we can setup EFK (Elasticsearch + Fluentd + Kibana) stack, so Fluentd will collect logs from a docker container and forward it to Elasticsearch and then we can search logs using Kibana.

Grafana team has released Loki, which is inspired by Prometheus to solve this issue. So now, we don't need to manage multiple stacks to monitor the running systems like Grafana and Prometheus to monitor and EFK to check the logs.

Grafana Loki

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. It uses labels from the log data to query.

Fluent Bit

Fluent Bit is an open-source and multi-platform Log Processor and Forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations. It's fully compatible with Docker and Kubernetes environments.

Docker Fluentd logging driver

The Fluentd logging driver sends container logs to the Fluentd collector as structured log data. Then, users can use any of the various output plugins of Fluentd to write these logs to various destinations.

We are going to use Fluent Bit to collect the Docker container logs and forward it to Loki and then visualize the logs on Grafana in tabular View.

Setup

We need to setup grafana, loki and fluent/fluent-bit to collect the Docker container logs using fluentd logging driver. Clone the sample project from here. It contains the below files.

docker-compose-grafana.yml
docker-compose-fluent-bit.yml
fluent-bit.conf
docker-compose-app.yml

We can combine all the yml files into one but I like it separated by the service group, more like kubernetes yml files. Let's see, what we have in those files.

Before running docker services, we need to create an external network loki because our services are in different files, so they will communicate in this network.

$ docker network create loki

docker-compose-grafana.yml
This file contains Grafana, Loki, and renderer services. run docker-compose -f docker-compose-grafana.yml up -d. This will start 3 containers, grafana, renderer, and Loki, we will use grafana dashboard for the visualization and loki to collect data from fluent-bit service. Now, go to http://localhost:3000/ and you will be able to access the Grafana Dashboard.

docker-compose-fluent-bit.yml
We will be using grafana/fluent-bit-plugin-loki:latest image instead of a fluent-bit image to collect Docker container logs because it contains Loki plugin which will send container logs to Loki service. For that, we need to pass LOKI_URL environment variable to the container and also mounting fluent-bit.conf as well for custom configuration.

  fluent-bit:
    image: grafana/fluent-bit-plugin-loki:latest
    container_name: fluent-bit
    environment:
      - LOKI_URL=http://loki:3100/loki/api/v1/push
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf

fluent-bit.conf

This file contains fluent-bit configuration. Here, for input, we are listening on 0.0.0.0:24224 port and forwarding whatever we are getting to output plugins. We are setting a few Loki configs like LabelKeys, LineFormat, LogLevel, Url. The main key is LabelKeys, using this, we will be able to see the container logs, to make it dynamic, we are setting it so container_name, which means when we will be running our services, we need to pass container_name in docker-compose file, using that name, we will be able to search and differentiate container logs. We can add as many LabelKeys as we want with a comma(',').

[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224
[Output]
    Name loki
    Match *
    Url ${LOKI_URL}
    RemoveKeys source
    Labels {job="fluent-bit"}
    LabelKeys container_name
    BatchWait 1
    BatchSize 1001024
    LineFormat json
    LogLevel info

Now, let's run it with docker-compose -f docker-compose-fluent-bit.yml up -d. This will start fluent-bit container, which will collect the docker container logs and everything that is printed using stdout and forward it to loki service using loki plugin.

docker-compose-app.yml
It contains the actual application/server image service. You can ignore this file but we have to add below config in our server to forward container logs to the fluent-bit container. Important parts of the configuration are container_name, logging. container_name is the one we will use to filter the container logs from the Grafana Dashboard. In fluent-address, set your fluent-bit host IP address, if you are running locally, it will be your PC ip address.

    container_name: express-app
    logging:
      driver: fluentd
      options:
        fluentd-address: FLUENT_BIT_ADDRESS:24224

Now, everything is up and running. Let's generate some logs, if you are running docker-compose-app.yml file, then go to http://localhost:4000 and refresh few times, go to http://localhost:4000/test, this will generate some logs.

So from docker container, logs will be sent to fluent-bit container, which will forward them to the Loki container using the Loki plugin. Now, we need to add Loki in Grafana data source, so that Grafana will be able to fetch the logs from Loki and we will be able to see it on the dashboard.

Setup Grafana Dashboard

To see the logs on Grafana dashboard, you can follow YouTube video or below steps.

Open the browser and go to http://localhost:3000, use default values admin and admin for username and password.
Now, go to http://localhost:3000/datasources and select Loki from Logging and document databases section.
Enter http://loki:3100 in URL under HTTP section. We can do this because we are running Loki and Grafana in the same network loki else you have to enter host IP address and port here, click on Save and Test button from the bottom of the page.
Now, go to 3rd tab Explore from the left sidebar or http://localhost:3000/explore, click on Log Labels dropdown, here you will see container_name and job labels, these are same labels that we have mentioned in the fluent-bit.conf file with LabelKeys key.
Click on container_name, now, you should see our app service container name in the next step else type {container_name="express-app"} in the Loki query search. Click on that and that's it, now you should be able to see the container logs, these are the logs that we generated after starting up our app service.

Now, we can tweak the view add this to our Grafana Dashboard and that's it.

So, like this, we have setup fluentd-grafana-loki stack to collect and view the container logs on Grafana Dashboard.

Below are the links that I have mentioned in the blog, that will help you to setup the stack.

Links:

CockroachDB auto-backup with Docker

Yash Thakkar — Mon, 04 May 2020 22:09:24 +0000

CockroachDB is one of the most popular cloud-native databases. CockroachDB is an ACID compliant, relational database that’s wire compatible with PostgreSQL. CockroachDB delivers full ACID transactions at scale even in a distributed environment and guarantees serializable isolation in a cloud-neutral distributed database. We can deploy it using Docker and Kubernetes without any issue. CockroachDB delivers on the key cloud-native primitives of horizontal scale, no single points of failure, survivability, automatable operations, and no platform-specific encumbrances.

How to run CockroachDB?

We can run CockroachDB using a single executable binary or using Docker and Kubernetes. We will see how to run CockroachDB in Docker and how to take backup as well. You can follow the blog or Docs and follow the backup instructions.

Start CockroachDB using Docker

We can run a single or multi node cluster with Docker but for development, we will be using single node cluster only.

Below command will pull the latest CockroachDB image from Docker hub, bind 26257, 8080 ports, bind the volume with cockroach_data/roach1. Run the command and go to http://localhost:8080, you will be able to see the CockroachDB Dashboard.

docker run -d \
      --name=roach1 \
      --hostname=roach1 \
      -p 26257:26257 -p 8080:8080  \
      -v "${PWD}/cockroach_data/roach1:/cockroach/cockroach-data"  \
      cockroachdb/cockroach:latest start \
      --insecure

Now, our CockroachDB database server is up and running. Now, Let's create some data using workload init command or you can dump existing database as well. Run below commands to do that. Here is the docs explaining in details what we are doing in below command.

docker exec -it roach1 bash

./cockroach workload init movr 'postgresql://root@localhost:26257?sslmode=disable'
./cockroach sql --insecure --host=localhost:26257
USE movr;
SHOW tables;

As you can understand, it will create a database movr with some tables and records. You should be able to see below output on your terminal.

          table_name
+----------------------------+
  promo_codes
  rides
  user_promo_codes
  users
  vehicle_location_histories
  vehicles
(6 rows)

Time: 9.579561ms

This means, we have successfully generated demo data and it's time to take a backup of it. Run exit 2 times to get out of the container. 1st exit command to get out of cockroach sql command line and 2nd is to exit from the container.

CockroachDB Database backup

Our database is ready, now it's time to take a backup of it. To do that, there are 2 ways, we can take a backup of it.

1. BACKUP

CockroachDB's BACKUP statement allows you to create full or incremental backups of your cluster's schema and data that are consistent as of a given timestamp. Backups can be with or without revision history.

There are many advantages of this process. We can setup whether we want to take a full backup or Incremental backup, automate backup with JOBS, upload it to Amazon, Azure, Google Cloud, NFS or any S3-compatible services, backup a single table or view and many more. You can read more on it here.

This happens inside cockroachDB container environment, so cockroachDB has full control over it.

The only disadvantage is this is available only for enterprise users. This means that, if we are running cockroachDB locally or on a small server, where we may not want enterprise support, we can't use this feature. We can CockroachCloud, if we are running it on a small scale and planning to scale it in the future. CockroachCloud provides a Fully hosted and managed, Self-service platform with Enterprise features and basic support. You can read more here.

So how can we do it without BACKUP feature, which is available only for CockroachCloud and CockroachDB Enterprise users.

2. cockroach dump

CockroachDB provides dump command, which is similar to pg_dump. The cockroach dump command outputs the SQL statements required to recreate tables, views, and sequences. This command can be used to back up or export each database in a cluster. The output should also be suitable for importing into other relational databases, with minimal adjustments. You can read more here.

That's great, so now, all we need to do it run a cron that executes cockroach dump whenever we want and that's it.

But, there are many things we have to think about. Like, how and where we are going to store our backup, how to take a backup on-demand etc etc.

Exactly, for that, I have created an open source docker image with the above features. So let's go through it one by one.

Features

Customize cron with CRON_SCHEDULE env. https://godoc.org/github.com/robfig/cron
Manual backup at any time
Optional backup AWS S3/Spaces upload, if you provide ACCESS_KEY_ID, then it will take it as you want backup to uploaded on S3 or Spaces or anywhere compatible with s3 API.
All cockroach image env variable support, you can override COCKROACH_USER, COCKROACH_INSECURE etc. docs
It exposes /data as volume, which contains backup zip file, so we can use backup from here if we don't want to upload it to any S3 services.

Environment Variables

Required:

COCKROACH_DATABASE: Database name
CRON_SCHEDULE: Cron value in double quotes. https://godoc.org/github.com/robfig/cron

Optional:

ACCESS_KEY_ID: Spaces access key id
BUCKET_NAME: Spaces bucket name
S3_URL: AWS S3(s3.ap-south-1.amazonaws.com) or DO Spaces(nyc3.digitaloceanspaces.com)
SECRET_ACCESS_KEY: Spaces secret access key

Now, let's look at how we can run our docker image to take a backup. Below, is the sample, how we can run the Docker container, which will take a backup of our movr database everyday and upload it to S3 service like AWS S3, Digital Ocean Spaces etc.

docker run -d \
      --name cockroach-backup \
      -v $(pwd)/data:/data \
      -v $(pwd)/cockroach-certs:/cockroach-certs \
      -e ACCESS_KEY_ID=ACCESS_KEY_ID \
      -e BUCKET_NAME=BUCKET_NAME \
      -e S3_URL=S3_URL \
      -e SECRET_ACCESS_KEY=SECRET_ACCESS_KEY \
      -e COCKROACH_DATABASE=movr \
      -e COCKROACH_HOST=localhost \
      -e CRON_SCHEDULE="0 0 * * *" \
      -e COCKROACH_INSECURE=true
      -e COCKROACH_USER=root
      docker.pkg.github.com/thakkaryash94/docker-cockroachdb-backup/docker-cockroachdb-backup:latest

With the above command, we will run our backup container name cockroach-backup with /data volume, which will contain all the backup files and with ACCESS_KEY_ID, we can upload it to wherever we want. It supports every client connection params.

Now, let's say, you want to take a database backup of the current moment, you can run a below command to trigger manual backup as well.

curl -X POST http://localhost:9000/backup

These features are already available and few more already on the list. Feel free to open an issue to add more features.

This is just a docker image so it works with Kubernetes as well.

Upcoming features:

Flags support
Multiple database backup support

Note: This is an open source project with MIT. This is my first golang project, I am a newbie, so I may have made mistakes. Issues and pull requests are most welcome.

Links:

Kubernetes auto-deployment using Okteto, Skaffold & GitLab CI/CD

Yash Thakkar — Mon, 03 Feb 2020 19:08:20 +0000

Overview

CI/CD is the process that never ends. Previously, we used to auto-deploy our applications in VMs by writing scripts that ssh into the remote server and deploy it. Then containers arrived, we wrapped our code in containers and writing scripts that build docker container and deploy on servers by stopping existing services and starting up new images for the services. For K8s, things are a little different.

kubectl is the new ssh.

https://twitter.com/kelseyhightower/status/1070413458045202433

@kelseyhightower said this. In k8s, we don't care about our instances or infrastructure. We perform all the operations using kubectl, like deploying, upgrading, accessing the application. So to auto-deploy our application from CI/CD, we need K8s cluster, so let's set it up first. For the demo, we will be using Okteto Cloud.

Okteto

Okteto provides a free K8s cluster. There are two ways to set up our k8s cluster. CLI and GUI.

CLI

Go to Okteto CLI page and follow the instructions to setup k8s cluster using CLI.

GUI

Go to Okteto Cloud, log in using your GitHub account and create namespace accordingly. In my case, my Okteto namespace will be thakkaryash94. That's it. Now, we have a K8s cluster with our namespace. Now click on Credentials from the left sidebar. This will download okteto-kube.config file. This is actually our kube-config file, we can access our namespace by using this config file. Now, we need to set up our application, so that we can deploy it on Okteto cloud.

Skaffold

Skaffold handles the workflow for building, pushing and deploying your application, allowing you to focus on what matters most: writing code. - This statement is directly from the website.

Lightweight: client-side only, no on-cluster component
Works Everywhere: you can use profiles, local user config, environment variables
Feature Rich: Kubernetes-native development, including policy-based image tagging, resource port-forwarding and logging, file syncing, and much more
Optimized Development: instant feedback while developing

We have finalized the tools that we need in our CI/CD pipeline. We need kubectl and Skaffold. Our k8s cluster is also ready to access the deployments. Now, we need to set up our CI/CD pipeline. We will be using GitLab CI/CD pipeline because I find it very easy and convenient to setup.

Setup:

We need to download skaffold on our local machine. Follow this link and setup as per your OS. Skaffold provides 5 Pipeline Stages.

Build, Test, Tag, Render, Deploy. We can use Skaffold for local development as well with minikube.

Project folder structure:

skaffold-example
- backend (actual application)
- k8s
  - deployment.yaml
- .gitlab-ci.yml
- skaffold.yaml

You can use any existing docker project for this. Yes, it definetly requires Dockerfile or you can use any of the example projects. We will be deploying nodejs example for the demo.

Note: Keep your application folder under backend folder.

skaffold.yaml

apiVersion: skaffold/v2alpha2
kind: Config
build:
  tagPolicy:
    gitCommit: {}       # use git commit policy
  artifacts:
  - image: registry.gitlab.com/thakkaryash94/skaffold-example
    context: backend
    sync:
      manual:
      # Sync all the javascript files that are in the src folder
      # with the container src folder
      - src: 'src/**/*.js'
        dest: .

Now run below command:

$ skaffold dev

Now, skaffold will use the skaffold.yaml file and start local dev environment with nodejs docker container. Now, let's break down our yaml config file.

tagPolicy: There are various tag policies skaffold provies. gitCommit, sha256, ,envTemplate dateTime
context: Actual folder path. In our case, it is the backend folder.
sync: There are two modes.
- Inferred sync mode: only need to specify which files are eligible for syncing in the sync rules.
- Manual sync mode: A manual sync rule must specify the src and dest field. The src field is a glob pattern to match files relative to the artifact context directory, which may contain ** to match nested files.

We will be using manual mode for better file controlling.

When we execute skaffold deploy or skaffold run, by default it will look for k8s/*.yaml files and apply all the configs. we can change this by adding manifests in skaffold.yaml file.

k8s/deployment.yaml

apiVersion: v1
kind: Service
metadata:
  name: node
  annotations:
    dev.okteto.com/auto-ingress: "true"
spec:
  type: ClusterIP
  ports:
  - name: "node"
    port: 3000
  selector:
    app: node
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node
spec:
  selector:
    matchLabels:
      app: node
  template:
    metadata:
      labels:
        app: node
    spec:
      containers:
      - name: skaffold-example
        image: registry.gitlab.com/thakkaryash94/skaffold-example
        ports:
        - containerPort: 3000
      imagePullSecrets:
        - name: gitlab-secret        # private registry secret

Our development setup is ready. it's time to make it live on our Okteto k8s cluster. To do that, we will be using GitLab CI/CD pipeline to build and run the application container.

GitLab CI

GitLab CI/CD is a tool built into GitLab for software development through the continuous methodologies:

Continuous Integration (CI)
Continuous Delivery (CD)
Continuous Deployment (CD)

Our folder structure will be like below:

Now, we will encode our kube-config file into base64 and store it on GitLab Project> Settings > CI/CD > Variables. We store our kube-config encoded data into a variable name KUBE_CONFIG. So when our piepline runs, it will pickup the variable data and store it in a config file. Run below command to get the base64 value for our Okteto kubernetes namespace.

$ cat ~/Downloads/okteto-kube.config | base64

Now, the most important .gitlab-ci.yml file

image: docker

services:
 - docker:dind

stages:
  - deploy

variables:
  DOCKER_DRIVER: overlay2
  KUBE_CONFIG_FILE: /etc/deploy/config

deploy:
  stage: deploy
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - mkdir -p /etc/deploy                                  # Create a folder for config file
    - echo ${KUBE_CONFIG} | base64 -d > ${KUBE_CONFIG_FILE}      # Write kubernetes config in config file
    - apk add --update --no-cache curl git     # Install dependencies
    - curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl    # Download kubectl binary
    - chmod +x ./kubectl
    - mv ./kubectl /usr/local/bin/kubectl
    - curl -Lo skaffold https://storage.googleapis.com/skaffold/builds/latest/skaffold-linux-amd64              # Download skaffold binary
    - chmod +x skaffold
    - ./skaffold run --kubeconfig /etc/deploy/config       # actual build, tag, push and deploy command

Now, we push our code to GitLab. It will automatically start reading .gitlab-ci.yml file and start running pipeline for us. Depends upon the Dockerfile steps, it may take few minutes.

After Job successfully finished, go to Okteto cloud dashboard, you should be able to see our application deployment with running status. Click on the link and you should be able to acces the application.

Note:: Okteto will automatically creates and add the ingress for us based on our deployment.yaml file service config.

DEV Community: Yash Thakkar

Setup CI/CD for Serverless monorepos application on AWS using GitHub Actions

AWS Lambda

Serverless Framework

Setup

Setup Monorepo

AWS S3

AWS API Gateway

CI/CD Deployment

GitHub Actions

auto.yml

manual.yml

Links:

Postgres backup using Docker

Problem

Solution

Build NextJS Application Using GitHub Workflow and Docker

Setup NextJS Application

Setup Dockerfile

GtiHub Workflow

Workflow file:

Host Static website using AWS CDK for Terraform and CloudFront: Part 2

thakkaryash94 / terraform-cdk-react-example

Host react website using terraform CDK on AWS S3

Host Static website using AWS CDK for Terraform

Links:

Host Static website using AWS CDK for Terraform: Part 1

AWS Cloud Development Kit (AWS CDK)

Terraform

CDK for Terraform

Host Static Website

Getting Started

Initialize a New Project

Synthesize TypeScript to Terraform Configuration

thakkaryash94 / terraform-cdk-react-example

Host react website using terraform CDK on AWS S3

Host Static website using AWS CDK for Terraform

Many ways to build a container image

Help Links:

Docker centralized logging using Fluent Bit, Grafana and Loki

Grafana Loki

Fluent Bit

Docker Fluentd logging driver

Setup

Setup Grafana Dashboard

CockroachDB auto-backup with Docker

How to run CockroachDB?

Start CockroachDB using Docker

CockroachDB Database backup

1. BACKUP

2. cockroach dump

Features

Environment Variables

Required:

Optional:

Kubernetes auto-deployment using Okteto, Skaffold & GitLab CI/CD

Overview

Okteto

CLI

GUI

Skaffold

Setup:

GitLab CI

Help links: