DEV Community: Eric Yi

Building the Docker Image: Expert Tips and Tricks

Eric Yi — Sun, 25 Dec 2022 23:35:48 +0000

Background

Docker has revolutionised the way we deploy and run applications, by providing a lightweight and portable way to package and distribute software. Building a high-quality Docker image is essential for ensuring that your application runs smoothly and efficiently in production. In this blog post, we will discuss some key considerations and best practices for building a high-quality Docker image from my own experience.

How Container Works

To understand how container works, we need to understand how Linux overlay work.

Linux overlay is a filesystem that allows the user to stack multiple filesystems on top of one another. Each filesystem has its own set of files and directories that are independent from the underlying filesystem. This allows to merge two directories together and present them as a single unified filesystem. One of these directories, known as the "lower" directory, is read-only, while the other directory, known as the "upper" directory, can be both read from and written to. When a process reads a file from the overlay filesystem, the overlays driver first checks for the file in the upper directory. If it is not found there, it falls back to the lower directory. When a process writes a file, it is always written to the upper directory. This allows you to modify or add new files in the upper directory without changing the original files in the lower directory.

Let us apply this knowledge to Docker images. When pulling a new image, if layer 2 already exists, it will not pull any layers below layer 2.

1 - Multi-Stage Builds Orders

Using what we have learned earlier, to optimize the efficiency of your Docker builds, you should order the layers in your build from the least frequently changed to the most frequently changed. This will allow the build cache to be reusable and improve build times.

Here is an example:

FROM python:3.7-buster

# copy the requirements file 
COPY requirements.txt /app

# create and set the working directory
RUN mkdir -p /app
WORKDIR /app

# copy the source code
COPY . /app

# install system dependencies
RUN apt-get update && apt-get install -y build-essential libpq-dev

# install Python dependencies
RUN pip install -r requirements.txt

# expose the port that the app will run on
EXPOSE 5000

# run the app
CMD ["python", "app.py"]

Is this a good example? The answer is no. First, we should install system dependencies, then copy the requirements file only, and install Python dependencies. Always remember to order them from the less frequently changed to the more frequently changed.

Here's how we can fix it:

FROM python:3.7-buster

# install system dependencies
RUN apt-get update && apt-get install -y build-essential libpq-dev

# copy the requirements file and install Python dependencies
COPY requirements.txt /app
RUN pip install -r requirements.txt

# create and set the working directory
RUN mkdir -p /app
WORKDIR /app

# copy the source code
COPY . /app

# expose the port that the app will run on
EXPOSE 5000

# run the app
CMD ["python", "app.py"]

2 - Build Environment VS Runtime Environment

Use a multi-stage build to separate the build environment from the runtime environment. This will enable you to use a larger base image for the build stage, and then only copy the necessary files to the smaller runtime image.

Here is an example:

FROM maven:3.6.3-jdk-11 as build
COPY src /app/src
COPY pom.xml /app
RUN mvn -f /app/pom.xml clean package

FROM openjdk:11-jre-slim as run
COPY --from=build /app/target/*.jar /app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app.jar"]

This Dockerfile uses the maven:3.6.3-jdk-11 base image for the build stage and the openjdk:11-jre-slim base image for the runtime stage. The build stage compiles the source code and creates a JAR file, which is then copied to the runtime image.

This can be especially useful if the build stage requires a lot of dependencies or tools that are not needed at runtime. For example, if the build stage uses a base image that is 1 GB in size and the runtime stage uses a base image that is 100 MB in size, using this method can potentially save 900 MB in image size.

3 - The Package Cache

The package cache is a local repository of package files that are downloaded from the Internet or other sources. When you use apt-get to install a package, the package files are downloaded to the package cache so that they can be easily accessed and used to install the package. However, as you install more and more packages, the package cache can grow quite large.

Here is an example:

FROM ubuntu:18.04-slim
RUN apt-get update 
RUN apt-get install -y curl 
RUN apt-get install -y dpkg-sig 
RUN apt-get clean

Is the package cache cleaned? The answer is no. As previously discussed, each Docker layer has its own set of files and directories that are separate from the underlying filesystem.

Running apt-get clean only clears the package cache on layer 4.

Here's how we can fix it:

RUN apt-get update && apt-get install -y \
    curl \
    dpkg-sig \
 && apt-get clean

Reference

Evans, Julia. (2019). "How containers work: overlayfs." Retrieved from https://jvns.ca/blog/2019/11/18/how-containers-work--overlayfs/

Azure Runbook Asyncio Error

Eric Yi — Thu, 31 Mar 2022 12:13:10 +0000

We recently had a project that required the use of Azure runbook. I found my Python code was not working as expected. It works locally but not in Azure runbook.

After debugging, I found the issue is related to Asyncio. Even if I ran the following sample code, I still got an error.

import asyncio
loop = asyncio.get_event_loop()

The error message:

**Traceback (most recent call last):  File "C:\Temp\vls2h0kl.4gk\04a596c9-5b8c-4a2d-828b-2bb185668277", line 53, in <module>    loop = asyncio.ProactorEventLoop()  File "C:\WPy64-3800\python-3.8.0.amd64\lib\asyncio\windows_events.py", line 310, in __init__    super().__init__(proactor)  File "C:\WPy64-3800\python-3.8.0.amd64\lib\asyncio\proactor_events.py", line 629, in __init__    self._make_self_pipe()  File "C:\WPy64-3800\python-3.8.0.amd64\lib\asyncio\proactor_events.py", line 760, in _make_self_pipe    self._ssock, self._csock = socket.socketpair()  File "C:\WPy64-3800\python-3.8.0.amd64\lib\socket.py", line 597, in socketpair    lsock.listen()OSError: [WinError 10050] A socket operation encountered a dead networkException ignored in: <function BaseEventLoop.__del__ at 0x0000000784F8C310>Traceback (most recent call last):  File "C:\WPy64-3800\python-3.8.0.amd64\lib\asyncio\base_events.py", line 648, in __del__  File "C:\WPy64-3800\python-3.8.0.amd64\lib\asyncio\proactor_events.py", line 684, in close  File "C:\WPy64-3800\python-3.8.0.amd64\lib\asyncio\proactor_events.py", line 752, in _close_self_pipeAttributeError: 'ProactorEventLoop' object has no attribute '_ssock'**

I raised a ticket to Azure Team. I was told it is because Azure runbook only allows on 31 kernel calls and this module seems to be using something out of this range hence getting this error.

Their product team is working on enabling full stack of OS calls from Azure runbook but it will be delivered for Python by end of 2022.

So, the temporary workaround is customer can use hybrid runbook worker as of now.

Azure Automation Hybrid Runbook Worker overview

Use Prometheus and Grafana to Visualize ISP Packet Loss

Eric Yi — Thu, 16 Sep 2021 07:28:06 +0000

Background

Since people started working from home, poor video/voice call connections have become a very annoying problem.

However, it's hard for people to diagnose what is the root cause. People start blaming their ISPs, and I feel bad for them. In fact, there are a number of reasons that can cause poor video call connections, but if you can tell us with great confidence that my problem is not from your end.

You should probably start monitoring your ISP packet loss.

Prerequisites

Grafana(You can register a free Grafana Cloud account)
Prometheus (You can install it on your Raspberry Pi or NAS)
Open Source Router OS (OpenWRT, ClearOS, Asuswrt-Merlin etc)

Process

Set Up ping_exporter

SSH in to your router > Download ping_exporter

You can check which version you need here.

For example:

wget https://github.com/czerwonk/ping_exporter/releases#:~:text=ping_exporter_0.4.7_linux_arm64.tar.gz

Extract the file:

tar -xf ping_exporter_*.tar.gz ping_exporter

Move to /usr/local/bin:

mv ping_exporter /usr/local/bin

Choose the way you like to get the ISP's DNS servers on your router.

For example:

cat /etc/resolv.conf

Create a ping_exporter.yaml file under /etc/ping_exporter/:

targets:
  - <Your ISP's DNS servers>

ping:
  interval: 2s
  timeout: 3s
  history-size: 42
  payload-size: 120

Create a ping_exporter.service file under /etc/systemd/system/:

[Unit]
Description=ping_exporter

[Service]
User=root
ExecStart=/usr/local/bin/ping_exporter --config.path /etc/ping_exporter/ping_exporter.yaml
Restart=always

[Install]
WantedBy=multi-user.target

Reload the service files:

sudo systemctl daemon-reload

Start your ping_exporter.service:

sudo systemctl start ping_exporter.service

Check the status of your ping_exporter.service:

sudo systemctl status ping_exporter.service

To enable your ping_exporter.service on every reboot:

sudo systemctl enable ping_exporter.service

Get the results for testing via cURL:

curl http://localhost:9427/metrics

Add a target to Prometheus

In your Prometheus configuration file, add the target in the static_config section :

- targets: ["<Your router's ip/dns address>:9427"]

The reload your Prometheus’ configuration.

Create a Grafana Panel

Add a new Panel in your home network's dashboard, we are going to use the ping_loss_percent metric.

ping_loss_percent{instance="<Your router's ip/dns address>:9427",target="<Your ISP's DNS servers>"}

⚠️ Please note for voice and video calls, any packet loss below 0.05 could be considered acceptable.

Triggering Jenkins Parameterized Builds Behind A Firewall

Eric Yi — Thu, 05 Aug 2021 01:24:36 +0000

Background

In this post I would like to share with you the process of how to trigger Jenkins parameterized builds behind a firewall.

Imagine your Jenkins is behind a corporate firewall but still able to receive webhooks from Github, DockerHub and Microsoft Automate.

I have seen many posts discussing related topics, but most webhook forwarding services cannot forward query parameters, which means they do not have the ability to trigger Jenkins builds with parameters.

Prerequisites

Webhook Relay account - register here.
Jenkins server

Process

Configuring Webhook Relay

Create a New Bucket here . Buckets allow you to group inputs and outputs.

A newly created bucket will have a default input and you will need to create an output destination where webhooks will be sent.

Click on OUTPUT DESTINATION> Fill in your custom Output name > Fill in your Jenkins server address in Output destination > click create output

# Output Destination Example
# If your Jenkins allows anonymous users can have build access.
http://192.168.0.200:8080
# If only authenticated users can have build access.
http://username:token@192.168.0.200:8080

Click CONFIGURE under your Output Destinations > Switch off Lock destination path > Click on SAVE.

It will allow us to send any extra path.
(xyz.hooks.webhookrelay.com/foo/bar -> 192.168.0.200:8080/foo/bar)

Configuring Webhookrelayd agent

Create an access token here. Access tokens are used to authenticate Webhook Relay agents.

On the Jenkins VM or a VM that can talk on the Jenkins server, start a webhookrelayd agent.

Specify configuration through environment variables:

KEY=<your token key>
SECRET=<your token secret>
BUCKET=<bucket name>

Run the webhookrelayd container:

docker run -d \
           --name webhookrelayd \
           --restart=unless-stopped \
           webhookrelay/webhookrelayd

Now go back to your Bucket page, you should see the status has changed to Client Connected. If there are any problems, you should check your docker logs.

Testing Jenkins Builds

Open your Terminal, let us test your Jenkins build.

Here is an example when trigger the Jenkins build normally:

curl http://192.168.0.200:8080/job/demo/buildWithParameters?token=<buildToken>&env=dev&version=20210804&user=john.doe

When using Webhook Relay, it will be converted to the following url:

curl https://<unique domain>.hooks.webhookrelay.com/job/demo/buildWithParameters?token=<buildToken>&env=dev&version=20210804&user=john.doe

You can also find the Relay Logs here.

Using GitHub Actions to Deploy CloudFormation Stacks

Eric Yi — Tue, 20 Apr 2021 11:40:03 +0000

Background

GitHub Actions is generally available at the end of 2019. Thanks to the open marketplace of GitHub Actions, more and more projects are using it.

Whether you are in the Devs or Ops team, you must be familiar with infrastructure as code. However, if you don't have a good pipeline, the deployment process can be a headache.

In this article, you will learn how to use GitHub Actions to deploy CloudFormation stacks.

Prerequisites

A GitHub Account (Click here to see free tier information)
A Programmatic Access IAM user in AWS
CloudFormation & S3 permissions

Process

Create CFN Templates

Let us create a simple CloudFormation template for creating an S3 bucket.

The filename is s3.yml.



  Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256

Then we would like to name our own S3 bucket, and for our template to be reusable, we want to pass parameters to CloudFormation stacks.



Parameters:
  BucketName:
    Description: Name your Bucket
    Type: String

Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      BucketName: !Ref BucketName
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256

Finally, we can add descriptions and format version and checkin the template in our repository. Click here to view the full template.

Create GitHub Workflows

Create .github/workflows/main.yml under our git root directory. Let us write our first worflow.

Configure AWS Credentials

We are using this action to configure AWS credentials.

"Configure AWS Credentials" Action For GitHub Actions - GitHub Marketplace

We can write the step according to the README.



- name: Configure AWS Credentials
  id: creds
  uses: aws-actions/configure-aws-credentials@v1
  with:
     aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
     aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
     aws-region: "ap-southeast-2"

In AWS IAM console, get your access key ID and secret access key.

AWS Account and Access Keys

Then navigate to the main page of GitHub repository > Click Setting > In the left sidebar, click Secrets > Click New repository secret > Create AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Create Deployment Actions

We are using this action to deploy CFN stacks.

AWS CloudFormation "Deploy CloudFormation Stack" Action for GitHub Actions - GitHub Marketplace

We can write the step according to the README.

As I am writing this article, the latest version is v1.0.3.



- name: Deploy to AWS CloudFormation
  uses: aws-actions/aws-cloudformation-github-deploy@v1.0.3
  with:
    name: S3Bucket
    template: s3.yml

We also need to pass the input parameter BucketName.



# Controls when the action will run.
on:
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:
    inputs:
      BucketName:
        description: "Name your Bucket"
        required: true
        default: "demo-bucket"

Then we add parameter-overrides in the Deploy to AWS CloudFormation step.



- name: Deploy S3 Buckets CloudFormation Stacks
  uses: aws-actions/aws-cloudformation-github-deploy@v1.0.3
  with:
    name: s3-buckets
    template: s3.yml
        parameter-overrides: >-
        BucketName=${{ github.event.inputs.bucketName }}

Let's finally organise our action file main.yml.



name: Deploy CloudFormation Stacks

# Controls when the action will run.

on:

  # Allows you to run this workflow manually from the Actions tab

  workflow_dispatch:

    inputs:

      region:

        description: "AWS Region"

        required: true

        default: "ap-southeast-2"

      bucketName:

        description: "S3 Bucket Name"

        required: true

# A workflow run is made up of one or more jobs that can run sequentially or in parallel

jobs:

  cfn-deployment:

    runs-on: ubuntu-latest

    steps:

      - name: Checkout

        uses: actions/checkout@v2

  <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Configure AWS credentials</span>
    <span class="na">id</span><span class="pi">:</span> <span class="s">creds</span>
    <span class="na">uses</span><span class="pi">:</span> <span class="s">aws-actions/configure-aws-credentials@v1</span>
    <span class="na">with</span><span class="pi">:</span>
      <span class="na">aws-access-key-id</span><span class="pi">:</span> <span class="s">${{ secrets.AWS_ACCESS_KEY_ID }}</span>
      <span class="na">aws-secret-access-key</span><span class="pi">:</span> <span class="s">${{ secrets.AWS_SECRET_ACCESS_KEY }}</span>
      <span class="na">aws-region</span><span class="pi">:</span> <span class="s">${{ github.event.inputs.region }}</span> 

  <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Deploy S3 Buckets CloudFormation Stacks</span>
    <span class="na">id</span><span class="pi">:</span> <span class="s">s3-buckets</span>
    <span class="na">uses</span><span class="pi">:</span> <span class="s">aws-actions/aws-cloudformation-github-deploy@v1.0.3</span>
    <span class="na">with</span><span class="pi">:</span>
      <span class="na">name</span><span class="pi">:</span>  <span class="s">s3-buckets</span>
      <span class="na">template</span><span class="pi">:</span> <span class="s">s3.yml</span>
      <span class="na">parameter-overrides</span><span class="pi">:</span> <span class="pi">&gt;-</span>
        <span class="s">BucketName=${{ github.event.inputs.bucketName }}</span>

Use GitHub Workflows

Make sure you have pushed all the files. Just like the following repo.

yunpengeric/Cloudformation-Github-Actions-Demo

Then navigate to the main page of GitHub repository > Click Actions > In the left sidebar, click Deploy CloudFormation Stacks > Click Run Workflow .

You should see the following pre-filled data > Enter your S3 bucket name (Please note an S3 bucket name is globally unique) > Run Workflow

After a minute or so, you will see that our S3 bucket has been successfully deployed. If you see any errors, please leverage GitHub logs and CloudFormation console to troubleshoot.

Conclusion

We are still far from it if our goal is to create a good pipeline in production environment. You should take these things into consideration as well.

How you can rotate keys to ensure security.
How to modify your CFN templates so that when you deploy a second bucket, you don't delete the first one.