DEV Community: Yash Kumar Shah

Building Multi-Arch Images Inside Kubernetes Using BuildKit

Yash Kumar Shah — Wed, 10 Jan 2024 12:09:27 +0000

Problem Statement

A SaaS company, when rolling out applications in a customer's setup, often deals with a mix of different architectures. To ensure we can support all these variations, we create Docker images in a multi-architecture format. In the current landscape, many companies leverage CI/CD in Kubernetes and utilize tools like Kaniko for their Docker-building processes. It is important to note that Kaniko ( on the day of writing ) itself lacks support for the multi-architecture build of Docker images.

Prerequisite

Before implementing multi-architecture builds using Kubernetes, it's essential to have a Kubernetes cluster with Jenkins Setup and helm installed in your local to set up the buildkit helm chart. At Aurva, we streamline our Jenkins configuration using a shared library, coupled with Jenkins Configuration as Code (JCasC), to set up and manage Jenkins jobs efficiently.

Introduction to BuildKit

Introducing BuildKit, a solution for optimizing container image construction. This tool enhances efficiency, security, and speed in the image-building process and is seamlessly integrated into Docker's latest release (v18.06). As an integral component of the Moby project.

Setting Up BuildKit

Assuming you already have a Jenkins setup in a Kubernetes namespace named Jenkins the next step is to install the BuildKitservice within that namespace.

Step 1: Helm Chart Magic

Start by deploying the BuildKit service using Helm. Execute the following command in your terminal:

helm repo add andrcuns https://andrcuns.github.io/charts helm install buildkit andrcuns/buildkit-service --namespace jenkins

Step 2: Setting Up Docker Agent

Now, let's take the next stride by tweaking your Kubernetes agent pod in your Jenkins Groovy script. Use the below Dockerfile to create your docker image to be used for building agents

Dockerfile Example

`FROM --platform=${BUILDPLATFORM} docker:rc-dind-rootless as build
ARG TARGETARCH
WORKDIR /home/rootless
RUN wget -O docker-credential-ecr-login https://amazon-ecr-credential-helper-releases.s3.us-east-2.amazonaws.com/0.7.1/linux-${TARGETARCH}/docker-credential-ecr-login

FROM --platform=${TARGETPLATFORM} docker:rc-dind-rootless
COPY --from=build /home/rootless/docker-credential-ecr-login /home/rootless/.local/bin/docker-credential-ecr-login
RUN chmod +x /home/rootless/.local/bin/docker-credential-ecr-login
ENV PATH="/home/rootless/.local/bin:${PATH}"
USER rootless`

Jenkins Agent.yaml
`
def buildkit_agent() {
template = """
apiVersion: v1
kind: Pod
metadata:
name: buildkit-agent-pod
namespace: build
spec:
serviceAccount: "aurva-jenkins-worker-sa"
containers:

name: buildkit-agent image: docker:rc-dind-rootless command:
- /usr/bin/bash tty: true

"""
template
}`

Note: "In this blog, we won't delve into an ECR repo configuration. However, if you're keen on setting up an ECR repo, you can leverage AWS ECR Credential Helper to streamline pushing images to ECR."

Step 3: Create a Remote Driver & Execute the Docker Build

1. Remote Driver Setup:

BuildKit requires the creation of a remote driver that points to the BuildKit service. This command can be executed from the Docker docker-agent shell, and running the command will create a remote driver.

docker buildx create --use --driver=remote tcp://buildkit-buildkit-service.jenkins:1234

The above command will create a remote Docker drive and point to the BuildKit using a BuildKit service deployed in the Jenkins namespace.

2. Build Docker Images:

It's time for the final step, run the following command to build Docker images and push them to your Amazon ECR repository.

docker buildx build - push -fpwd/${dockerFilePath} -t ${imageFullName} - platform linux/arm64/v8,linux/amd64pwd`/${buildContextPath}

Output will be

docker buildx build --push -f /home/jenkins/agent/workspace/aurva-gateway/docker/dockerfiles/gateway.Dockerfile --build-arg ENV_TAG= -t .dkr.ecr.ap-south-1.amazonaws.com/: --platform linux/arm64/v8,linux/amd64 /home/jenkins/agent/workspace/aurva-gateway/`

In the aforementioned docker buildx command, there is a 'platform' parameter where you can define all the platforms for which you want to build the image

Conclusion

In a nutshell, this guide explores implementing multi-architecture Docker builds in Kubernetes using BuildKit. We addressed the challenge of a diverse docker arch, introduced BuildKit as a seamless solution integrated into Docker, and outlined key steps for setup. By adopting these practices, you'll boost your CI/CD pipelines for flexible and efficient deployment across various architectures, staying ahead in containerized application development.

AWS Anycast Service Global Accelerator

Yash Kumar Shah — Sat, 16 May 2020 14:34:58 +0000

Yash Shah

Problem Statement

Recently We Guys were facing issues with whitelisting our application to third parties behind an ALB. As ALB IP gets changes frequently. Came up on a solution using NLB and assign Elastic IP. But Still Many features of ALB we were missing.

We started looking for a solution and came up with the latest AWS service Global Accelerator. The following service not only helps what we need but also baked many use cases which are very fascinating.

Setting up Regional DR
Easily move endpoints between Availability Zones or AWS Regions without needing to update your DNS configuration or change client-facing applications
Control traffic up or down for a specific AWS Region

About AWS Global Accelerator?

Global Accelerator became publicly available in late 2018

The main use case for users is the ability to get a static IPv4 address that isn’t tied to a region

With global accelerator, customers get two globally anycast IPv4 addresses that can be used to load balance across 14 unique AWS regions

A user request will get routed to the closest AWS edge POP based on BGP routing. From there, you can load balance requests to the AWS regions where your applications are deployed

Global accelerator comes with traffic dials that allow you to control how much traffic goes to what region, as well as instances in that region. It also has built-in health checking to make sure traffic is only routed to healthy instances.

AWS Global Accelerator always routes user traffic to the optimal endpoint based on performance, reacting instantly to changes in application health, your user’s location, and policies that you configure

Some primitive in Global Accelerator

Global Accelerator, think of this of your anycast endpoint. It’s a global primitive and comes with two IPv4 addresses.

Listener which defines what port and protocol (TCP or UDP) to listen on. A Global Accelerator can have multiple listeners

For each Listener, you then create one or more Endpoint Groups. An endpoint group allows you to group endpoints together by region. For example a region ap-south-1 can be an endpoint group.You can even control the percentage of traffic that you want to send to that region.

For every Endpoint group we have multiple Endpoint, this can be either an Elastic IP address, a Network Load Balancer, or an Application Load Balancer. For each endpoint, you can configure a weight that controls how traffic is load-balanced over the various endpoints within an endpoint group.

Hand's On

Prerequisite:

Setup two small app behind alb in different region. So that we can connect to the Endpoint in Global Accelerator.

Whenever you open Global Accelerator you will be redirected to Oregon.

Step:-
The complete global Accelerator creation is a 4 step process that we will be going through step by step.

Step 1:

Create an Accelerator.

Provide the accelerator name. If you go Bring your own IP addresses (BYOIP) then you can choose the IPv4 option in the accelerator. Otherwise, as will assign two static IPs from amazon's pool of IP Addresses. Finally tag your accelerator with required key-name accordingly.

Step 2:

Add Listener to the accelerator. Specify a port, port range, or multiple port ranges that you want the listener to listen on.

Note :- If you have stateful applications, Global Accelerator can direct all requests from a user at a specific client IP address to the same endpoint resource, to maintain client affinity.

By default, client affinity is None and Global Accelerator distributes traffic equally between the endpoints in the endpoint groups for the listener.

Step 3

Add an endpoint group for each AWS Region that you want to direct traffic

An endpoint group is associated with a specific AWS Region. Endpoint groups include one or more endpoints in the Region.

For each AWS Region that you want to direct traffic to, add one endpoint group. You can't have more than one endpoint group per Region.

You can increase or reduce the percentage of traffic that would be otherwise directed to an endpoint group by adjusting a setting called a traffic dial.

Step 4

Last Step is the creation of the endpoint.

Note :- If you are going for EC2 Instance then you have to specify the health check for your application. If you are going for LoadBalancer's then it will use the health check used by loadbalancers for traffic sending.

Time to start testing.

Now that we have our anycast global load balancer up and running, it’s time to start testing. To check if load balancing works as expected, I’m using to following test:

for i in {1..100}; do curl -s -q http://13.248.138.197 ;done | sort -n | uniq -c | sort -rn

Global Accelerator Routing vs Normal Routing

An interesting difference between Global Accelerator and regular public ec2 IP addresses is how traffic is routed to AWS. For Global Accelerator, AWS will try and get traffic on its own network as soon as possible. As compared to the scenario with regular AWS public IP addresses; Amazon only announces those prefixes out of the region where the IP addresses are used. That means that for Global Accelerator IP addresses your traffic is handed off to AWS at its closest Global Accelerator POP and then uses the AWS backbone to get to the origin. For regular AWS public IP addresses, AWS relies on public transit and peering to get the traffic to the region it needs to get to

Let’s look at a traceroute to illustrate this. One of my origin servers in the ap-south-1 regions is 13.232.169.234, a traceroute to that ec2 instance in Mumbai from my local machine

In this case the handoff is from static-delhi.vsnl.net.in to static-mumbai.vsnl.net.in via public routing.

Now, as a comparison, we’ll look at a traceroute from the same server in Mumbai to the anycast Global Accelerator IP

The Global Accelerator only relies on lesser hops than the previous one. In this case, AWS is announcing the prefix locally via the Internet Exchange and it is handed off to AWS on the second hop. Quite a latency difference.

Conclusion

I think Global Accelerator is a powerful service. Having the ability to flexibly steer traffic to particular regions and even endpoints gives you a lot of control, which will be useful for high traffic applications. It also makes it easier to make applications highly available even on a per IP address basis (as compared to using DNS based load-balancing). A potentially useful feature for Global Accelerator would be to combine it with the Bring Your Own IP feature.

I personally used the Global Accelerator to provide 2 static IP to my ALB for third party whitelisting.

Centralized Monitoring Stack Setup Dilemma

Yash Kumar Shah — Mon, 10 Feb 2020 12:55:14 +0000

Yash Shah

When I started setting up centralized monitoring for an organization level. I came across a few of the stacks that attracted me. This post is regarding those who want to start setting up their monitoring can read and get a clear view of what things are indifferent stack and where to approach to set up a perfect "chowkidar 😊😊😊😊" to send anomalies happening over the entire Infra

Tick Stack
Prometheus and Grafana
Sensu

Tick Stack

T (Telegraf): Telegraph is an open-source agent that help to collect metrics from server sensor and systems.

I (Influx): Influx DB is a time-series database designed to handle high write and query loads.

C (Chronograf): Chronograf is the user interface and administrative component of the InfluxDB 1.x platform.

K (Kapacitor): Kapacitor is a native data processing engine for InfluxDB 1.x and is an integrated component in the InfluxDB 2.0 platform.
Kapacitor can process both stream and batch data from InfluxDB, acting on this data in real-time via its programming language TICKscript.

Tick uses a more traditional method where agent connects to a central monitoring system, Here the agent (Telegraf) which is a pluggable piece of software and supports multiple inputs and output plugins for specific infrastructure monitoring. Its plugins also allow it to connect to different communication methods like statsD, Nagios Plugins and two-way integration with Prometheus.

TICK is free, easy to install, based on one server/DB combo as the monitoring engine and since it is backed by a company, it also provides Enterprise support and database clustering to those who don’t care to shell out some cache for a complete solution

TICK is both easy to deploy and based on an official DB, it is free an OpenSource.

Prometheus

Prometheus is based on pull-based metrics it means the Prometheus server approaches the open port in agent server for the metrics rather than the agent connecting to the Prometheus server.

Prometheus can also use Tick Stack agent telegraf as an exporter.

Prometheus is easy to install, using one server as the central monitoring system and storage, it has a built-in DB created just for saving monitoring data

For high availability the instruction is to use two different servers, both monitoring the same exporters. Here is where the pull method comes in handy, since the exporters don’t know of the server’s address, any number of servers can connect to them and pull the data. The alerting component has the ability to de-duplicate alert when connected to two servers.

Prometheus is also 100% free which makes it the easiest & cheapest HA monitoring solution in the market.

Sensu

Sensu has been around since the past 2011 and has a noticeable market share of around 7%, its architecture is geared towards massive amounts of data, so it uses RabbitMQ to pipe and buffer the monitoring information between its collectors and it’s the main server.

Sensu has its own collection plugins and supports Nagios plugins as well.
The Sensu servers are built for High Availability out of the box, but they don’t include a DB for storing the data, it uses Redis by default but for storing more than several hours, you will need to include either InfluxDB or ElasticSearch in your installation, both of which will require an enterprise license if you want enterprise features (InfluxDB charges for clustering, ElasticSearch charges for security)

In Sensu alerts are called checks, and creating one involves creating a cron job and configuring a JSON file which tells it what ruby script to run, what statistic to check (including some calculations) and whom to notify. It doesn’t seem as straight forward as the other solutions.

For a free solution i would definitely go with prometheus, if DB clustering is a specific requirement TICK is your solution, both of them are easy to install with a helm script and some tinkering. for creating nice dashboards, there is no dilemma Grafana is the absolute best and while TICK still develops it’s own UI (Chronograf), all of them integrate with it

Puzzled Around ARG ENV and .env in Dockerfile

Yash Kumar Shah — Sun, 19 Jan 2020 07:58:39 +0000

Yash Shah

Prerequisite :- Basic understanding of Dockerfile

Recently I came to a use case where i had to build image on the basis of params which depend on the environment i am currently building my container. Hence i use ARG to pass Arguments while building the container. And setting up ENV on the basis of ARG that i have provided.

The .env file

It works with docker-compose.yml. You can simple use $ notation in your compose file to pick value from .env, if both docker-compose.yml and .env is in same directory

The .env looks something like this

first_variable=a
second_variable=b

The key-value pair is used in docker-compose.yml by simply using $ notation within the docker-compose.yml file. Lets look and example.

services:
  demo:
    image: linuxserver/demo
      environment:
        - env_var_name=${first_variable}    # here it is

1. To check thing are working fine in docker-compose.yml you can simply use docker-compose config, This way you can check how your docker-compose.yml will look after substituting values from .env

2. There is a glitch the environment variable in host machine can override the variable in your .env file

Different types of variable in Dockerfile

ARG : ARG also known as build time variable, there availability is available the moment they are announced in the dockerfile till the docker image is build. Somewhat like CMD and ENTRYPOINT which tells the container what to run by default. If you tell a Dockerfile to expect various ARG variables (without a default value) but none are provided when running the build command, there will be an error message

ENV: ENV are available during the run time, you can also override them during run time.

How to use ARG variables

So if you have a Dockerfile and if you want to set them? You can set them blank or leave a default value and you can override them during the docker build.

FROM ubuntu:18.04
ARG any_default_variable
# or with a hard-coded default:
#ARG any_default_variable=any_default_value

RUN echo "Hi $any_default_variable"
# you could also use braces - ${any_default_variable}

You can use the docker build command to send the build args

docker build -t demo --build-arg any_default_variable=yash  .

The above command will produce the following output

Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM ubuntu:18.04
 ---> ccc6e87d482b
Step 2/3 : ARG any_default_variable
 ---> Running in ec62ad011b88
Removing intermediate container ec62ad011b88
 ---> 3d8df126a04c
Step 3/3 : RUN echo "Hi $any_default_variable"
 ---> Running in e9f5fb20897e
Hi yash
Removing intermediate container e9f5fb20897e
 ---> d6fe8d6e0f9a
Successfully built d6fe8d6e0f9a
Successfully tagged demo:latest

How to use ENV variables

ENV variables are also available during the build, as soon as you introduce them with an ENV instruction. However, unlike ARG, they are also accessible by containers started from the final image. ENV values can be overridden when starting a container

Setting up ENV variables

You can do it when starting your containers, but you can also provide default ENV values directly in your Dockerfile by hard-coding them. Also, you can set dynamic default values for environment variables!

When building an image, the only thing you can provide are ARG values, as described above. You can’t provide values for ENV variables directly. However, both ARG and ENV can work together. You can use ARG to set the default values of ENV vars. Here is a basic Dockerfile, using hard-coded default values:

# no default value
ENV hey
# a default value
ENV foo /bar
# or ENV foo=/bar

# ENV values can be used during the build
ADD . $foo
# or ADD . ${foo}
# translates to: ADD . /bar

And here is a snippet for a Dockerfile, using dynamic on-build env values:

# expect a build-time variable
ARG A_VARIABLE
# use the value to set the ENV var default
ENV an_env_var=$A_VARIABLE
# if not overridden, that value of an_env_var will be available to your containers!

Once the image is built, you can launch containers and provide values for ENV variables in three different ways, either from the command line or using a docker-compose.yml file. All of those will override any default ENV values in the Dockerfile. Unlike ARG, you can pass all kinds of environment variables to the container. Even ones not explicitly defined in the Dockerfile. It depends on your application whether that’ll do anything however.

1. Provide values one by one

From the commandline, use the -e flag:

docker run -e "env_var_name=another_value" alpine env

2. Pass environment variable values from your host

It’s the same as the above method. The only difference is, you don’t provide a value, but just name the variable. This will make Docker access the current value in the host environment and pass it on to the container.

docker run -e env_var_name alpine env

3. Take values from a file (env_file)

The file is called env_file_name (name arbitrary) and it’s located in the current directory. You can reference the filename, which is parsed to extract the environment variables to set:

docker run --env-file=env_file_name alpine env

if you’re new to Docker and not used to think about images and containers: if you try to set the value of an environment variable from inside a RUN statement like RUN export VARI=5 && ..., you won’t have access to it in any of the next RUN statements. The reason for this, is that for each RUN statement, a new container is launched from an intermediate image. An image is saved by the end of the command, but environment variables do not persist that way.

If you’re curious about an image, and would like to know if it provides default ENV variable values before the container is started, you can inspect images, and see which ENV entries are set by default:

# first, get the images on your system and their ids
$ docker images
# use one of those ids to take a closer look
$ docker inspect image-id

# look out for the "Env" entries

Conclusion

By now, you should have a really good overview build-time arguments, environment variables, env_files and docker-compose templating with .env files. I hope you got a lot of value out of it, and can use the knowledge to save yourself lots of bugs in the future. You have to see them in action and apply them to your own work to truly make them part of your tool belt. The best way to make sure you will be able to make use of this information, is to learn by doing – go ahead and try some of those techniques