DEV Community: moeed-k

Containers, Clusters & K8s: My Experience with DevOps

moeed-k — Sat, 10 Jun 2023 12:57:37 +0000

How do you even pronounce Kubernetes?

Hint: coo-ber-net-ees

In this post I'll be recounting some of the invaluable lessons I've learned over the past few months. At the start of the final semester of my undergrad, I elected an introductory course to DevOps. I'd heard the term here and there, and I also had some idea about just how popular the role of a DevOps engineer had become in recent years. But other than that, I really had no clue about what the role entailed.

Fast forward 4 months to today and I can proudly say that I'm far less clueless than when I had started out. Heck, I even managed to deploy my own Kubernetes cluster in a working project.

In a nutshell, DevOps is all about streamlining the processes involved in software development & deployment to make life easier for everyone (the developer, the software users, and the maintainers).

Here's some of the stuff I learned about (and will be discussing in this post as well):

Virtualization
Containers
Pods
Dockerfiles
Building dockerfiles
Container Clusters
Achieving High-Availability
Micro-Services
Service meshes
CI/CD
GitOps

Not to mention a whole plethora of tools. Really, there's just so many great DevOps tools out there, that it's equally rewarding and daunting trying to learn about them all. I mean, just take a look at this periodic table of DevOps tools.

But more important than the tools, is to understand the need for DevOps in the first place.

What's a container, and why does my cluster have so many of them?

Before we answer that question, we need to know a bit about virtualization. Traditionally, Virtual Machines were used to run several distinct applications on a single hardware system. In most cases, this involved installing and configuring the VM software, installing the OS, and then finally deploying the application. As you can see, there's many different layers to this process. What if all I wanted to do was deploy a small service that only provides me some kind of button highlighting functionality in a web app? Do I really need to spin up an entire VM? That doesn't sound too efficient.

This is where containers come in. Compared to regular VMs, containers use a highly streamlined form of OS virtualization. They can be thought of as logically isolated processes that contain everything needed to run a particular piece of software. All the dependencies, source files and environment configurations are bundled into a neat little container. Moreover, these containers are highly portable and function exactly the same no matter where you run them. There's no need to configure the developer environment for each machine you chose to deploy a container on. Just get the container up and running and you'll execute the exact same code across every machine you choose to run it on.

In order to run a container, there's a three step process involved. You first write something called a dockerfile, which is basically a schematic defining the structure of your container. Then using this schematic you build a container image with the help of an image builder. Then finally, you run the container in a container runtime.

If you take one or more containers and bundle them into a wrapper, you get something called a pod. Each pod exists to serve a particular function, something that you can call a service. A machine that runs multiple pods together is called a node, and a container orchestration software like Kubernetes manages a group of several nodes called a cluster. It handles high-availability and resilience, auto-scales up or down based on needs, provides authorization, and does many, many more things.

That's pretty cool! But there's one glaring problem to address. How do we make each of these little services running on their own pods (micro-services basically) communicate with each other in a safe and efficient manner? For that we use something like Istio. It's a piece of software that injects a separate container called a sidecar-proxy inside a pod (pods are wrappers for multiple containers, remember?), and all communication going to and from the pod passes through the sidecar-proxy first. This sidecar provides many extremely useful features that vastly improve and standardize the communication between services. A whole network of these proxies is called a service mesh.

There's a whole lot more to the world of containers and container orchestration, but this is pretty much the gist of it.

CI/CD, Version Control, Source Control, Single Source of Truth...

A big part of DevOps is creating CI/CD pipelines that make the process of software development efficient and reliable. CI/CD is short for Continuous Integration/Continuous Development, and it basically means creating a set of automated steps that allow a developer to have changes in their code quickly deployed into a running form.

Here's what a basic CI/CD workflow would look like:

A developer writes some code and makes a commit to their code repository.
A tool like GitHub Actions or Jenkins builds a container image from the code and runs some user-defined tests against it.
If the tests clear, the image is pushed to a Container Registry (think of it like GitHub but for containers).
The image is pulled from the registry and deployed inside a Kubernetes cluster.

If you use a tool like Flux or Argo CD, you can add one more dimension to this whole process by introducing GitOps to your workflow. Flux, for example, will constantly monitor the configuration of your current cluster and compare it against a configuration repo defined by the developers (this is usually just a git repo too). If it finds any discrepancy between the desired state and the current state, Flux will bring the cluster state in line with the configuration defined in the git repo. This is why its called Git Ops, because Git acts as the Single Source of Truth.

Another crucial concept is Version/Source control. This basically refers to keeping track of changes made to code. Most of us are familiar with version control in one way or the other (such as git), but there's a different dimension to it that I discovered recently. Rolling new updates to an already running software platform can be tricky. But tools like Kubernetes provide features like Deployments that help route traffic between different software versions, and Istio does something similar at a more granular level to route traffic between canary builds of different services in the service mesh.

The Open Source Community

An amazing thing about the world of DevOps is that so much of all the magic making it possible is open-source. The communities behind each tool are constantly pushing out new updates and improving features by the day.

During my course, this inspired me to take a deeper look at some of these tools in the hopes of contributing to them. A crucial thing to remember is that you don't have to start out big, in fact that's near impossible. Start small, but start early. Remember, the quicker you fail at something, the quicker you get to learn and improve.

In order to get started with open-source contributions, you can begin by scoping out the GitHub repo for the project you'd like to work on. Take a look at the issues list posted under each repo, and see if you can find something that makes sense to you. Usually, you'll find issues labeled with tags such as 'bug' or 'good first issue', and this will help you identify the nature of the problem you want to solve.

Do be mindful of the guidelines that each project follows. If you can't find them in the documentation or on the project website, you can try looking through previously merged pull-requests and try to mimic their code structure.

It can take a while to get a grasp of everything and to know where and how to start, but don't give up. Just keep at it and soon enough you'll find yourself making just a tiny bit of sense of things. After that, write down your solution, make a Pull-Request, and then wait for the code to get approved. Don't lose heart if it doesn't get merged. Use any feedback you get to improve your next contribution. Even something as simple as updating the documentation can be really useful to the wider community, so don't shy away from the vast world of open-source!

A wonderful platform that incubates many amazing tools is called the Cloud Native Computing Foundation. They bring together a great many useful tools under one hub, promote development, and are a rich source of information for anyone interested in following open-source development. I highly recommend that you check it out.

Conclusion

These past few months have been a highly educative experience. Even though I have only scratched the surface of the vast world of DevOps, I have already begun to appreciate the complexity of all the processes involved in giving life to so much of modern software. Development isn't just about coding. It's also about getting that code to run everywhere, all the time, for everyone. My course has come to an end, but my journey with DevOps has just begun.

How to Deploy a PostgreSQL cluster on Kubernetes

moeed-k — Wed, 10 May 2023 15:45:02 +0000

A PostgreSQL cluster can be thought of as a collection of databases copied across several instances. Each instance can be thought of as an independent node containing all your databases. Normally, we have more than one node per cluster in order to deploy a highly-available database solution.

A PostgreSQL cluster cannot be directly deployed on Kubernetes because it has some specific requirements. These requirements can be fulfilled by a Kubernetes Operator that uses a Custom Resource Definition (CRD) to help manage and deploy PostgreSQL.

Today we'll be taking a look at using the Zalando Postgres Operator to deploy a PostgreSQL cluster using K8s.

Pre-requisites:

Have Kubernetes installed (for this blog I'll be using minikube).
Have kubectl installed.

Manual Deployment:

Create an empty directory to start working in. First we'll clone the zalando postgres operator repo from github.



git clone https://github.com/zalando/postgres-operator.git
cd postgres-operator

Next we'll apply the YAML manifests in the given order:



kubectl create -f manifests/configmap.yaml  
kubectl create -f manifests/operator-service-account-rbac.yaml 
kubectl create -f manifests/postgres-operator.yaml  
kubectl create -f manifests/api-service.yaml

Next we run a script provided in the repo that helps us set-up an acid-minimal-cluster.



./run_operator_locally.sh

That's it! We now check if the postgres operator is operational.



kubectl get pod -l name=postgres-operator

This should show us the pod in a RUNNING state.

Deploying the UI

Next we'll create our cluster. But first, we'll deploy the UI so that it makes this process easier to visualize. To deploy the UI, run the following command:



kubectl apply -f ui/manifests/

This can take a while to go into the RUNNING state, so check with the following command till the pod is up.



kubectl get pod -l name=postgres-operator-ui

After both the Operator and UI pods are up, we now port-forward to access the interface from our browser.



kubectl port-forward svc/postgres-operator-ui 8081:80

Creating a Cluster

Open up your local browser and access localhost:8081.

You can configure your cluster any way you like. I'll be going forward with the configuration. Press the create cluster button when you're done.

Wait until all the status checks have cleared.

Connecting to the cluster using PSQL

PSQL is a CLI that allows us to query postgres. To use it with our newly deployed cluster, we run port-forward the default 5432 used by Postgres to 6432. We also set up the name of the master pod of our acid-minimal cluster as an environment variable.



export PGMASTER=$(kubectl get pods -o jsonpath={.items..metadata.name} -l application=spilo,cluster-name=acid-minimal-cluster,spilo-role=master -n default)


kubectl port-forward $PGMASTER 6432:5432 -n default

Setup the following environment variables as well (one define the SSLMODE and the other defines the K8s secret when creating acid-minimal-cluster):



export PGPASSWORD=$(kubectl get secret postgres.acid-minimal-cluster.credentials.postgresql.acid.zalan.do -o 'jsonpath={.data.password}' | base64 -d)
export PGSSLMODE=allow

Now connect with PSQL to the default postgres database:



psql -U postgres -h localhost -p 6432

We have successfully deployed our K8s PG cluster. You can now run queries as you like. In the next part we'll take a look at deploying Apache-AGE load balancing using Pg-Pool II for AGE.

References:

https://github.com/zalando/postgres-operator/blob/master/docs/quickstart.md

Improving build times with Docker Layer Caching

moeed-k — Thu, 23 Mar 2023 08:55:29 +0000

An important concept in container image building is layer-caching. It is important to understand in order to optimize your build times and to streamline your CI/CD workflows.

In this post, we'll be taking a look at three different aspects of layer-caching:

Optimizing dockerfiles to re-use the maximum number of layers.
Layer-caching between two different images using --cache-from.
Using BuildKit's inline cache.

Understanding Layers

Let's take a look at an example of a simple dockerfile:

# pull base image
FROM ubuntu:latest

# install wget
RUN apt-get update && \
    apt-get -y install wget

Here, we can observe two layers. The first one pulls the base image, and the second one installs wget.

With layer caching, our aim is to build a minimal amount of layers in each build, while re-using previous layers as much as possible. In essence, we're trying to reduce the number of steps in our build process.

Optimizing Dockerfiles to re-use layers

If we build the simple dockerfile we took as an example above, it will take a few seconds for the base image to be pulled. If we change the second layer to:

# install wget and nginx
RUN apt-get update && \
    apt-get -y install wget && \
    apt-get -y nginx

And then re-build, we will see that only the second layer will be run again, and the base image will not be pulled again. This is because the first layer has been cached and is being re-used.

However, if we change our first layer, we will be invalidating all subsequent layers that come after it.

# pull different image
FROM ubuntu:18.04

# install wget nginx
RUN apt-get update && \
    apt-get -y install wget && \
    apt-get -y nginx

This time, all layers after the first layer will be built again.

Keeping this in mind, we should always try to follow the two following best practices:

Keep commands that are not likely to change at the start of the dockerfile.
Commands that are going to change often should ideally be near the end of the dockerfile.

Using layer-caching between Images

If there is an image that has already been built and it shares some of the layers with your own dockerfile, you can use that image as part of the build cache with the help of the --cache-from flag.

For example:

IMG="my-image"

# Pull an existing image
docker pull ${IMG}:old-ver

docker build --cache-from ${IMG}:old-ver -t ${IMG}:new-ver .

BuildKit Inline Cache

The problem with the above approach is that it requires us pulling an image from a remote registry first. By using BuildKit's inline cache, we can cache images to our local registry in order to avoid expensive pull operations.

export DOCKER_BUILDKIT=1

# Build and cache image
$ docker build --build-arg BUILDKIT_INLINE_CACHE=1 .

Set up Apache-AGE for development: Installing and Modifying the Source

moeed-k — Mon, 20 Feb 2023 19:09:07 +0000

Apache-AGE is an exciting open source graph-database extension for PostgreSQL. It basically turns PGSQL into a multi-model DB. Which is to say, it allows you to enhance your good ol' relational database so that it supports graph DB functionality too.

Contributing to open source projects can be very rewarding. It is both a great way to improve your coding skills and also to help grow your favorite tools.

However, it can be daunting if you're a beginner and aren't quite sure where to start. It's always a good idea to hunt down any existing documentation and then work your way through that. You'll eventually exhaust your resources though, and at that point it is best just to start reading through the code itself. Read comments inside the code. Use a debugger. Set breakpoints. Try to understand the flow of execution.

But even before you get to that, you need to set up an environment for development! You want to be able to make changes in the code and quickly see it get reflected in the running program.

That's what this post will (hopefully) help you with. We'll go through the process of:

Installing Postgres from source
Installing AGE from source
Using a debugger with AGE
Updating the AGE source code

NOTE: This guide has been made using Ubuntu in mind.

1. Installing Postgres from Source

The first thing we need to do is get a version of PG that is compatible with AGE. At the time of this writing, AGE supports both PG 11 and 12, so we'll be going with V12.

Download the source from here:
https://www.postgresql.org/ftp/source/v12.0/

This guide assumes that you download the .gz file. Go to your Downloads directory, open up a new terminal (right click inside the directory and select 'open terminal') and decompress the file using the following command:

gzip -d postgresql-12.0.tar.gz

Next, we unpack the tarball:

tar -xf postgresql-12.0.tar

Now cd into the newly created directory. But before we do anything else, we're going to install two packages that are used by both AGE and PG to parse/lex queries. These packages are FLEX (a tokenizer) and BISON (a parser).

sudo apt-get install build-essential libreadline-dev zlib1g-dev flex bison

Now we're ready to install PG. First we'll run the configure command:

./configure --enable-debug --enable-cassert

NOTE: If you're wondering more about what the --enable-debug and --enable-cassert flags do, you can check it out from the PG docs at: PG Docs.

Now lets run the make commands:

make all
sudo make install

Now we should have Postgres installed. Lets try running it. We can start the PG server using the pg_ctl utility, but first we need to tell our system where the binaries for that are located. So we'll add the (default is /usr/local/pgsql/bin) location to our PATH variable:

PATH=/usr/local/pgsql/bin:$PATH
export PATH

Note that if you want to make it permanent (so that the PATH is updated each time you open a new terminal), you should edit your ~/.bash_profile file and append the export command.

Now we can initialize our first DB cluster:

initdb -D $HOME/pgdata

The -D flag just specifies the location of the DB cluster. I'm choosing to make it inside a folder called pgdata inside my $HOME path.

Time to start the server and test it out:

pg_ctl -D $HOME/pgdata start

If everything has been done correctly till this point, our server should have started successfully!

Let's test it out. We'll use a tool called psql, which is a command line interface to interact with Postgres. For this guide, I'll just connect to the default DB named 'postgres'.

psql postgres

If everything is working good so far, it's time to install AGE. Use \q to quit out of psql for now.

2. Installing AGE from Source

Download or clone AGE from the github repo:
AGE Github

Extract it just like before. Now before we install anything, we'll make a slight change to make debugging easier down the line.

Open the makefile, and go the line that starts with 'PG_CPPFLAGS'. At the end of this line, append the -O0 flag. This will tell compiler to keep the optimization level at 0, which will make it easier to step through the code with the debugger.

Now to install AGE. Assuming you didn't change anything and kept all the default paths, your PG install dir should be =/usr/local/pgsql/. With that in mind, run the following:

sudo make PG_CONFIG=/usr/local/pgsql/bin/pg_config install

We've now installed AGE!

3. Using a Debugger with AGE

Now let's go through the process of using a debugger to analyze the AGE code. We'll be using the GNU Debugger, commonly known as GDB (it comes pre-installed with Ubuntu).

First, let's load up our newly installed AGE extension. Use psql to connect to the 'postgres' DB again.

psql postgres

Now run the following commands:

CREATE EXTENSION age;
LOAD ‘age’;
SET search_path = ag_catalog, "$user", public;

The CREATE EXTENSION only has to be run once, but do keep in mind that the LOAD and SET commands have to be re-entered every time you log back into psql.

Using the create_graph command in the ag_catalog namespace, we'll create our first graph called 'people'.

If you want more information on how to write cypher queries using AGE, you can look up the AGE documentation at: AGE Docs.

SELECT * FROM ag_catalog.create_graph('people');

Now let's add one person to this graph.

SELECT * 
FROM cypher('people', $$
    CREATE (a {name: 'Andres'})
    RETURN a
$$) as (a agtype);

Okay, now time to debug the code. Open a new terminal. We need to attach GDB to the running PGSQL process. To find the PID of the process, we use:

ps -ef | grep postgres

You'll get a lot of IDs (since PG runs many background processes), but we only need to look for the PID of the process connected to the default database named 'postgres'.

In my case, it looks like this:

From here I can see that the PID I'm looking for is 21812.

Now we start gdb:

sudo gdb

From inside the GDB interface, we attach to the process:

attach 21812

But even now GDB doesn't know where the source files are stored on our system. So we need to update its search path. In my case, the files are inside my Downloads directory.

dir /home/moeed/Downloads/age-PG12-v1.1.1-rc1

GDB commands refresher:
b for breakpoint, (b )
c for continue - continues to the next breakpoint
n for next line
s for step into
p for print, (p *) for pointers
bt for call stack
d for delete all breakpoints
list for context
q for quit

Now let's set a breakpoint for a function call somewhere. A good point to start would be the AGE parser. If you go through the source code, you'll find a function called 'parse_cypher' inside the cypher_parser.c file.

b parse_cypher

With the breakpoint set, go back to the psql terminal and run the following query:

SELECT * 
FROM cypher('people', $$
    MATCH (a)
    RETURN a
$$) as (a agtype);

This will return us the one node we created earlier inside the 'people' graph. But since we've attached the process to GDB, our code should have gone into a blocked state.

Going back to the GDB terminal, we can press c to continue the code until our breakpoint.

Once the breakpoint is reached, type finish to run the code until the parse_cypher function returns. You'll see that we exit to line 458 of cypher_analyze.c.

Type bt (short for backtrace) to check out the call stack so far.

From here we can see that we're inside the 'convert_cypher_to_subqery' function call.

Now run list to see where we are in the code.

It's around line 458. We can see that the result of the parse_cypher call is assigned to stmt.

For now, press c again in GDB to finish running the query.

4. Updating the AGE source code

Now let's make some changes to our code. We’ll add another stmt, called stmt2, just below the first. So open up the cypher_analyze.c source file and make the following changes on line 458:

Now go to the root of the AGE source code directory and run the make command again:

sudo make PG_CONFIG=/usr/local/pgsql/bin/pg_config install

Time to test if our changes have been updated.

We'll run the entire process again. Quit out of GDB with q, quit the psql terminal with \q, and start afresh.

Run psql postgres, LOAD and SET age like before.

Find the PID of the psql process and use it to start GDB. Set GDB search path to the source files.

Set a break point for the parse_cypher function, run the MATCH query inside the psql terminal once more. Go to the GDB terminal and run the code till the breakpoint using c.

The parse_cypher function should be called twice now (once for stmt and once for stmt2). After the first breakpoint, if you press c again, the code will break again upon the second call. You can also use the list command to see the updated code context. If you press c for the third and final time, the code will run till the query is completely executed.

I hope you were able to learn something new from this post! Thanks for reading :)

Set up Django on a Linux VM with Google Cloud Platform

moeed-k — Sat, 18 Feb 2023 14:05:42 +0000

As a senior in my BSCS program I've been working on my Final Year Project since the last several months. Early on, I decided to develop my project as a client-server web-application, and since the majority of the backend code was in Python, it was a pretty easy choice for me to choose Django as the web framework. Having the backend all be in one language streamlines a lot of things, but that isn't the point of this post.

So coming to the actual point: Up until now I've been hosting django on my local machine, and while that's worked fine until now, I'll eventually have to move on to using a real server.

To this end, I decided to experiment by setting up a basic Django VM using Google Cloud Platform, and this post is going to be a documentation of that process (for both my own future reference and for anyone else who might be interested in doing something similar).

1. Creating a Google Cloud Platform Account

For now, I went with the free trial handed out by GCP. While you do need to setup a credit/debit card, you won't be charged anything initially and will be given 300$ in credits for the first 90 days.

2. Enabling the Google Compute API

Next up we want to enable the Google Engine API. On the left-side panel hover over the Compute Engine section, and then select VM instances.
You'll be taken to a page that will ask you to enable the Compute API.
Enable the computation!
This might take a minute or two, but you'll be notified by the little bell icon on the top right once it's ready.

3. Creating the VM Instance

Now we're reading to create our VM. Once again, click on Compute Engine on your left, select VM instances, and you'll be taken to a page where you can configure your machine. Everything here is pretty straight forward, but there's a few things I want to focus on.

Here we want to be sure of two things:

The boot disk is selected as a Linux image (I'll be using Ubuntu).
The 'Allow HTTPS traffic' box is checked.

While we could always add in HTTPS traffic rules later, checking this box here simplifies the process from the get-go.

4. Configuring SSH Keys

Quick refresher: SSH is used to connect securely to a shell running on a remote machine. There's two Keys, a public one a private one. Since I'm working on Windows, I'll be using PuTTy to generate my keys.

PuTTy download link: https://www.puttygen.com/

Open up PuTTYgen after installation, choose an encryption algorithm (I'm going with RSA), decide on a Key Comment, and click generate. You should get something like this:

Save the private key somewhere on your local PC, and copy the public key generated in the box on top onto your clipboard.

Now open up the Google Cloud Console again. This time, select Metadata from the left hand-side panel. Go to the SSH keys tab, and add your public key here. All the VMs under the current project will inherit these keys.

Alright we're done with the setup! Now we just have to connect to our VM remotely. Now open up PuTTy (not PuTTygen). From here, enter the External IP of your VM instance (if you're not sure what the IP is you can look it up from the Google Cloud Console).

From the left hand-side panel in PuTTY, expand SSH, expand Auth, select credentials, and select your previously generated private key. Now click open. When asked to 'login as', choose what you wrote down as a Key Comment earlier.

We've successfully connected to our VM using SSH.

5. Modifying Firewall rules

Remember what I said about firewall rules earlier when creating the VM instance? Well we're mostly setup already and only have to add one extra rule for Django.

From Cloud Console, go to the VPC section and then select 'Firewall' from the left hand-side. Now click on the Create Firewall Rule button. Here are the settings I used:

In particular, I set the IP range to 0.0.0.0/0 (to allow all public IPs to connect to the VM), and I specified port 8000 as open (since that is what Django uses by default).

6. Setting up Django

Now all that's left to do is download and install django on our VM. On the terminal, run the following commands to install (if you're wondering about python, it came pre-installed with the Ubuntu image):

sudo apt install python-django sudo apt install python-django-common

Now we create a new django project called 'mysite' using the following command:

sudo django-admin startproject mysite

Just one more thing left to do before we run our server. We have to add the IP of our VM to the ALLOWED_HOST list in our django settings. So go into the mysite folder (inside the mysite root directory), open 'settings.py' using nano (command: sudo nano settings.py), and add the external IP of the VM (in single quotes) to the ALLOWED_HOSTS list.

Now go to the root of the 'mysite' directory, and start the server using the following command:

sudo python manage.py runserver 0.0.0.0:8000

Now let's confirm if our server is working. From your local machine's browser, enter the external IP and port of your VM instance to connect to the django server.

All done!