DEV Community: Olufisayo Bamidele

Notify Yourself After Completing a Long-Running Bash Process

Olufisayo Bamidele — Sat, 09 Dec 2023 01:16:03 +0000

Recently, I worked on projects that take a long time to build. I often stared at the screen, waiting for these to complete, which often resulted in me sleeping off at my desk. Later, I improvised sleeping while listening to some podcasts, but I usually lost valuable time since I couldn't tell when the long-running tasks were done. If only I could get notified if a bash process is done.

Say hello to the "say" command

The say command takes a string of text and reads it out loud. The example below says "brew upgrade done" when you're done upgrading your brew packages.

brew upgrade; say, "brew upgrade done."

Perhaps you often sleep off while listening to one of those monotonic dialogues between Lex Fridman and Elon; you could repeat the alert until you wake up with a little sprinkle of bash.

brew upgrade; for i in {1..1000}; do; say "brew upgrade done"; done;

That would say "brew upgrade done" 1000 times unless you manually stop it(with ctrl+c).

If you don't wake up after 1000 iterations, you should close your laptop for the weekend sleep. Just sleep 😆.

ℹ️ The say command comes preinstalled with Macos (I think), but its equivalent should be available on your Linux distro. For example, on my Ubuntu desktop installation, I have spd-say.

Alright, that's it. Have a great weekend, you all.

Future of Software Engineering: Is Maintainability Still Important?

Olufisayo Bamidele — Tue, 28 Nov 2023 03:11:22 +0000

I recently stumbled upon a YouTube talk by an AI salesman that got me thinking about the future of Software Engineering as a Profession. The speaker said something along the line of "maintainable code for humans". I mean, I have always known that but this time, it got me thinking – all those acronyms we toss around like confetti (DRY, WET, YAGNI, SOLID) are essentially for us. When it comes down to CPUs, all they see is bits and bytes.

Today's programming language, tomorrow's joke.

At the end, today's bleeding edge is tomorrow's legacy;

So, what if the programming language of tomorrow are not lines of code but Natural Language

Note that not long ago, any serious software had to be written in Assembly. Fast forward to today, compilers had become better at generating Assembly code than most programmers. I mean, I can't remember the last time I had to inspect the byte code generated by the V8 engine due to a pesky bug.

So, I ask again, if AI generated code gets better and it starts working 99.9% of the time, of what use is maintainability?

Kubernetes Services; Expose your app to the Internet

Olufisayo Bamidele — Thu, 23 Nov 2023 22:29:25 +0000

In the last articles in this series, we talked about the main pillars of Kubernetes: Cluster, control pane, nodes, workloads, pods, containers, and services. We dockerized a simple nodejs echo server and ran it on our machine inside a single-node Kubernetes cluster. In this article, we run that same workload in a Kubernetes cluster in the Cloud, but before that, let's talk about a topic I graced on quickly in the last article: Services.

Kubernetes Services.

While writing the last articles, I realized that the word "service" is overloaded in tech; It could mean a web backend application, a daemon running in an OS, a cloud offering, or even a module in a codebase. A "Service" is also a thing in Kubernetes that differs from all the meanings above. For clarity, we would be very specific when discussing "Services" in Kubernetes. We would call them k8s-Services

What is a K8S-Service?

A k8s-service is an abstraction that controls communication with a target group of Pods within a cluster and the exposure of those pods outside a Cluster. To be more specific, a k8s-service abstracts the networking of related pods.

Why Do We Need K8s-Services

In the last part of this series, we mentioned that a Pod gets assigned a unique IP address that we can use to communicate with the Pod inside the Cluster. Why can't we talk to pods directly using their IP addresses? Yes, we can, but we shouldn't. Pods, by nature, are short-lived. They come in and out of existence depending on many factors. Factors like

The Pod's health: If it maxes out its allocated memory and CPU, it will stop receiving traffic. Once Kubernetes detects this, it kills that Pod and replaces it with a new one.
Initial Configuration Of The Deployment That Started The Pod: Pods are usually started as deployment members. A deployment is a Kubernetes configuration that describes the desired state of a pod, including when we would like to have a replica. In a deployment, it is possible to specify the number of pods you want when a specific event occurs. For example, one can add a configuration like this: "If the total % of CPU usage goes up to 60%, create a new replica of this Pod to handle extra requests". In the configuration described above, replicas of a pod are expected to come in and out of existence depending on how intensively each Pod is being used.

Each new Pod replica gets a unique IP address, so using Pod's IP addresses is unreliable as a new replica receives a new IP Address. See the image below.

K8s-services solve the problem described above. To ensure that we can communicate with a group of Pods without manually keeping track of the Pods' IP addresses, we must create a k8s-service that sits between a request and the destination Pods. Like Pods, k8s-services get assigned unique IP addresses on creation, but unlike Pods, k8s-services are not ephemera because nothing capable of crashing is running inside them. They are there until deleted. They are just a data structure translated to a network routing configuration on the operating system.

With the setup in the image above, clients of Pod replicas do not need to remember the IP address of each Pod. That responsibility is given to k8s-service. Clients only need to remember the k8s-service's IP address.

Geek Bit ℹ️: The image above shows that k8s-services keep track of destination pod IP addresses in a table. While most of the diagram oversimplifies a Kubernetes cluster, this part is literal. A component of Kubernetes called Kubeproxy is responsible for translating the specification of your k8s-service to a network configuration. The configuration is usually implemented on your OS as a NAT iptable or as ipvs. Most cloud services providers like AWS and Google run kubeproxy in NAT IPTable mode. If you're running Kubernetes on Linux, you can view the translation of your k8s-service configuration as NAT table by running sudo iptables -t nat -L KUBE-SERVICES -n | column -t. As a Kubernetes user. Of course, you are usually not concerned about this implementation detail unless you're an administrator.

Types Of k8s-Service

ClusterIP K8S-Service

Whenever you create a k8s-service without specifying the type, the ClusterIP k8s-service is the type that Kubernetes would create. When you create a ClusterIP k8s-service, Kubernetes assigns a private static IP address to the k8s-service, which routes the request to matching pods. ClusterIPs only allow communication within the Cluster; Machines outside the Cluster cannot communicate with Pods through ClusterIP k8s service by default unless you allow this through something called an ingress. We will cover ingresses in another part of this series. See the illustration below.

Defining a ClusterIP in Kubernetes

This example edits the k8s-service example in the last article.
file_name: node-echo-service.yaml



apiVersion: v1
kind: Service
metadata:
  name: node-echo-service
spec:
  selector:
    app: node-echo # the label of the group of pods that this service maps to
  ports:
    - port: 80 
      targetPort: 5001

selector: app: node-echo instructs Kubernetes to put this ClusterIP k8s-service in front of any pod with the label app=node-echo. port:80 is the port the service binds to. targetPort:50001 is the port that our container is listening on; that is where the k8s-service will forward traffic to

To create the service, run kubectl apply -f node-echo-service. yaml. If the configuration does not contain a syntax error, you should get an output that says service/node-echo-service created on your terminal.

To confirm the creation of our service, type



kubectl get services

You should see the following output.

Geek Bit ℹ️: Pay attention attention, especially to the external IP column. Notice that the value is none. Also, notice that the ClusterIP column is a class A private IP address. Networking 101: Private IPs are only used in LAN. The Kubernetes Cluster's network is the LAN in this scenario. This ClusterIP k8s-service cannot receive internet traffic by default. The other two kinds of k8s services are built on top of ClusterIPs.

NodePort K8S-Service

NodePort k8s-Services builds on top of ClusterIP. In addition to getting a static private IP address, the NodePort k8s-service receives traffic from outside the Kubernetes cluster by opening up a Port in every Node. The traffic from these open ports can hit any node, and the service can forward requests to the available matching Pods. When configuring a NodePort k8s-service, we must provide three ports.

targetPort: The destination port of the matching pods. Usually, the port that your container is running on
port: The port that the k8s service binds to
nodePort: The port on each Node that accepts public traffic

See the the illustration below.

💡 I like to describe NodePort K8s-Service as a k8s-service whose public IP address is the address of every Node in the Kubernetes Cluster.

Although NodePort k8s-services allow clients from outside the Cluster to communicate with our Pods, they are not production-ready for the following reasons:

NodePort k8s-services only allow traffic from ports 30000 to 32767. Those are non-standard ports in a production environment. Browsers and HTTP clients look at port 80 by default and port 443 for HTTPS. Any other port would require users to be specific. Imagine having to remember the port of every website that you visit
NodePort k8s-service receives internet traffic through all the Nodes available in the Cluster. This is problematic in production because clients must keep track of all those IP addresses. At the very least, you need a static, permanent, public IP address associated with the K8S-service for your workload to be production-ready. You can achieve this through the creation of an Ingress(don't think about this for now) or using the next k8s-service type called LoadBalancer

Defining a NodePort K8s-service

To define a NodePort k8s-service, we need to add two new properties to the configuration in the ClusterIP section.

Under the spec property, add the property "type" whose value is NodePort, i.e., type: NodePort
Under the port object, add the property "nodePort", whose value is any port you choose, i.e., nodePort: 30000

file_name: node-echo-service.yaml



apiVersion: v1
kind: Service
metadata:
  name: node-echo-service
spec:
  type: NodePort # telling k8s that we are talking about NodePort
  selector:
    app: node-echo # the label of the group of pods that this service maps to
  ports:
    - port: 80 
      targetPort: 5001
       nodePort: 30000 # The port port that receives traffic from the internet

Note: If you skip the nodePort property, Kubernetes will automatically choose your value.

Submit the new configuration to Kubernetes by running kubectl apply-f node-echo-service.yaml. If your configuration contains no syntax error, you should get an output that says service/node-echo-service configured.

To see the result, run kubectl get services node-echo-service -o wide. Your result should look similar to the screenshot below.

Pay attention to the type and the port column. The type column now says "NodePort," the ports column maps the k8s-service's port 80 to port 30000 on our machine.

We can now communicate with our workload by running curl -d "amazing" 127.0.0.1:30000

GeekBit ℹ️: NodePort is not useless in production; it's just not unsuitable for most web applications. Assuming I run a compute-intensive workload(say, image processing) in Kubernetes, I have dedicated an entire Cluster to this workload. I want to balance incoming tasks across Nodes so that every Node in the cluster always has the same number of tasks running inside them. I'd go for a NodePort k8s-service and set the externalTrafficPolicy to Local, ensuring that traffic to a Node only fulfills a request inside that Node. Finally, I'd put a network load balancer in front of the k8s-service. Of course, don't worry about it if you don't understand everything. Keep following this series, and it'll eventually make sense.

LoadBalancer K8s-service

With the LoadBalancer K8s-Service type, Kubernetes assigns it a static public IP address. This is what we want in production for web servers 🤗. The IP address is then announced across the underlying network infrastructure.
See the illustration below.

⚠️ Note that Kubernetes doesn't come by default with a network Loadbalancer, so one would usually have to install a plugin such as metallb for load balancing but you don't need to worry about this since your cloud provider would have made this available to your Cluster by default

There are two other types of Kubernetes services which I am intentionally skipping in this part. As we dive deeper into Kubernetes networking in the future, I will talk about these in more detail.

Defining a LoadBalancer K8s-service

To define a k8s-service of type LoadBalancer, take the yaml config file from the NodePort section and

Change the type from NodePort to LoadBalancer, i.e. type: LoadBalancer
remove the nodePort property. The resulting yaml should look like so

file_name: node-echo-service.yaml



apiVersion: v1
kind: Service
metadata:
  name: node-echo-service
spec:
  type: LoadBalancer
  selector:
    app: node-echo # the label of the group of pods that this service maps to
  ports:
    - port: 80 
      targetPort: 5001

Apply the configuration by running kubectl apply -f node-echo-service.yaml. You should get the following output; service/node-echo-service configured

Running kubectl get services node-echo-service -o wide, you should get an output similar to the screenshot below.

Observe the external ip column. Now it says localhost; This is because I don't have a load balancer installed in the Cluster, but you would get a public IP address in the Cloud.
If we run curl -d "load balancers are amazing" localhost without specifying any port, we should get those exact words echoed back to us.

Deploying our Kubernetes Workload to the Cloud

From the first part of this series to this particular article, we have learned the very basics things we need to nail down to deploy stuff workloads to a Kubernetes Cluster in the Cloud. Now it's time to do the do. Let's take our workload to the Cloud.

I chose Google Cloud for this demo because I've had more experience with Kubernetes on GCP.

Step 1: Set up the projects we want to deploy

We will be using the project we used in the previous part.

Clone the repository and duplicate the folder "k8s-node-echo
Rename the duplicate folder with a name of your choice. I'm calling mine "k8s-node-echo-with-loadbalancer".
cd into the k8s-node-echo-with-loadbalancer

Step 2: Build The Project As a Docker Image

Create a Docker Hub account - Docker Hub, like GitHub for Docker Images. This is where we would push our docker image. Note: DockerHub is not the only place we can push our images, just as GitHub is not the only place to push our code. Docker hub is just one of the popular destinations for your open-source docker images. Take note of your username while signing up. It would be useful later on
Log in to your docker hub account on your docker desktop. Click on the Login icon on the docker hub UI, as shown in the screenshot below.
Back in your terminal, in our project folder, run the following command



 build -f Dockerfile . -t <yourdockerhubusername>/node-echo:v1 -t <yourdockerhubusername>/node-echo:latest

For example, for me, that would be



docker build -f Dockerfile  . -t ngfizzy/node-echo:v1 -t ngfizzy/node-echo:latest

🚨 If you're working on an M1 and above Macbook, remember to add the --platform linux/amd flag, i.e docker build --platform linux/amd -f Dockerfile. -t ngfizzy/node-echo:v1 -t. This is because arm architecture(which m1 chips are based on) is not the default chip most cloud service providers use.

The -t flag specifies the name of your image. The image name contains three parts.
ngfizzy: your docker hub username
node-echo: our application name
v1: the version of our application

The second -t option only aliases v1 as the latest version.

Running docker images | grep "REPOSITORY\|ngfizzy" should show you more information about the image you just built, like the screenshot below

Step 4: Push the image to Docker Hub

Run



docker push ngfizzy/node-echo:v1

If everything works out, your output should look similar to mine in the screenshot below.

Run the same command for the ngfizzy/node-echo:latest. If you visit your docker hub account, you should be able to see those images there now. Here's mine

Step 5: Update your `node-echo-deployment.yaml` file1.

Clear the excessive comments I used for explaining the file in the last part of this article
Update the image property ngfizzy/node-echo:latest
Change the imagePullPolicy property's value to Always

The resulting configuration should look like this.



apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-echo-deployment 
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: node-echo
  template:
    metadata:
      labels:
        app: node-echo
    spec:
      containers:
        - name: node-echo
          image: ngfizzy/node-echo:latest
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 1 
              memory: 256Mi
          ports:
            - name: node-echo-port 
              containerPort: 5001
          livenessProbe:
            httpGet:
              path: /
              port: node-echo-port 
          readinessProbe:
            httpGet:
              path: /
              port: node-echo-port
          startupProbe: # configuration for endpoints
            httpGet:
              path: / 
              port: node-echo-port

Only lines 16 - 18 changed in the configuration above.

Step 6: Update the node-echo-service.yaml file

Replace the content of node-echo-service.yaml with the LoadBalancer configuration in the load balancer section of this article. Here's the configuration to save you from scrolling



apiVersion: v1
kind: Service
metadata:
  name: node-echo-service
spec:
  type: LoadBalancer
  selector:
    app: node-echo # the label of the group of pods that this service maps to
  ports:
    - port: 80 
      targetPort: 5001

Step 7: Create a GKE Cluster On GCP

Create a GCP account if you don't already have one
Create a GCP Project if you don't have one previously
On the home page, click on the Create GKE Cluster button as shown in the image below

⚠️ If you have previously enabled if you have not previously enabled Cloud Compute and GKE API, you'd be prompted to do so by following the prompts. When you're done, return to the home page and click the Create GKE Cluster button again.

You'd be presented with the following page settings page after clicking.

For demo purposes, we would accept all the default settings and click the submit button at the bottom of the screen.

That should redirect you to this, as seen in the screenshot below.

It takes a couple of minutes for the Cluster to be created.

Step 8: Install GCloud CLI if you've not done that already

Follow the instructions here https://cloud.google.com/sdk/docs/install

Step 9: Log in to Google Cloud on your CLI and your gcloud project

gcloud auth login
gcloud config set project <your-project-id>

Step 9: Connect to your GKE Cluster on your local machine

On the GKE Cluster page, click on the connect button. Follow the numbers in the screenshot below for Navigation.

Click on the pop-up box after clicking connect, then click the copy icon to copy the connection command to your clipboard. Go back to your CLI and paste the command. You should get the following output.



Fetching cluster endpoint and auth data.
kubeconfig entry generated for <your cluster name>.

Confirm that you are now connected by running kubectl config get-contexts, you should see at least one entry in the table. The name of one of them should start with gke_*

Deploy To GKE Cluster

Now that we are connected to the GKE cluster

Apply our deployment by running kubectl apply -f node-echo-deployment.yaml. You might get a warning saying, Warning: autopilot-default-resources-mutator:Autopilot updated Deployment... Don't worry about this
Apply your k8s-service config by running kubectl apply -f node-echo-service.yaml
Confirm the deployment by running kubectl get all. You should see an output similar to the screenshot below.

Test your deployment

Run kubectl get services node-echo-service
Copy the IP address under the External IP column and send a post request to it like the one below ```bash

curl -d "hello world" 34.123.423.124

The server would echo "Hello world" back to you


## Summary
In this article, we took a more detailed look at Kubernetes services; we then used our knowledge to deploy a simple server to the Internet. We are just scratching the surface of Kubernetes. In this series, I aim to gradually reveal containers, and Kubernetes features until we can paint a complete picture of how everything works from end to end.

In the next article, we will take a full circle look deeper at containers, and explore how they do what they do.

Infer Function Return Type In Typescript

Olufisayo Bamidele — Fri, 10 Nov 2023 12:34:14 +0000

Scenario

You are using a function called doFunkyStuff from a library called funky-lib. Do doFunkyStuff returns an interface FunkyStuff, but funky-lib does not export FunkyStuff

The lib code

// node_modules/funcky-lib/index.ts

interface FunkyStuff {
    message: string,
    funkinessLevel: number
}

export function doFunkyStuff(): FunkyStuff {
   console.log('doing funky stuff')

   return { 
      message: 'did some funky stuff',
      funkinessLevel: Math.random(),
   }
}

Our Code

// index.ts
import { doFunkyStuff } from 'funky-stuff'

// eslint yells no-explicit-any
function repeatFunkyStuff(fStuff: any) {
  console.log('repeating funky stuff', fStuff)

  return {...fStuff}
}

repeatFunkyStuff(doFunkyStuff)

Depending on your eslint-config, eslint would either yell "no-implicit-any" or "no-explicit-any"

Bad Solution

Redefine FunkyStuff

function repeatFunkyStuff(fStuff: { message: string; funkinessLevel: number})

The solution above is bad because it becomes unreadable if FunkyStuff has more than a handful of properties.

Also, we must update our code each time something changes in FunkyStuff.

Good solution

Use the ReturnType utility type.

// our code in index.ts
...
type AppFunkyStuff = ReturnType<typeof doFunkyStuff>

function repeatFunkyStuff(fStuff: AppFunkyStuff) {
   ...
}
...

Bonus

You can combine ReturnType<T> with Awaited<T> util type for functions that return promises.

Assuming doFunkyStuff returns Promise, then AppFunkyStuff would be defined like this.

type AppFunkyStuff = Awaited<ReturnType<typeof doFunkyStuff>>

⚠️ Awaited util became only available in typescript 4.5. Achieving the same effect can be a little tricky. Previous versions

Summary

This tip is just the tip of the iceberg regarding what typescript offers.

Docker and Kubernetes From Localhost To Production: Kubernetes - Container Orchestrators, The What, Why and How

Olufisayo Bamidele — Wed, 08 Nov 2023 14:08:23 +0000

💡 This is a very long article. It is so because Kubernetes pulls many computing concepts along before one can do anything with it. I broke those concepts down as much as possible in this article, but because of the length of this article, I'd encourage the readers to read it three times.
The first read should be a quick scan to get a general idea of what the article is about. Come back to it at some later point in time for a second read, this time, a slower, more careful read. Then, come back again at a later point in time for a more engaged form of learning. This time, you would take notes, Google unclear terms, ask questions in the comment section, and try out the examples.

What is Kubernetes

Kubernetes is a container orchestrator. Just what exactly does that mean?

In the previous article, we learned about containers and their role in the deployment/management of web applications.

In modern software engineering, we often find more than one web service communicating with each other over a network to serve a single purpose. For example, an e-commerce website would have a recommendation-service that recommends products to users based on their preferences, a cart-service that takes care of items in the user's cart, a payment-gateway-service that integrates multiple payment services, etc. Services that work in coordination as described above are called Distributed Systems, but you might be more familiar with the buzzword Microservices.

Microservices are distributed systems that contain highly specialized web services. They are also usually deployed in containers due to their lightweight and scalable nature. Learn why containers are lightweight in the previous article

When deploying a group of services, a container orchestrator becomes critical. An orchestrator ensures the deployed services adhere to the configuration specified during the deployment. A developer(or a platform engineer) would give a configuration file that describes the desired state of the distributed system. The configuration usually looks something like this in human language.

Question	Answer
What are you trying to configure?	A deployment
How Can I Find The Executable For the App	Look for a docker image called `auth-service:latest` on this machine and run it
How many CPUs do you want to assign to this app	1
How much memory do you want to assign to the app	1Gi

The platform engineer /developer would then submit that configuration to the container orchestrator. Once received, the orchestrator would constantly look at the configuration and compare it to the current state of your app(s). If there is any variation between the comparisons, the orchestrator will automatically adjust the state of your services to match the developer's desired state.

Let's learn those concepts by associating them with concepts we already know

In the previous article in this series, I drew a parallel comparison between a program and a container. As a reminder, a program that is running is called a process, and an image that is running is a container. Let's continue with that analogy.

Processes Kubernetes

At any moment, there are hundreds of processes running in your system. To get a snapshot of all the processes loaded to your computer memory right now, run ps -A At any point in a distributed system, there is more than one container running in your deployment.

Most modern laptop CPUs have between 2 to 8 cores, which means they can only execute instructions from 2 to 8 processes at once. To cater to all the processes in memory, a component of the Operating System kernel called the Scheduler ensures that all processes take turns executing a bit of their instructions in fractions of a second. The schedulers(usually more than one) decide which process gets the CPU's attention at any time. A cloud platform typically has millions of machines that they configure to act like one or multiple supercomputers. Orchestrators work like OS schedulers in the cloud, assigning your services to the suitable machine while making sure they are using the right amount of memory, CPU, and disk space.

Processes	Kubernetes
At any moment, there are hundreds of processes running in your system. To get a snapshot of all the processes loaded to your computer memory right now, run `ps -A`	At any point in a distributed system, there is more than one container running in your deployment.
Most modern laptop CPUs have between 2 to 8 cores, which means they can only execute instructions from 2 to 8 processes at once. To cater to all the processes in memory, a component of the Operating System kernel called the Scheduler ensures that all processes take turns executing a bit of their instructions in fractions of a second. The schedulers(usually more than one) decide which process gets the CPU's attention at any time.	A cloud platform typically has millions of machines that they configure to act like one or multiple supercomputers. Orchestrators work like OS schedulers in the cloud, assigning your services to the suitable machine while making sure they are using the right amount of memory, CPU, and disk space.

Essential Kubernetes Components/Terminologies

Workload

A workload is an application(usually a web service) running in Kubernetes. From this point onward, we refer to applications running inside Kubernetes as a workload.

Pod

Techies love to wrap things, and Kubernetes is no exception. Kubernetes developers, at some point, decided that containers were not enough; they needed to wrap them within something else called Pods. Whenever you deploy a workload on Kubernetes, Kubernetes puts that workload(i.e., the container your application is running in) inside a pod. So, in Kubernetes, the smallest unit of a deployment is a pod, not a container.

Let's associate pods with what we already know

An application running on an operating system is called a process. When you run that program in a Docker environment, it becomes a container; When you run your container directly in Kubernetes, it becomes a Pod.

⚠️It is possible to run multiple containers inside a single pod just as it is possible to run more than one application inside a single container. However, for 99% of use cases, you should only have one application running in a container and one container running in a pod. We will discuss the exceptions as we advance in this series.

Like many techies, you're probably asking why. Isn't container isolation enough? Are pods not redundant? No, Pods are not redundant. I'll answer this entirely in the Whys section of this article.

Node

Node is Kubernetes' lingo for a physical or virtual machine. Pods run in nodes(Obviously). Before now, I said workload runs in Kubernetes, but the right way to phrase that is "workloads run in nodes." From now on, when talking about computers/machines/virtual machines in the context of Kubernetes, we will refer to them as nodes.

Cluster

A cluster is a group of Nodes available for your workload(s). A cluster contains one master node and one or more worker nodes. The master node in Kubernetes lingo is called the Control Pane, while the worker nodes are simply called nodes.

The Control Pane is the node responsible for assigning workloads to other Nodes. It is the component of Kubernetes that fits more into the OS scheduler analogy. But wait! Does this not suggest that Kubernetes requires at least two machines to work correctly? How are people now able to run them on their laptops? In production, yes, Kubernetes requires at least one Control Pane Node and one Worker Node. That said, for experimental purposes and local development, it is possible to set up a single-node Kubernetes cluster where a single node serves as both the Control Pane and the Worker.

Now, let's learn that concept from bottom to top.
A cluster contains nodes -> a node runs one or more pods -> a pod runs containers(most time, one container)

The Whys

Are Containers not Enough Isolation? Why Do We Need Pods?

As redundant as they may look, they serve a purpose. A pod controls how a container can be discovered within a node. One problem it solves is the problem of clashing port bindings on a node. If you bind an internal port of a container to a port on a host machine and you try to bind another container port to that same port, you'll get an error. Pods solve this problem because each pod in a cluster is assigned unique IP addresses at creation time. Pods can communicate using those IP addresses through their node's port.

But why do I need Kubernetes?

That is a very valid question. After all, there are simpler alternatives like Google Cloud Run, Heroku, AWS Lambda, and many more. I'll answer with a quote from Eskil Steenberg;

In the beginning, you always want results; In the end, all you've always wanted is control.".

When building a product, you want to get to the market quickly and beat your competition. Beyond that stage, things can get very expensive with all these serverless tools. Even Amazon had to move one of its applications off serverless recently.

Beyond scheduling your pods, Kubernetes abstracts underlying infrastructure, exposing them to you as just resources (resources such as memory, disk, and CPUs, giving you an illusion that all your web services are running on a humongous machine and you can assign resources to each application by simply specifying the amount of memory, CPU and disk space it needs; This is the kind of control required by scaleups and enterprises.

But I Only Have A Portfolio Website. Why should I use Kubernetes?

Please don't unless you enjoy pain 🙊.

Why is it so complicated?

Well, it is, and it's not. If you asked a regular developer, it probably is. If you ask a system admin who's had to provision virtual machines manually in the past, supervise port mappings, sit and watch system logs to figure out when something is wrong with one of the numerous applications running on his servers, and manually restart applications on VMS when they crash, to me, Kubernetes is not that complicated.

Don't get me wrong, I'm not insinuating that Kubernetes is easy, but it's not rocket science; it's learnable. Programming was complex, but you learned it; Kubernetes is no different.

The How

How to Set up Kubernetes On Your Computer System

I'm currently on a Macbook; I'll speak quickly about setup on macOS. If you need step-by-step instructions about how to set it up on Windows and Linux, drop a comment, and I'm sure either I or other members of the community would help out.

Step 1: Download and install Docker Desktop for Mac. Follow the instructions here
Step 2: Once you start the docker desktop, go to settings. Under settings, click on the Kubernetes tab. In this tab, you'll see a check box with a label that says "Enable Kubernetes". Click on the check box and then the apply and restart button.
Step 3: Confirm that you have Kubernetes by running with the command, kubectl cluster-info. You should see an output that looks like the screenshot below.

Let's get a simple application to work with

In the previous article, we ran a simple echo server written in nodejs inside a container. We would continue from where that post stopped.

Step 1: Clone the repository, cd into the docker-and-k8s-from-localhost-to-prod directory. Duplicate the folder called node-echo and rename the duplicate to k8s-node-echo and cd into it.

Step 2: Build the application as a docker image

docker build -f Dockerfile . -t  node-echo

Run the image in a Kubernetes node

Step 1: Create a Kubernetes Deployment Config

Kubernetes accepts configuration in YAML or JSON format. I'll choose YAML because it's the most popular choice in the community. Besides, it allows me to add comments to explain my configuration, so it is a favorable choice for tutorials like this.

Create a file called node-echo-deployment.yaml and add the following config to it. It's preferrable if you type it out yourself. I have added an explanation to each line of the config

# The kubernetes api version; because kubernetes' api is constantly revised
apiVersion: apps/v1
# The Kubernetes object that we are defining with this configuration. In this case, it's a Deployment
# A deployment is a Kubernetes _Object_ that stores desired state of pods
kind: Deployment
metadata:
  # We are just giving a human-readable name to this deployment here
  name: node-echo-deployment 
spec:
  # We are telling Kubernetes that we only want one instance of our workload to run simultaneously. If this instance crashes for any reason, Kubernetes will create a new instance of our 
  replicas: 1 
  # The next 7 lines assign metadata to our node to easily query them in the cluster. More on these later in the series
  selector:
    matchLabels:
      app: node-echo
  template:
    metadata:
      labels:
        app: node-echo
    # From here downward, we are describing how to run our web app to Kubernetes 
    spec:
      # We are telling Kubernetes to find the docker image "node-echo:latest" and
      # run it as a container. The container's name should also be called node-echo
      containers:
        - name: node-echo
          image: node-echo:latest
          # Next, we are telling Kubernetes to look for node-echo:latest image on our machine first. if
          # If it isn't present on our local machine, then Kubernetes will go look for it on the
          # internet
          imagePullPolicy: IfNotPresent 
          # The next 4 lines define the limit of the resources that Kubernetes should make available in our app 
          resources:
            limits:
              cpu: 1 
              memory: 256Mi
          # We are telling Kubernetes the ports that our workload will be using
          ports:
            # We are giving the port 5001 a name. Similar to declaring a variable
            - name: node-echo-port 
              containerPort: 5001
          livenessProbe:
            httpGet:
              path: /
              port: node-echo-port 
              # In the 4 lines above, told Kubernetes to check localhost:5001/ in the pod node-echo
              # the application is still running properly. If the endpoints return any status code > 299
              # kubernetes restarts the application
          readinessProbe:
            httpGet:
              path: /
              port: node-echo-port
              # The 4 lines above defines the endpoint that Kubernetes would check to determine that
              # the pod is ready to receive trafic 
          startupProbe: # configuration for endpoints
            httpGet:
              path: / 
              port: node-echo-port
              # The 4 lines above are uses by Kubernetes to determine if our application has started

Comparing the table in the opening section of this article might clarify this even further.

Step 2: Submit That Config To Kubernetes

Run kubectl apply -f node-echo-deployment.yaml where the -f flag specifies the file containing your config. If the file has no syntax error, you should get a message that says "deployment.apps/node-echo-deployment created"

Step 3: Confirm The Created Deployment

To check that your deployments are running, run kubectl get deployments. The output should look similar to the screenshot below.

You can also see your running pods by running the following command: kubectl get pods. Your output should also look like the screenshot below.

Step 4: Test our running workload

We know that our application is running on the localhost:5001, so if we run

curl -d 'hello' localhost:5001/

Our server should respond with "Hello", right? No. This is because a Kubernetes cluster's network is Isolated from the outside world by default. Going inside the pod, we can hit that endpoint. What a Bummer! I know, right 🙂? We can't tell users to always enter a pod before they can use our service.

We can fix this by creating another Kubernetes object called a Service. Oh no. 🤦‍♂️ Not another concept, right? Hold on a minute. We are almost done. This is the last concept you have to learn in this article.

Step 5: Create a Kubernetes Service

A Kubernetes service defines networking between pods and how pod networks are exposed to the internet. There is more to it, but for now, in development, we need the following configuration.

Create a node-echo-service-deployment.yaml file and add the following config.

# specifying which version of Kubernetes API we are using
apiVersion: v1
# Telling Kubernetes that we are configuring a service. 
kind: Service
# assigning some metadata to the services
metadata:
  # We are assigning a name to the service here. We are calling it
  # node-echo-service
  name: node-echo-service
# From here below, we are specifying the configuration of the Service
spec:
  # There are different types of services.
  # For demo purposes, we stick to NodePort. Node+Port or Machine+Port.
  # With this config, we are telling Kubernetes that we would like to bind the Pods port to a Port on the Node running our Pod
  type: NodePort
  selector:
    # We are specifying the app that we are configuring this service for
    # If you go back to the deployment.yaml file, you'd remember that we defined
    # this app metadata over there. It has proved to be useful when defining other
    # Objects for our deployment
    app: node-echo
  # Define port mapping
  ports:
    # The port that this service would listen on
    - port: 80 
      # The Pod's port to which the Kubernetes service would forward requests.
      # This is usually the port that your workload binds to
      targetPort: 5001
      # The port on your Node that the service binds to
      nodePort: 30001

Step 6: Submit the Kubernetes service config

Then run kubectl apply -f node-echo-service.yaml. If your configuration is fine, you should get the following message: service/node-echo-service configured

Now we can communicate with our service on localhost:30001/

curl -d "Hurray!! Now I can talk to my service in Kubernetes" localhost:30001/

To see all the code base and the configuration files in one, place, check out this repository

Conclusion

In this article, we learned the main pillars of Kubernetes: Cluster -> Node -> Services -> Deployment -> Pods -> Containers. We discussed the relevance of Kubernetes and why it can benefit a developer to learn it.

It is OK if everything stays the same. Kubernetes is best learned by trying stuff in it, running into issues, and fixing them.

In the next part of this series, we will go in-depth on services and deploy our node-echo to a cloud provider.

If you learned something, leave a
reaction, and do not hesitate to correct me if I have mixed up certain explanations.

See you on the next one.

Docker and Kubernetes From Localhost To Production: Containers; The what, why, and How

Olufisayo Bamidele — Wed, 01 Nov 2023 06:06:26 +0000

Warnings:
1. Some concepts in this article are explained like it's meant for 5-year-olds. If you're already an advanced user, this blog post is not for you.
2. If you want to see some code, scroll to the How subheading of this blog post. If everything there seems foreign to you or you don't get the point, you can come back to the very beginning.

For many people, Docker and Kubernetes are the things they learn halfway and drop, never to touch again after finding alternatives. Some, after learning it, never get to use it on their jobs because they have DevOps/platform teams, but once in a while, they pop up again. They're required to learn/relearn it. Usually, they have to relearn it from scratch.

Why Do Docker and Kubernetes Never Stick For Many People?

Why is Docker and Kubernetes(especially Kubernetes) tricky for many to be proficient in? Why is it so easy to forget everything once you stop using them?

As humans, we learn best through association. It is easy to remember new things when you can associate that learning with a simpler but related concept. Your entire knowledge base is a tree, and new learnings need to be attached to existing branches of that tree. The reason why people forget Docker and Kubernetes(k8s) quickly is that they encapsulate the significant pillars of computer systems, i.e., Operating systems, programs(running in userland), and computer networking. My aim in this series is not to graze over these fundamental topics but to go in-depth as necessary. Understanding those basic concepts would increase your chances of not having to relearn everything from scratch each time you return to them.

The Whats

What is

A Container

A container is a basic unit of deployment. That's vague, so let's rephrase it. A container is your program coupled with a virtualized operating system alongside every other dependency needed to run it. The virtualized operating system part of that definition is essential.

A Container Image:

A container image is "an executable", runnable by a container Engine.

A Container Engine/Container Runtime:

A container engine is a program that knows how to build and execute a container image.

Docker

Docker is a container technology company. The Container engine they created is called the Docker Engine; a container image built with docker-engine is called Docker Image. An executed Docker image is called a Docker Container.

A DockerFile

A docker file contains information on what a docker image should have.

Let's Relearn All The Concepts Above By Associating It With What We Already Know From Bottom Back To The Top

I assume anyone reading this article has written a program at some point, so let's compare the process of creating a program to creating a container.

To write a C program, you must first write some instructions in a .c file.
- A docker file is the source code that contains instructions about what a docker image should have, which is usually a description of the OS your program is built to run on, the dependencies of your program, your program itself, and an instruction about how to start it.
To transform your C source file into something your machine can understand, you must build the .c file using a c compiler.
- To transform your DockerFile to a docker image, you must "compile" it with the "docker image builder." The builder comes with the docker engine, and you have access to it through docker-cli
To run the C executable, you can either open a command line interface and call the executable directly or, if you have access to a GUI, you will double-click on the executable.
- To run the docker image, you run it through docker-cli.

The Why

I already know how to run a program on a computer; why do I also need to run it in a container?

Yes, you could rent a physical machine and run your web app on it. Well, these days, cloud companies will only give you that deal if you're Facebook or Netflix. If you're a regular Joe like me who wants to run a portfolio website, you won't get a contract to run it on a physical server.

Yes, I know this already, but these businesses can still run multiple programs on a single machine.

Yes, they can. There is already a business model for this. It's called Shared Hosting. In this hosting model, the provider runs multiple customer programs on a single machine. This model is more cost-effective for both you(the customer) and the service provider but not suitable for fast-growing startups. One downside to this approach is that websites with heavy traffic might use the shared server resources (CPU and memory), leaving your website with little to no resources.

Another issue that arises in shared hosting is security. A customer running a malicious program on the same server as yours can also infect your website. Of course, these businesses take measures against worse-case scenarios like this, but it requires the expertise of specialized system admins.

Programs running in containers, on the other hand, by default don't have this problem because a container is a completely isolated environment, and resource limits can be easily assigned to the container as part of their startup instructions.

Talking about Isolation and resource allocation, isn't that achievable Through Virtual Machines? Why Do I Still Need Containers?

Yes, there is also already a business model for that. Today, you can rent a virtual machine on any cloud platform. They give these things different marketing names, like EC2 on AWS or Cloud Compute Engine on Google Cloud. Irrespective of what cloud platforms call it, you get a virtual machine running some open-source or proprietary distribution of the Linux operating system.

So, if virtual machines solve the problem with shared hosting, why do you still need containers?

Well, businesses are always trying to do two things: make more money and spend less money. Virtual machines virtualize the hardware, so for every virtual machine running on a physical machine, at the very minimum, a virtual CPU, memory, hard drive, and a virtual ROM containing a virtual bootloader would be created. The virtual machine also boots up a full-fledged operating system just to run a Hello World program. Also, for every client hosting software on a physical machine, a new virtual machine would have to be provisioned for them.

All of these leave a huge resource footprint on the host machine. So, in 2006, google added more features to an existing Linux OS kernel feature called namespaces.
Additionally, they implemented something called cgroups. Those two features enabled system administrators to run programs in isolation without hardware virtualization. That work serves as the foundation for container technologies. I would dive deeper into cgroups and namespaces later in this series.

So, with containers, cloud service providers don't need to provision a virtual machine for each customer. With the help of container technologies taking advantage of namespaces and cgroups, you can run your programs in isolated environments with a much lower resource footprint. Additionally, since containers are just programs with limited access to system resources, they have a faster startup time than virtual machines. This kind of efficiency translates to substantial cost savings on hardware for cloud service providers at the same time, allows them to provide cheaper offerings to you, the customer.

Follow the money, my friends; it will lead you to the answers to most modern tech questions. 🙊

Do Container Technologies Solve the: "But it works on my machine problem"

From my experience, yes, it could, but it takes some proficiency to containerize your application such that it works the same way everywhere outside your machine. In short, dockerizing your application does not automatically solve that problem.

The How

Talk is cheap; show me some code.
Prerequisite: Make sure you have docker desktop installed.

Step 1: Let's create a simple echo server in nodejs

// index.js
const http = require('http')

http.createServer((req, res) => {
    const {method} = req;
    if(method.toLowerCase() === 'get') {
        res.statusCode = 200
        return res. end()
    }

    const chunks = []
    req.on('data', (chunk) => {
        chunks.push(chunk)
    })
    .on('end', () => {
        res.statusCode = 200
        return res.end(Buffer.concat(chunks).toString());
    })
})
.listen(5001, () => {
    console.log('started server at localhost:5001')
});

To test that our echo server works, run node index.js and then in another terminal, run curl -d 'hello world" loalhost:5001. The server should echo back "hello world" to you.

Step 2: Write the requirements of your docker image in a Dockerfile

  1 FROM node:20.9.0-alpine-3.18
  2
  3 WORKDIR /web
  4 COPY ./index.js ./index.js
  5
  6 ENTRYPOINT ["node," "./index.js"]

Starting from line one

We instructed Docker to pull nodejs 20.9 from the docker hub. If "docker hub" sounds foreign, consider it as GitHub for docker images. That nodejs image is the base(or foundation) to build our application image. A Dockerfile always starts with the FROM instruction.
Next, we define the default working directory of the container. The working directory is the default directory when your container starts. This is similar to the $HOME directory in the Linux operating system or the ~ directory on Mac.
Next, we copy our source code into a docker image. And finally, on line six, we tell Docker the command to run when our application container starts up.

Step 3: Build the image using the DockerFile we just created; run the following command

docker build -f Dockerfile . -t node-echo`

The -f argument specifies where the Dockefile is located, in this case the current directory.
The . specifies the execution context of the build process, i.e., the folder where the source code(or executable) of your application lives. The command above assumes you're running the "docker build" in the same folder as your source code.
The -t argument assigns a name to the resulting image.
To see information about the resulting image, run docker images
The output should be similar to what is in the screenshot below

Step 4: Create a running container with the image

docker run -t -i -p 5001:5001 node-echo

Breaking that command down,

The -t flag allocates a pseudo-tty to the docker container. Assigning a pseudo-tty is a "Linuxy" way of saying, "We allow our program, in this case our container, to accept input from the keyboard.
The -i flag instructs docker that we want to keep the container's stdin open. For beginners, stdin is where your keyboard input goes to.
The -p 5001:5001 publishes port 5001 on the container to port 5001 to the host's operating system. That's also a fancy way of saying that we want users to be able to send HTTP requests to port 5001 in the container via port 5001 on the host operating system. This part is essential because, by default, everything running a container, including the network configurations of a container, is isolated from that of the host machine. To expose anything inside the container to the host machine, we need to let the docker engine know we intend to expose it.
node-echo is the docker image we are executing After running that command, our echo server should start up quickly. Sending a request to our server like so, curl -d "hello" localhost:5001 should send the word hello back to us.

Edit 1/10/2023

You can play with the full sourcecode here

Kubernetes

Kubernetes is a container orchestration tool for automating the deployment and management of containers. Usually, you don't deploy only one container in the modern web; you deploy tens to hundreds of them, and they have to work in a coordinated manner. This is the job of Kubernetes. I know those words do not hold significant meanings if you're a beginner, but this blog post is too long, so I'll pick up from here in the next part of this series.

Summary

In this first part of the series, we learned what a container is, why we need a container, the problems container technologies solve, and how you can create a simple nodejs server container using docker. We also introduced Kubernetes. We have just scratched the surface of this subject, so stay tuned for more parts.

You Don't Need Axios

Olufisayo Bamidele — Tue, 24 Oct 2023 01:38:15 +0000

You don't need Axios, well, for most use cases. The web is maturing, and the days of Axios are ending.

Why we needed Axios

For newcomers, not long ago, we needed to use XMLHttpRequest to fetch data from servers. To fetch data from https://jsonplaceholder.typicode.com/todos/, we would have needed the following lines of code at the very minimum.

const req = new XMLHttpRequest();
req.addEventListener("load", function() {
  console.log(JSON.parse(this.responseText))
})

req.open('GET', 'https://jsonplaceholder.typicode.com/todos/')
req.send()

We needed an elegant abstraction over XMLHttpRequest, and axios filled that space perfectly, especially with the introduction of Promise in ES6.

With axios, we can achieve the same thing with just one line of code

    const data = (await axios.get('https://jsonplaceholder.typicode.com/todos/')).data

But time has passed, the web has matured, and now, we have fetch()

Use fetch() Instead

With fetch, you get most of the benefits of Axios with zero external dependencies in your application. I am a proponent of minimal dependencies. If the platform I'm on provides a functionality, and it's relatively easy to implement, I'll explore it first. This is even more important with JavaScript runtimes, as every abstraction comes at a cost.

GET Request Using Fetch

    const data = await fetch('https://jsonplaceholder.typicode.com/todos/')).then(res => res.json())

POST Request Using Fetch

The fetch API takes in a second parameter, which allows you to specify other options, including your HTTP method, headers, and body. The example below shows how to make a post request using fetch API. This example also applies to other HTTP methods that mutate the server;

    const data = await  fetch('https://jsonplaceholder.typicode.com/todos/', { 
  method: 'POST', 
  body: JSON.stringify({title: 'buy milk'})
}).then(res => res.json())

Uploading Files

Assuming you have the following HTML file

<!Docktype html>
<html lang=en>
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width initial-scale=1.0">
        <meta http-eqiv="X-UA-Compatibly" content="ie=edge">
        <title>File Upload</title>
    </head>
    <body>
        <form id="fileform" enctype="multipart/form-data">
            <input name="upload"  type="file" placeholder="upload file" />
            <button type="submit">submit</button>
        </form>
        <script src="./index.js"></script>
    </body>
</html>

then your fetch call would look like this:

const form = document.getElementById("fileform")

form.addEventListener("submit", (event) => {
    event.preventDefault();
    const fileInput = document.getElementsByName('upload')[0]

    const fd = new FormData()
    fd.append('upload', fileInput.files[0]);

    fetch('http://localhost:5001/upload', { method: 'POST', body: fd}).then(res => res.json().then(body => console.log('result', body)))
})

Pro Tip

Always wrap your fetch function in with another function; This gives you leeway for abstracting HTTP call patterns emerging as your application grows. See the code snippet below.

 const originalFetch = window.fetch
window.fetch = async (...args) => {
    // intercept request
    const res = originalFetch(...args)
    // intercept response e.g
    if(res.ok) {
        return res.json()
    } else
        //Do whatever you want to handle the error
    }
}

The Whataboutisms

What about Request and Response Interceptors

We've talked about that briefly in the previous subheading

    const interceptRequest = (...args) => {
        // perform some magic
        return args
    }

    const interceptResponse = (res) => {
        // perform some magic
        return res;
    }

 const originalFetch = window.fetch
window.fetch = async (...args) => {
    const modifiedArgs = interceptRequest(...args)
    const res = await orginalFetch(modifiedArgs)
    return interceptResponse(res)
}

Edit 24 Oct 2023 More On Intercepting Requests

Due to questions from colleagues, I realized I needed to address this a little more.

To intercept means to stop something from proceeding. Interceptors in HTTP libraries(you could call them middleware), stop requests and modify the request configuration using your provided function. For response, interceptors take raw server responses and modify them based on your logic before returning them to the library consumers.

These can be implemented as simple functions that get called before and after your call to fetch

Axios and other JS libraries don't do magic with your interceptors. They only call the functions you give to them and call them for you

What About Timeouts

Timeouts should primarily be the business of your backend. It would help if you also had a backend that integrates the third-party platforms, but should you choose to incorporate an unreliable backend with your frontend, you can still implement timeouts with fetch API so

// timeout request after 5 seconds
const res = await fetch('https://jsonplaceholder.typicode.com/todos/', { abort: AbortSignal.timeout(5000) })

What About Javascript In The Backend, Aka Nodejs/Deno/Bun

The three significant runtimes now implement fetch, and if you must install any HTTP client for your javascript backend, install undici. Undici is an HTTP client built from the ground for nodejs, and the nodejs organization maintains it. Undici also implements fetch API

What about a progress update for large file uploads

Use XMLHttpRequest ;). Unfortunately, fetch() has yet to include this feature.

Conclusion

Regarding dependencies, I follow the YAGNI principle in extreme programming. There is no need to bloat your application with thousands of dependencies. The web is maturing, and like JQuery, axios' time is gradually ending. With current developments, Axios and other popular HTTP client libraries are no longer required to have good developer experience.

SQL JOINS; Identifying The Left And Right Table

Olufisayo Bamidele — Sun, 22 Oct 2023 20:10:30 +0000

If the blog post's title caught your attention, then you're probably like me; you remember the big things and always need to remind yourself of the small stuff. You spell length as "lenght" all the time, then go back to correct yourself; you need someone to remind you of the correct way to end a formal letter: yours faithfully, yours sincerely, yours violently, or yours sarcastically, and when joining tables in SQL, you argue with yourself about which table you're joining is the left or right table.

So Which One Is The Left Table?

I'll tell you now, for the one hundred and sixty-fifth times, the table that follows your FROM keyword is always the left table. The right table is the one that comes immediately after the JOIN keywords. You just need to remember one of those.

Let's Practice

SELECT post.* FROM users LEFT JOIN  posts ON   users.id = posts.user_id

Now, which table is the left table? The left table in the query above is the users' table. Why? Because it comes immediately after the "FROM" keyword. That makes the posts table the right table.

Let's flip it around.

SELECT users.* FROM users RIGHT JOIN posts ON posts.user_id = users.id

Now, which table is the right table? The right table is still the posts table. To identify the right table, first identify the left table. The left table is the table that comes immediately after theFROM keyword, which is the users table. This automatically makes the posts table the right table.

Now we get it but let's bookmark this blog post because I know we'll forget again 😉.

Frontend Vs Backend: An Objective Look

Olufisayo Bamidele — Fri, 20 Oct 2023 23:29:52 +0000

Image credit: geeks for geeks

I wrote a short post about how backend development is more critical than frontend development and shared it among a few friends and colleagues yesterday; that's just me being silly.

Today, I have decided to write a more abjective blog post about the debate. Without further ado, here's what I think.

There are no Frontends Or Backends.

Yes, there are no frontends or backends. We only have software that runs on machines; any software can be a client or server at anytime. The end user is the only one who doesn't get to serve as a server. What do I mean?

Let's start at the very top of the stack

The UI: There are many kinds of UI, but in this instance, I'm talking about the web UI. Yes, it is a client to a web backend (say a RESTful API), but simultaneously a server for the end user. But a RESTFUL API is also a client to a database server.

It might surprise you that databases also have frontends and backends. Well, databases are as complex as operating systems, but to keep it as simple as possible, I'd say most databases contain two main parts:

The Query Engine: This is the front end of your database. It takes your query, plans execution, and optimizes it when possible. The query engine then passes your question to the next component.

The Storage Engine: This is your database's backend. It takes the prepared query from the query engine and executes it(oversimplification). This layer handles data storage and retriever from disk, what gets stored in memory, scheduled database cleanups, etc. Of course, it achieves this by talking to the operating system, so we can accurately say that a database's storage engine is a client of the operating system. But wait a minute; the operating system also has a front and back end, namely, the userland and kernel, which are the front and back end of the operating system.

Well, by now, you get the gist. No matter what kind of software you choose to write, your software is serving and being served simultaneously.

But which one is more complex?

I can tell you this: creating a database, a compiler, an operating system, and anything that directly talks to the hardware is always difficult. Every other thing is easy, but at scale, nothing is.

You could sort an array containing five items, but let's make that five billion items, and now you'd have to deal with all sorts of demons.

We are all software developers; some prefer to work on the top of the stack, while others prefer to work very close to the bottom. Whatever area you like to stay, ensure you're competent and can build software that works at various scales.

Tip: Never Forget a console.log In Your Patch Again

Olufisayo Bamidele — Thu, 19 Oct 2023 15:37:55 +0000

TLDR; stage your changes git add . -p to slow down your personal code review process.

Yes, we've all been there. You've spent the last 5 hours looking for a bug in your codebase and finally came up with a fix. You implemented the patch and removed all your log statements or thought you did. You called for a review only to get rejected because you left some unnecessary logs in your patch.

Have you tried running `git add` with the `-p` flag?

With git add <path/to/the/changed/files> -p, git brings back your changes a chunk at a time or, in git lingo, one hunk at a time.

Geek bit: See here for more info about what a hunk is.

As git gradually reveals the changes you made, it presents the following options:
y - stage this hunk
n - do not stage this hunk
q - quit; do not stage this hunk or any of the remaining ones
a - stage this hunk and all later hunks in the file
d - do not stage this hunk or any of the later hunks in the file
e - manually edit the current hunk
? - print help

Side Note:
For the e option, git presents Vim as your default editor. Again, you meet Vim 😈 . To change git's default editor from Vim to vscode, run git config --global core.editor = "code" assuming you have already added vscode to your path variable. I'll advise against this, though; take this opportunity to learn basic Vim motions.

Once git has presented you with all the hunks, you'd be left with only the ones you accepted. The rejected hunks would be left unstaged. Staging your file in git like this allows you to slow down and do a personal code review, allowing you to catch those annoying leftover comments, logs, and debug statements.

Gotchas

The -p flag only works for files already known by git. If you've never staged a file before, git will never present you with hunks from new files because there are no hunks. A hunk is nothing but a diff between the previous version of a file and the newest version, revealed one chunk at a time. There is nothing to diff for a new file, so I like to stage new files with the -N flag, i.e., git add . -N. The "N" flag does not stage the new file but instructs git that you intend to do so later. Now git knows about the file and will present the entire file to you when you're staging your changes with the -p flag.

Conclusion

I couldn't think of a way to conclude this article, so I inserted a dry joke here.👇🏽

If git is a tool, and this article presents tips about git, does that make it a tooltip❓🤔

Okay, I'll give you the time to cringe now. See you on the next one.

POSTGRESQL PSEUDOCOLUMNS: CTID

Olufisayo Bamidele — Tue, 17 Oct 2023 07:24:07 +0000

Postgres CTID is an internal table column identifying how Postgres stores table data physically on the disc. It comes as two comma-separated numbers, e.g. (0, 1), 0 here serving as the page number and 1 is the row's location on the page.

Let's establish some knowledge about the CTID column.

Set Up a Postgres Database

Let's spin up a Postgres container on our local machine.



docker run --rm -it --name pg-internals-ctid -p 5438:5432 -e POSTGRES_PASSWORD=verysecret -e POSTGRES_USER=verysecretuser -e POSTGRES_DB=ctidplayground  postgres

The command above spins up a docker container called pg-internal-ctid, and publishes the container's internal port 5432 to port 5438 on your local machine with the default password, "verysecret", a user named "verysecretuser" and a default db called "ctidplayground".

If that command works, you should get some outputs on your command line, and the last line of that output should look like this:



2023-10-15 01:32:35.278 UTC [1] LOG:  database system is ready to accept connections.

Note:If all of the above is foreign to you, drop a comment if you'd like me to start a series on docker.

Let's Connect To Our Database

Note: You can skip the node project setups and run the queries directly in the docker container created above.

I assume you already know how to set up a typescript/nodejs project, so let's install a Postgres client for our database.

npm install --save postgres

In src/index.ts, let's connect to our Postgres instance and create a "users" table.



import postgres from 'postgres';

// connect to the postgres instance
const sql = postgres();

async function createUsersTable() {
    const result = await sql`
CREATE TABLE IF NOT EXISTS users (
    name varchar(100) NOT NULL,
    country varchar(100) NOT NULL,
    age integer NOT NULL
    )
`;

    console.log('the creation results', result);
}

async function main() {
    await createUsersTable();
}

main();

In the snippet above, the call to postgres() looks for the following environment variables for connection details:

PGHOST, PGPORT, PGDATABASE, PGUSER, PGPASSWORD

The snippet, then, creates a "users" table with columns name, country, and age.

Next, let's run our code to see if this works.

In your package.json file, add the following script.



"start:dev":  "PGHOST=localhost PGPORT=5438, PGDATABASE=ctidplayground PGUSER=verysecretuser PGPASSWORD=verysecret nodemon src/index.ts"

Now run the just-added script in your terminal



npm run start:dev

You should see the following logged in your console.



the creation results Result(0) []

Execute the following commands to confirm that the table was created.



docker exec -it  pg-internals-ctid psql -d ctidplayground -U verysecretuser

You should get the following prompt if everything works well.



ctidplayground=#

Now instruct Postgres to describe the user table with the command \d users. Your output should be similar to the one in the screenshot below

If something went wrong in our code, the output would be "Did not find any relation named users."

Now, let's insert some records into our user's table.

Inserts three users into the database and calls the function in the main.



async function insertUsers() {
    const result = await sql`
        INSERT INTO users (name, country, age)
        VALUES(
            UNNEST(ARRAY['John Doe', 'Tom Williman', 'Billy Wilson']),
            UNNEST(ARRAY['Nigeria', 'Canada', 'USA']),
            UNNEST(ARRAY[32, 16, 16])
        )
        RETURNING *

    `;
    console.log('insersion results', result);
}

async function main() {
    await insertUsers();
}

If that works, the inserted users should be logged on your console like so:



insertion results Result(3) [
  { name: 'John Doe', country: 'Nigeria', age: 32 },
  { name: 'Tom Williman', country: 'Canada', age: 16 },
  { name: 'Billy Wilson', country: 'USA', age: 16 }

Confirming The Characteristics Of CTID the Column

Now, let's establish some characteristics of the CTID column.

1. Inserted columns have Sequential CTIDs, and the latest modified records always have higher CTID values.

If you've already closed your Postgres shell, reopen it by running the previous docker exec command



docker exec -it  pg-internals-ctid psql -d ctidplayground -U verysecretuser

Next, list the inserted record with their ctid by running the following query.



SELECT *, ctid FROM users;

Your result should look like the one in the screenshot below, confirming the above facts.

2. CTIDs cannot be a unique identifier

You cannot use a ctid as a unique identifier because an update operation or a full vacuum command after deletion can reassign new CTIDs existing rows. Let's see this in action.

Run the following query against the user database.



UPDATE users SET age=35 WHERE name = 'John Doe';

Now select all records from the user's table again;



SELECT *, ctid FROM users;

John Doe now has a CTID of (0, 4) instead of the original (0, 1) ctid. The first record on the table is now that of Tom Williman with a CTID of (0,2).

What happened to ctid (0,1)? Just by this observation, it can be established that Postgres doesn't mutate existing tuples. An update inserts a new record into the database and assigns the pointer to the old record to the new one. That leaves old records as garbage, which are automatically skipped during scans.

To reclaim this space taken by the garbage, run the VACUUM FULL; command in your pg shell. Making a select *, ctid again. This should make our record completely sequential again; the ctid now starts from (0, 1).

To summarise the preceding paragraphs, the last modified record always has the highest ctid on a table, and ctids should never be used as identifiers. However, if there is a table without a updated_at column, or for some reason (e.g., during a bulk concurrent insert), multiple columns end up having the same updated_at timestamp, the ctid column can be used to decide which record gets updated last.

USE CASE FOR CTID: Identifying the last/first inserted record in duplicates and Deduplication.

Since we've observed that the last modified record always holds the highest ctid value, ctid is a great candidate for picking the latest record between two duplicates. At some point in our career, we would have to insert records from CSV files prone to human errors into a database table; Human errors like multiple entries of the same record. Whenever you have a guarantee that a database table is an append-only table (i.e., the records are immutable), selecting the record with the max ctid value would give you the latest value of that record.

Let's modify our insertUsers function to



async function insertUsers() {
    // clear users table
    await sql'TRUNCATE users;`;

    // insert data into the users table;
    const result = await sql`
        INSERT INTO users (name, country, age)
        VALUES(
            UNNEST(ARRAY['John Doe',' Tom Williman', 'Billy Wilson',' John Doe',' Tom Williman', 'Billy Wilson']),
            UNNEST(ARRAY['Nigeria', 'Canada', 'USA', 'Nigeria', 'Canada', 'USA']),
            UNNEST(ARRAY[32, 16, 16, 25, 21, 22])
        )
        RETURNING *

    `;
    console.log('insersion results', result);
}
```
The snippet would insert six records into the database. Each unique record has one duplicate. Our table should now look like that in the screenshot below.


![Insert duplicate rows](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kgdo50tjbxcgoja598z8.png)


To select the latest duplicate record, use the query below

```sql
SELECT users.*, users.ctid  FROM  users INNER JOIN (SELECT max(ctid) AS max_ctid, name FROM users GROUP BY name) AS latest_duplicate
ON users.ctid = latest_duplicate.max_ctid;
```
The query above has a subquery that groups the records in the user's table by name(assuming the name is supposed to be the user identifier here). From each group, we select the record with the maximum user ctid. We join this subquery with the original users' table to select all users' properties.

Your output should look like the screenshot below

![Deduplicating with ctid](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lss7qfzprtm4sy5r1d2v.png)
We ended up with the last inserted duplicate record.

## What if You can't Guarantee That Your Database Table is an Append-Only Table

Solving the above problem in your database strictly depends on problem-solving skills. I would solve this problem by creating an intermediate table that guarantees immutability(I would probably use a Postgresql materialized view for this). I would always perform the Deduplication on the intermediate table and merge the Deduplication result with the final table. The algorithm would be something like this.

• insert records from the users CSV file into the `temp_users` table
• deduplicate by selecting max ctid record from the DB
• merge the deduplicated record with the existing users' table, i.e
    • if a user record already exists in the users' table, update the record with the latest record, `temp_users`
    • else insert a new user record.

## Why Can't I DO This In Memory?

Sometimes, you just can't. Let's assume you're to insert and dedupe a file with 10 million rows, and all you have is a virtual machine with just 1 gig of memory. Well, you could try doing that in memory which would end up eating up all the resources on your server. Or you could actually do the deduplication in the database, which was built to handle that kind of task.

Conclusion

Beyond just coding, having a deep understanding of how your tool works internally is always beneficial. If you're interested in topics like this, follow for more.

If you're a beginner, this article introduces some new database concepts. Feel free to request a deep dive on any of them; as usual, I appreciate all forms of feedback.

Tool: Preview Markdown Document on Your Terminal

Olufisayo Bamidele — Fri, 13 Oct 2023 14:53:16 +0000

If you've recently been initiated into the Vim users cult and, like me, you also don't like leaving your terminal that much anymore, you're probably being forced to do so whenever you want to do stuff like previewing your markdown documents. Sweat no more; glow got you covered.

Installation

On Mac, brew install glow.

Usage

Usage: glow MyWonderfulDoc.md

Alternative: markdown + lynx

Markdown: This is a command line tool for converting your MD files to HTML
Lynx: A terminal-based web browser for rendering HTML pages.

Installation

brew install lynx markdown

Usage

Convert your MD file to HTML with markdown and pipe the output to Lynx

markdown MyWonderfulDoc.md | lynx -stdin

DEV Community: Olufisayo Bamidele

Notify Yourself After Completing a Long-Running Bash Process

Say hello to the "say" command

Future of Software Engineering: Is Maintainability Still Important?

Today's programming language, tomorrow's joke.

Kubernetes Services; Expose your app to the Internet

Kubernetes Services.

What is a K8S-Service?

Why Do We Need K8s-Services

Types Of k8s-Service

ClusterIP K8S-Service

Defining a ClusterIP in Kubernetes

NodePort K8S-Service

Defining a NodePort K8s-service

LoadBalancer K8s-service

Defining a LoadBalancer K8s-service

Deploying our Kubernetes Workload to the Cloud

Step 1: Set up the projects we want to deploy

Step 2: Build The Project As a Docker Image

Step 4: Push the image to Docker Hub

Step 5: Update your node-echo-deployment.yaml file1.

Step 6: Update the node-echo-service.yaml file

Step 7: Create a GKE Cluster On GCP

Step 8: Install GCloud CLI if you've not done that already

Step 9: Log in to Google Cloud on your CLI and your gcloud project

Step 9: Connect to your GKE Cluster on your local machine

Deploy To GKE Cluster

Test your deployment

The server would echo "Hello world" back to you

Infer Function Return Type In Typescript

Scenario

Bad Solution

Good solution

Bonus

Summary

Docker and Kubernetes From Localhost To Production: Kubernetes - Container Orchestrators, The What, Why and How

What is Kubernetes

Let's learn those concepts by associating them with concepts we already know

Essential Kubernetes Components/Terminologies

Workload

Pod

Let's associate pods with what we already know

Node

Cluster

The Whys

Are Containers not Enough Isolation? Why Do We Need Pods?

But why do I need Kubernetes?

But I Only Have A Portfolio Website. Why should I use Kubernetes?

Why is it so complicated?

The How

How to Set up Kubernetes On Your Computer System

Let's get a simple application to work with

Run the image in a Kubernetes node

Step 1: Create a Kubernetes Deployment Config

Step 2: Submit That Config To Kubernetes

Step 3: Confirm The Created Deployment

Step 4: Test our running workload

Step 5: Create a Kubernetes Service

Step 6: Submit the Kubernetes service config

Conclusion

Docker and Kubernetes From Localhost To Production: Containers; The what, why, and How

Why Do Docker and Kubernetes Never Stick For Many People?

The Whats

A Container

A Container Image:

A Container Engine/Container Runtime:

Docker

A DockerFile

Let's Relearn All The Concepts Above By Associating It With What We Already Know From Bottom Back To The Top

The Why

I already know how to run a program on a computer; why do I also need to run it in a container?

Yes, I know this already, but these businesses can still run multiple programs on a single machine.

Talking about Isolation and resource allocation, isn't that achievable Through Virtual Machines? Why Do I Still Need Containers?

So, if virtual machines solve the problem with shared hosting, why do you still need containers?

Do Container Technologies Solve the: "But it works on my machine problem"

The How

Step 1: Let's create a simple echo server in nodejs

Step 2: Write the requirements of your docker image in a Dockerfile

Step 3: Build the image using the DockerFile we just created; run the following command

Step 4: Create a running container with the image

Step 5: Update your `node-echo-deployment.yaml` file1.

Have you tried running `git add` with the `-p` flag?