DEV Community: Vinoth Mohan

Kubernetes - An Overview at High-Level

Vinoth Mohan — Mon, 07 Mar 2022 08:32:36 +0000

What Is Kubernetes?

Kubernetes is popularly known as K8s is a production-grade open-source container orchestration tool developed by Google to help you manage the containerized/dockerized applications supporting multiple deployment environments like On-premise, cloud, or virtual machines.

Why is it popular and powerful ?

Kubernetes can speed up the development process by making easy, automated deployments, updates (rolling-update) and by managing our apps and services with almost zero downtime. It also provides self-healing. Kubernetes can detect and restart services when a process crashes inside the container.Its is highly performant,scalable and provides reliable infrastructure to support data recovery with ease.

Architecture Of Kubernetes

The most basic architecture of k8s has two major Nodes.

Control plane
Worker Nodes

Control plane

The master node is also known as a control plane that is responsible to manage worker nodes efficiently. User enters commands and configuration files from control plane. It controls all cluster.

Master Node Processes:

kube-apiserver : It exposes the Kubernetes API. The API server is the front end for the Kubernetes control plane.
kube-controller-manager : Control plane component that runs controller processes.
Logically, each controller is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process. Some types of these controllers are:
- Node controller: Responsible for noticing and responding when nodes go down.
- Job controller: Watches for Job objects that represent one-off tasks, then creates Pods to run those tasks to completion.
- Endpoints controller: Populates the Endpoints object (that is, joins Services & Pods).
- Service Account & Token controllers: Create default accounts and API access tokens for new namespaces.

kube-scheduler: Control plane component that watches for newly created Pods with no assigned node, and selects a node for them to run on.

Factors taken into account for scheduling decisions include:
- individual and collective resource requirements,
- hardware/software/policy constraints,
- affinity and anti-affinity specifications,
- data locality,
- inter-workload interference,
- deadlines.

etcd: Consistent and highly-available key value store used as Kubernetes' backing store for all cluster data (meta data, objects, etc.).

Cloud Controller Manager : It embeds cloud-specific control logic. The cloud controller manager lets you link your cluster into your cloud provider's API, and separates out the components that interact with that cloud platform from components that only interact with your cluster. The cloud-controller-manager only runs controllers that are specific to your cloud provider. The following controllers can have cloud provider dependencies:
- Node controller: For checking the cloud provider to determine if a node has been deleted in the cloud after it stops responding
- Route controller: For setting up routes in the underlying cloud infrastructure
- Service controller: For creating, updating and deleting cloud provider load balancers.

Worker Node

Node components run on every node, maintaining running pods and providing the Kubernetes runtime environment.

Kubelet: An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy.
Kube-proxy: It is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept.
- It maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster.
- It uses the operating system packet filtering layer if there is one and it's available. Otherwise, kube-proxy forwards the traffic itself.

Container Runtime: The container runtime is the software that is responsible for running containers.
- Kubernetes supports several container runtimes: Docker, containerd, CRI-O, and any implementation of the Kubernetes CRI (Container Runtime Interface)"

Components of K8s:

Pods:

A pod is a smallest and simplest unit that you create or deploy in k8s, which is an abstraction of the container application.A single pod has usually one, or multiple containers, and their shared resources.

Deployments

Deployments are best used for stateless applications. Pods managed by deployment workload are treated as independent and disposable.

If a pod encounters disruption, Kubernetes removes it and then recreates it.

DaemonSets

Daemonsets ensures that every node in the cluster runs a copy of the pod.

For use cases where you're collecting logs or monitoring node performance, this daemon-like workload works best.

ReplicaSets

A ReplicaSet's purpose is to run a specified number of pods at any given time.

StatefulSets

Like a Deployment , a StatefulSet manages Pods, are best used when your application needs to maintain its identity and store data.

An application would be something like Zookeeper - an application that requires a database for storage.

Jobs

Jobs launch one or more pods and ensure that a specified number of them successfully terminate.

Jobs are best used to run a finite task to completion as opposed to managing an ongoing desired application state.

CronJobs

CronJobs are similar to jobs. CronJobs, however, runs to completion on a cron-based schedule.

Services

An abstract way to expose an application running on a set of Pods as a network service.
With Kubernetes you don't need to modify your application to use an unfamiliar service discovery mechanism. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods, and can load-balance across them.

Type of services

ClusterIP. Exposes a service which is only accessible from within the cluster.
NodePort. Exposes a service via a static port on each node’s IP.
LoadBalancer. Exposes the service via the cloud provider’s load balancer.
ExternalName. Maps a service to a predefined externalName field by returning a value for the CNAME record.

Kubernetes Ingress

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

Ingress Controller

In order for the Ingress resource to work, the cluster must have an ingress controller running.

Some third-party ingress controller:

Istio Ingress is an Istio based ingress controller.
The NGINX Ingress Controller for Kubernetes works with the NGINX webserver (as a proxy).
The Traefik Kubernetes Ingress provider is an ingress controller for the Traefik proxy.

Secrets

A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret means that you don't need to include confidential data in your application code.

ConfigMap

A ConfigMap is an API object used to store non-confidential data in key-value pairs. Pods can consume ConfigMaps as environment variables, command-line arguments, or as configuration files in a volume.

A ConfigMap separates your configurations from your Pod and components, which helps keep your workloads portable. This makes their configurations easier to change and manage, and prevents hardcoding configuration data to Pod specifications.

Persistent Volumes

As we know data stored in container is ephermal. So we need persistant volume to preserve data across container restart.

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes.

Persistant Volume Claim

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes

Advanced

Some advanced topics you can explore which helps you become better kubernetes adminstrator.

Liveness and Readiness Probe
Requests and Limits
Resource Quotas
Auto Scaling
RBAC Authorization

These are some basic topics which help you getting started in kubernetes. In order to get hands-on in kubernetes, You can easily set up Minikube or Amazon EKS cluster using eksctl.

Thats it!! See you 😁

Setting up Minikube in EC2 - The Easy Way

Vinoth Mohan — Thu, 24 Feb 2022 18:31:49 +0000

Minikube is a tool that lets you run a single-node Kubernetes cluster on your personal computer or in EC2 instance so that you can try out Kubernetes.

Step 1 : Launch an EC2 Instance using the following configuration.

AMI : Ubuntu Server 20.04 LTS (HVM), (64-bit x86)
Instance Type : t2.medium
Configure Instance Details :
- In this section scroll to the bottom, there you will find user data field, Copy this script and paste it in that field

#!bin/bash
sudo apt update
sudo apt upgrade -y
sudo hostnamectl set-hostname minikube
sudo apt-get install -y apt-transport-https ca-certificates curl
sudo apt-get update -y &&  sudo apt-get install -y docker.io
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
snap install kubectl --classic
sudo apt install conntrack -y

Add Storage : 10GiB
Add Tags :
- Key : Name
- Value : minikube
Security Group : Create a New Security Group
- Type : All Traffic
- Source : My IP (or) Leave it as 0.0.0.0/0
Review and Launch : Select or Create a new key pair
- Launch

Step 2 : SSH into your EC2 instance

$ ssh -i [keypair] ubuntu@[Instance Public IP]

Step 3 : Verify

As we already bootstrapped the all necessary installation, Only thing that we need to do is to verify the installed Minikube by checking the version.

$ minikube version

Step 4 : Start running Minikube

Become a root user

$ sudo -i

Start the minikube

$ minikube start --vm-driver=none

Check Status

$ minikube status

Thats it!!! Minikube is now running your EC2 instance !.
You can play with kubernetes using this single node kubernetes cluster.

Terraform - An Overview at High-Level

Vinoth Mohan — Wed, 23 Feb 2022 16:00:36 +0000

What is Terraform? 🤔

Terraform is an open-source, cloud-agnostic, one of the most popular Infrastructure-as-code (IaC) tool developed by HashiCorp. It is used by DevOps teams to automate infrastructure tasks such as provisioning of your cloud resources.

Terraform supported immutable infrastructure, a declarative language, a masterless and agentless architecture, and had a large community and a mature codebase.

Benefits of using Terraform:

Does orchestration, not just configuration management
Supports multiple providers such as AWS, Azure, Oracle, GCP, and many more
Provide immutable infrastructure where configuration changes smoothly
Uses easy to understand language, HCL (HashiCorp configuration language)
Easily portable to any other provider

How Terraform Works?

Terraform has two main components that make up its architecture:

Terraform Core
Terraform Providers

Terraform Configuration Files

Configuration files are a set of files used to describe infrastructure in Terraform and have the file extensions .tf and .tf.json. Terraform uses a declarative model for defining infrastructure. Configuration files let you write a configuration that declares your desired state. Configuration files are made up of resources with settings and values representing the desired state of your infrastructure.

A Terraform configuration is made up of one or more files in a directory, provider binaries, plan files, and state files once Terraform has run the configuration.

Configuration file (*.tf files): Here we declare the provider and resources to be deployed along with the type of resource and all resources specific settings
Variable declaration file (variables.tf or variables.tf.json): Here we declare the input variables required to provision resources
Variable definition files (terraform.tfvars): Here we assign values to the input variables
State file (terraform.tfstate): a state file is created once after Terraform is run. It stores state about our managed infrastructure.

Terraform Providers

A provider is responsible for understanding API interactions and exposing resources. It is an executable plug-in that contains code necessary to interact with the API of the service. Terraform configurations must declare which providers they require so that Terraform can install and use them.

Terraform has over a hundred providers for different technologies, and each provider then gives terraform user access to its resources. So through AWS provider, for example, you have access to hundreds of AWS resources like EC2 instances, the AWS users, etc.

Terraform Core Concepts

Variables: Terraform has input and output variables, it is a key-value pair. Input variables are used as parameters to input values at run time to customize our deployments. Output variables are return values of a terraform module that can be used by other configurations.
Provider: Terraform users provision their infrastructure on the major cloud providers such as AWS, Azure, OCI, and others. A provider is a plugin that interacts with the various APIs required to create, update, and delete various resources.
Module: Any set of Terraform configuration files in a folder is a module. Every Terraform configuration has at least one module, known as its root module.
State: Terraform records information about what infrastructure is created in a Terraform state file. With the state file, Terraform is able to find the resources it created previously, supposed to manage and update them accordingly.
Resources: Cloud Providers provides various services in their offerings, they are referenced as Resources in Terraform. Terraform resources can be anything from compute instances, virtual networks to higher-level components such as DNS records. Each resource has its own attributes to define that resource.
Data Source: Data source performs a read-only operation. It allows data to be fetched or computed from resources/entities that are not defined or managed by Terraform or the current Terraform configuration.
Plan: It is one of the stages in the Terraform lifecycle where it determines what needs to be created, updated, or destroyed to move from the real/current state of the infrastructure to the desired state.
Apply: It is one of the stages in the Terraform lifecycle where it applies the changes real/current state of the infrastructure in order to achieve the desired state.

Terraform Lifecycle

Terraform lifecycle consists of – init, plan, apply, and destroy.

Terraform-lifecycle

Terraform init initializes the (local) Terraform environment. Usually executed only once per session.
Terraform plan compares the Terraform state with the as-is state in the cloud, build and display an execution plan. This does not change the deployment (read-only).
Terraform apply executes the plan. This potentially changes the deployment.
Terraform destroy deletes all resources that are governed by this specific terraform environment.

Hands-on : Create a EC2 instance using Terraform

Prerequisite: A laptop or machine with terraform installed and AWS cli configured

Step 1: Create main.tf and variables.tf inside a folder

main.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  profile = "default"
  region  = "ap-south-1"
}

resource "aws_instance" "app_server"{
  ami           = var.ami
  instance_type = var.instance_type
  tags = {
    Name = "TerraInstance"
  }
}

variables.tf

variable "ami" {
    type = string
    default = "ami-0c6615d1e95c98aca"
}
variable "instance_type" {
    type = string
    default = "t2.micro"
}

Step 2 : Initialize the directory with $ terraform init command
Step 3 : Run $ terraform plan command to print out the execution plan
Step 4 : Apply the configuration now with the $ terraform apply command.
Step 5 : Inspect the current state using $ terraform show and check AWS console whether a instance is created.
Step 6 : Run $ terraform destroy command to destroy the created instance.

Thats it!!! Now that you sucessfully created a EC2 instance with terraform.

Checkout my github repository for more 💁:
https://github.com/iamvinot/terraform-project

Docker - An Overview at High-Level

Vinoth Mohan — Sat, 19 Feb 2022 17:17:16 +0000

What is Docker ?

Docker is an open-source platform for building distributed software using “containerization."

Docker allows you to decouple the application/software from the underlying infrastructure into standardized units called containers that have everything the software needs to run including libraries, system tools, code, and runtime.

Why Docker ?

Docker containers are minimalistic and enable portability.
- Docker lets applications and their environments be kept clean and minimal by isolating them, which allows for more granular control and greater portability.
Docker containers enable composability.
- Containers make it easier for developers to compose the building blocks of an application into a modular unit with easily interchangeable parts, which can speed up development cycles, feature releases, and bug fixes.
Docker containers ease orchestration and scaling.
- Because containers are lightweight, developers can launch lots of them for better scaling of services. These clusters of containers do then need to be orchestrated, which is where Kubernetes typically comes in.

What is Virtualization?

Virtualization is the process of creating virtual enviroment or virtual machine by spliting one system into many different sections which act like separate, distinct individual systems. A software called Hypervisor makes this kind of splitting possible.

What is Containerization ?

Containerization is a form of virtualization through which applications are run in containers (isolated user spaces) all using a shared OS. It packs or encapsulates software code and all its dependencies for it to run in a consistent and uniform manner on any infrastructure.

Virtual Machine vs Docker ?

Virtual Machines (VMs) virtualize the underlying hardware. They run on physical hardware via an intermediation layer known as a hypervisor. They require additional resources are required to scale-up VMs.

They are more suitable for monolithic applications. Whereas, Docker is operating system level virtualization. Docker containers userspace on top the of host kernel, making them lightweight and fast. Up-scaling is simpler, just need to create another container from an image.

Virtual Machine	Containers
A virtualization technique where each VM has an individual operating system.	A virtualization technique where all containers share a host operating system.
Virtual machines are isolated at the hardware level	Each container is isolated at the operating system level.
Virtual machines take time to create	Containers are created fast
Increased management overhead	Decreased management overhead as only one host operating system needs to be cared for.


VM	Docker

What is Hypervisor?

A hypervisor is a software that makes virtualization possible. It is also called Virtual Machine Monitor. It divides the host system and allocates the resources to each divided virtual environment.

What are Docker Images?

A Docker image is an executable file, that creates a Docker container. An image is built from the executable version of an application together with its dependencies and configurations. Running instance of an image is a container.

Docker image includes system libraries, tools, and other files and dependencies for the application. An image is made up of multiple layers.

What is Docker Hub?

Docker images create docker containers. There has to be a registry where these docker images live. This registry is Docker Hub. Users can pick up images from Docker Hub and use them to create customized images and containers. Currently, the Docker Hub is the world’s largest public repository of image containers.

Components of Docker Architecture.

The four major components of Docker are daemon, Client, Host, and Registry

Docker daemon: It is also referred to as ‘dockerd’ and it accepts Docker API requests and manages Docker objects such as images, containers, networks, and volumes. It can also communicate with other daemons to manage Docker services.
Docker Client: It is the predominant way that enables Docker users to interact with Docker. It sends the docker commands to docker, which actually executes them using Docker API. The Docker client can communicate with more than one daemon.
Docker Registry: It hosts the Docker images and is used to pull and push the docker images from the configured registry. Docker Hub is the public registry that anyone can use, and Docker is configured to look for images on Docker Hub by default. However, it is always recommended for organizations to use own private registry.
Docker Host: It is the physical host (VM) on which Docker Daemon is running and docker images and containers are created.

What is Docker Engine?

Docker daemon or Docker engine represents the server. The docker daemon and the clients should be run on the same or remote host, which can communicate through command-line client binary and full RESTful API.

What is Docker Image Registry?

A Docker image registry, in simple terms, is an area where the docker images are stored. Instead of converting the applications to containers each and every time, a developer can directly use the images stored in the registry.
This image registry can either be public or private and Docker hub is the most popular and famous public registry available.

What are Dockerfiles?

Dockerfile is a text file that has instructions to build a Docker image. All commands in dockerfile could also be used from the command line to build images.

Sample Dockerfile :

FROM ubuntu:16.04
COPY . /app
RUN make /app
CMD python /app/app.py

Each instruction in a dockerfile creates one read-only layer:

Docker Commands

Pull Command

This command pulls an image from a docker public registry.

docker pull docker/whalesay

Build Command

This command builds an image according to Dockerfile.

docker build [-t <name_of_image>] [-f <name_of_Dockerfile>] <path_to_Dockerfile>

Run Command

This command runs an container of an image.

docker run --name nginx-container nginx:1.16

ps Command

This command lists the docker containers.

docker ps

Stop Command

This command stops a running container(s).

docker stop nginx-container

Remove Command

This command removes a stopped container(s).

docker rm nginx-container

List Image Command

This command lists the docker images.

docker images

Remove Image Command

This command removes image(s).

docker rmi nginx

Attach Command

This command attaches the terminal to a container running in the background (detached mode).

docker attach <container id or name>

Inspect Command

This command returns details of the container in a JSON format.

docker inspect <container id or name>

Logs Command

This command returns logs of the container running in the background (detached mode).

docker logs <container id or name>

Push Command

This command pushes an image to your account on a docker public registry (dockerhub).

docker push vinothmohan/pro-postgres

Create a Docker Container

Following command creates the docker container with the required images.

docker create --name <container-name> <image-name>

Pause Container

Processes running inside the container is paused. Following command helps us to achieve this.

docker pause <container-id/name>

Container can’t be removed if in a paused state.

Unpause Container

Unpause moves the container back to run the state. Below command helps us to do this.

docker unpause <container-id/name>

Start Container

If container is in a stopped state, container is started.

docker start <container-id/name>

Stop Container

Container with all its processes is stopped with below command.

docker stop <container-id/name>

To stop all the running Docker containers use the below command

docker stop $(docker ps -a -q)

Restart Container

Container along with its processes are restarted

docker restart <container-id/name>

Kill Container

A container can be killed with below command

docker kill <container-id/name>

Destroy Container

The entire container is discarded. It is preferred to do this when the container is in a stopped state rather than do it forcefully.

docker rm <container-id/name>

Docker Network

Bridge

The Bridge network assigns IPs in the range of 172.17.x.x to the containers within it. To access these containers from outside you need to map the ports of these containers to the ports on the host.

Host

The Host network will remove any network isolation between the docker host and the containers. For instance, if you run a container on port 5000, it will be accessible on the same port on the docker host without any explicit port mapping. The only downside of this approach is that you can not use the same port twice for any container.

None

The None network keeps the container in complete isolation, i.e. they are not connected to any network or container.

To create Network

docker network create --driver driver_name network_name

Types of Volume mounts in Docker.

There are three mount types available in Docker

Volume mounts are the best way to persist data in Docker. Data are stored in a part of the host filesystem which is managed by Docker containers. (/var/lib/docker/volumes/ on Linux)

-v or --volume flag and --mount flag could be used for docker swarm services and standalone containers.

To create a docker volume. For eg:

docker volume create my-vol

Inspect a volume

docker volume inspect my-vol

If we need to start a container with “my-vol”

With -v flag

docker run -d  --name devtest -v my-vol:/app nginx:latest

Here nginx images with the latest tag are executed with using volume mount “my-vol”

With --mount flag

docker run -d --name devtest --mount \ source=my-vol,target=/app nginx:latest

Bind mounts may be stored anywhere on the host system. A file or directory on the host machine is mounted into a container unlike volume mounts where a new directory is created within Docker’s storage directory on the host machine, and Docker manages that directory’s contents. Non-Docker processes on the Docker host or a Docker container can modify them at any time.

tmpfs mounts are stored in the host system’s memory only and are never written to the host system’s file system. When the container stops, the tmpfs mount is removed, and files won’t persist.

Docker Compose

Docker Compose is a tool provided by Docker for defining and running multi-container applications together in an isolated environment. Either a YAML or JSON file can be used to configure all the required services like Database, Messaging Queue along with the application server. Then, with a single command, we can create and start all the services from the configuration file.

It comes handy to reproduce the entire application along with its services in various environments like development, testing, staging and most importantly in CI as well.

Typically the configuration file is named as docker-compose.yml. Below is a sample file:

version: '3'
services:
  app:
    image: appName:latest
    build: .
    ports:          
    - "8080"   
    depends_on:
      - oracledb
    restart: on-failure:10    
  oracledb:
    image: db:latest 
    volumes:
      - /opt/oracle/oradata
    ports:       
      - "1521"

docker-compose up

Git - An Overview at High-Level

Vinoth Mohan — Tue, 15 Feb 2022 08:04:04 +0000

What is GIT?

Git is a free and open-source distributed version control system for tracking changes in computer files and is used to help coordinate work among several people on a project while tracking progress over time. In other words, it’s a tool that facilitates source code management in software development.

Git favors both programmers and non-technical users by keeping track of their project files. It enables multiple users to work together and handles large projects efficiently.

What do you understand by the term ‘Version Control System’?

A version control system (VCS) records all the changes made to a file or set of data, so a specific version may be called later if needed.

This helps ensure that all team members are working on the latest version of the file

Uses of Git

Git is used to tracking changes in the source code.
Distributed version control tool used for source code management.
Allows multiple developers to work together.
Supports non-linear development because of its thousands of parallel branches.

Features of Git

Free and open-source
Tracks history
Supports non-linear development
Creates backup
Scalable
Supports collaboration
Branching is easier
Distributed development.

Git WorkFlow

The following image shows the git workflow diagram:

In Git, the workflow is mainly divided into three areas -

Working directory - This is the area where you modify your existing files.
Staging area (Index) - In this, the files in your working directory are staged and snapshots are added.
Git directory or repository - It is basically where you perform all the changes that need to be made i.e. perform commits to branch, checkout branch, make changes etc.

Git Init

git init is one way to start a new project with Git. To start a repository, use either git init or git clone - not both.

git init

Git Clone

git clone is a command which is used to clone or copy a target repository.

git clone <REPO_URL>

Clone a specific branch from the repository.

git clone -b <Branch_name> <Repo_URL>

Git Status

git status is mainly used to display the state of the staging area and the repository. It helps us to track all the changes made, point out untracked files.

git status

Git Add

The git add command adds new or changed files in your working directory to the Git staging area.

git add is an important command - without it, no git commit would ever do anything.

git add <file_name>or<path>

To add all the changes

git add .

Git Commit

Git commit is used to record all the changes in the repository. The git commit will commit all the changes and make a commit-id for the same for tracking down the changes made as shown in the image below.

As shown in the image, the command git commit creates a commit-id to track down changes and commits all the changes to the git repository.

Command:

git commit
git commit -m

The -m along with the command lets us write the commit message on the command line.

Command:

git commit -m "Commit message"
git commit -am

The -am along with the command is to write the commit message on the command line for already staged files.

Command:

git commit -am "Commit message"
git commit -amend

The amend is used to edit the last commit. In case we need to change the last committed message, this command can be used.

Command:

git commit -amend

rm stands for remove. It is used to remove a collection of files. The git rm command is used to remove or delete files from the working tree and index.

Command:

git rm <file_name>

Now, if you use the command git status, it would show, that the file has been deleted.

Git Branch

A branch in Git is used to keep your changes until they are ready.
You can do your work on a branch while the main branch(main) remains stable. After you are done with your work, you can merge it to the main branch.
For creating a new branch, the following command is used :

git branch <branch_name>

For switch from one branch to another.

git checkout <branch_name>

Create a local branch and switch to that branch:

git checkout -b <branch_name>

Push in the local branch:

git push -u origin <branch_name>

Note: origin is the default name of remote repository

Now, if someone wants to fetch some information, one can simply run:

git fetch

Delete Branches

Once the work is done on a branch and merged with the Main branch, one can delete the branch.

The following command is used to delete branches:

git delete -d <branch_name>

Note: This command deletes a copy of the branch, but the original branch can still exist in remote repositories.

To delete remote branches, use the following command:

git push origin --delete <branch_name>

Git Checkout

The Git checkout is used to command Git on which branch changes have to be made. Checkout is simply used to change branches in repositories. It can also be used to restore files.

git checkout <branch_name>

Git Checkout Branch To checkout or create a branch, the following command can be used:

git checkout -b  <branch_name>

This will simply switch to the new branch branch_name.

Git Checkout Tag While working on a large codebase, it because easy to have some reference point. That is where the checkout tag is used.

The following command is used to specify the tagname as well as the branch that will be checked out.

git checkout tag</tag> <branch_name>

Git Merge

Git merge is a command that allows you to merge branches from Git. It preserves the complete history and chronological order and maintains the context of the branch.

The following image demonstrates how we can create different features by branching from the main branch and how we can merge the newly created features after the final review to the main branch.

The command git merge is used to merge the branches.

Command :

git merge <branch_name>

GIT Rebase

Git Rebase is a process of combining a sequence of commits to a new base commit.

The primary reason for rebasing is to maintain a linear project history.
When you rebase, you ‘unplug’ a branch and ‘replug’ it on the tip of another branch(usually main).
The goal of rebasing is to take all the commits from a feature branch and put it on the main branch.

The following rebase command is used for rebasing the commits:

git rebase <branch_name>

Git Push

The git push command is used to upload local repository content to a remote repository. Pushing is how you transfer commits from your local repository to a remote repo.

git push <remote> <branch>

Push the specified branch to , along with all of the necessary commits and internal objects. This creates a local branch in the destination repository. To prevent you from overwriting commits, Git won’t let you push when it results in a non-fast-forward merge in the destination repository.

git push <remote> --force

Same as the above command, but force the push even if it results in a non-fast-forward merge. Do not use the --force flag unless you’re absolutely sure you know what you’re doing.

git push <remote> --all

Push all of your local branches to the specified remote.

git push <remote> --tags

Tags are not automatically pushed when you push a branch or use the --all option. The --tags flag sends all of your local tags to the remote repository.

Git Fetch

Git Fetch only downloads the latest changes into the local repository. It downloads fresh changes that other developers have pushed to the remote repository since the last fetch and allows you to review and merge manually at a later time using Git Merge. As it doesn’t change the working directory or the staging area, it is safe to use.

The below illustration shows the working of the command git fetch. It fetches all the latest changes that have been made in the remote repository and lets us make changes accordingly.

The command used is :

git fetch <branch_name>

Git Pull Remote Branch

You can pull in any changes that have been made from your forked remote repository to the local repository.

Using the git pull command, all the changes and content can be fetched from the remote repository and can be immediately updated in the local repository to match the content.

We can simply pull a remote repository by using the git pull command. The syntax is as follows:

git pull

This command is equivalent to

git fetch origin head

Use the following command to check if there has been any change:

git pull <RemoteName> <BranchName>

If there is no change, it will show “Already up to date”. Else, it will simply merge those changes in the local repository.

Git Stash

Sometimes in large codebases, there might be some cases when we do not want to commit our code, but at the same time don’t want to lose the unfinished code. This is where git stash comes into play. The git stash command is used to record the current state of the working directory and index in a stash.

It stores the unfinished code in a stash and cleans the current branch from any uncommitted changes. Now, we can work on a clean working directory.

If in the future, we again need to visit that code, we can simply use the stash and apply those changes back to the working repository.

As shown below, using the command git stash, we can temporarily stash the changes we have made on the working copy and can work on something else. Later, when needed, we can git stash pop and again start working on it.

How to stash changes in Git?

The syntax for stashing is as follows:

git stash

Suppose, you are working on a website and the code is stored in a repository.

Now let's say, you have some files named design.css and design.js. Now you want to stash these files so that you can again use them later, while you work on something else.

Therefore, later you can use the git stash list command to view all the changes.

Drop Stash

In case, you no longer require a stash, you can delete it with the following command:

git stash drop <stash_id>

If you want to delete all the stashes, simply use:

git stash clear

Git-Ignore

At times, there are some files that we might want Git to ignore while commiting. For example, private files or folders containing passwords, APIs etc. These files are user-specific and hence, we can ignore these using the .gitignore.

.gitignore is generated automatically inside the project directory and ignores the files to get committed to the repositories.

How to use the .gitignore?

Follow the below steps to use add the files you want Git to ignore.

Open your project directory on your PC.
Create a .gitignore file inside it.
Inside the .gitignore write the names of all the files you want Git to ignore.
Now add the .gitignore in your repository.

Now, if you check the status of your repo, you will see, all the files which were written in the .gitignore file have been ignored.

Advanced Git Concepts

git pull --rebase

Git rebase is used to rewrite commits from one branch to another branch. In order to combine unpublished local changes with the published remote changes, git pull is performed.

With git pull --rebase, the unpublished changes will be again applied on the published changes and no new commit will be added to history.

git merge --squash

The squash along with git merge produces the working tree. It indexes in the same way as that of the real merge but discards the merge history.

Command:

git merge --squash origin/main

When to use git merge --squash?

When you have merged main into your branch and resolved conflicts.
When you need to overwrite the original commits.

git reflog

The reflog records every change that is made in the repository. Apart from this, if some branch is lost from the repo, the recovery can be done using this command.

git revert

Revert simply means to undo the changes. Therefore, it is an undo command in Git. Unlike traditional undo operation, the revert command does not delete any data. git revert is a commit operation, as it undo the specified commit.

Options:

This option is used to revert back a commit.

git revert <commit_id>

Edit commit message before reverting commit: In case, we want to edit the commit message before reverting, -e is used for the same.

git revert -e <commit_id>

git bisect

Git bisect is a git tool used for debugging. Suppose, you have a large codebase and some commit causes a bug, but you are not sure of which commit causes it.

Git bisect goes through all the previous commits and uses binary search to find the bugged commit.

It is applied as follows:

git bisect start - Starts the bisect
git bisect good v1.0 - Mention the last working commit.
git bisect bad- Mentioning that the current commit has a bug. It will return the commit which causes the bug and one can debug the issue efficiently.

git blame

git blame is used to know who/which commit is responsible for the latest changes in the repository. The author/commit of each line is visible through this.

git blame <file_name>

This command shows the commits which are responsible for changes of all lines of code.

git cherry-pick

Choosing a commit from one branch and applying it to another is known as cherry picking in Git. Following are the steps to cherry pick a commit:

Visit the branch you want to apply to commit and use the following command:

 git switch master

Run the following command:

git cherry-pick <commit_id>

Git Submodules

Submodules are a tool that allows attaching an external repository inside another repository at a specific path. It allows us to keep a git repository as a subdirectory of another git repository.

Commands:

Add git submodule: This takes the git URL as the parameter and clones the pointer repo as a submodule. The syntax to add git submodule is:

git submodule add <URL_link>

git submodule init git submodule init is to copy the mapping from .gitmodules file into ./.git/config file. git submodule init has extend behavior in which it accepts a list of explicit module names.

This enables a workflow of activating only specific submodules that are needed for work on the repository.

Command:

git submodule init

Git Subtrees

git subtree lets you nest one repository inside another as a sub-directory. It is one of several ways Git projects can manage project dependencies.

git-subtree is a wrapper shell script to facilitate a more natural syntax. This is actually still a part of contrib and not fully integrated into git with the usual man pages.

A subtree is just a subdirectory that can be committed to, branched from, and merged along with your project in any way you want.

Commands:

add: Let’s assume that you have a local repository that you would like to add an external vendor library to. In this case we will add the git-subtree repository as a subdirectory of your already existing git-extensions repository in ~/git-extensions/:

git subtree add --prefix=git-subtree --squash \<Git_repo_link>

pull : It is similar to pull from the repository with added prefix. Command:

git subtree pull --prefix <URL_link>

Git Submodules VS Subtrees

Git Submodules	Git Subtrees
It is a link to a commit ref in another repository	Code is merged in the outer repository’s history
Requires the submodule to be accessible in a server (like GitHub)	Git subtree is decentralised, which basically means that its components are shared across a bunch of linked computers.
Git submodule is a better fit for component-based development, where your main project depends on a fixed version of another component (repo).	Git subtree is more like a system-based development, where your all repo contains everything at once, and you can modify any part.
Suitable for smaller repository size	Suitable for bigger repository size

Git Commands

S. No	Command Name	Use
1	git init	Initialise a local Git Repository
2	git add.	Add one or more files to the staging area
3	git commit -m “Commit Message”	Commit changes to the head but not to the remote repository.
4	git status	Check the status of your current repository and list the files you have changed.
5	git log	Provides a list of all commits made on a branch
6	git diff	View the changes you have made to the file
7	git push origin	Push the branch to the remote repository so that others can use it.
8	git config --global user.name “Name”	Tell Git who you are by configuring the author name
9	git config --global user.email user@email.com	Tell Git who you are by configuring the author email id.
10	git clone	Creates a Git repository copy from a remote source
11	git remote add origin	Connect your local repository to the remote server and add the server to be able to push it.
12	git branch	Create a new branch
13	git checkout	Switch from one branch to another
14	git merge	Merge the branch into the active branch
15	git rebase	Reapply commits on top of another base tip
16	git checkout -b	Creates a new branch and switch to it
17	git stash	Stash changes into a dirty working directory
18	git pull	Update local repository to the newest commit
19	git revert	Revert commit changes
20	git clean -n	Shows which files would be removed from working directory. Use the -f flag in place of the -n flag to execute the clean.
21	git log --summary	View changes (detailed)
22	git diff HEAD	Show difference between working directory and last commit.
23	git log --oneline	View changes (briefly)
24	git reflog	Show a log of changes to the local repository’s HEAD. Add --relative-date flag to show date info or --all to show all refs.
25	git rebase -i	Interactively rebase current branch onto . Launches editor to enter commands for how each commit will be transferred to the new base.
26	git restore --staged	Resetting a staged file
27	git rm -r [File_name]	Remove a file (or folder)
28	git config --list	List all variables set in config file, along with their values
29	git branch -d	Delete local branch in Git
30	git push -d	Delete remote branch in Git
31	git stash pop	Unstash the changes
32	git commit -am	The -am along with the command is to write the commit message on the command line for already staged files.
33	git commit -amend	The amend is used to edit the last commit. Incase we need to change the last committed message, this command can be used.
34	git rm	The git rm command is used to remove or delete files from working tree and index.
35	git pull --rebase	Git rebase is used to rewrite commits from one branch to another branch.
36	git merge --squash	The squash along with git merge produces the working tree. It indexes in the same way as that of the real merge, but discards the merge history.
37	git revert -e	edit the commit mesage before reverting, -e is used for the same.
38	git bisect	Git bisect goes through all the previous commit and uses binary search to find the bugged commit.
39	git blame	git blame is used to know who/which commit is responsible for the lastest changes in the repository.
40	git cherry-pick	Choosing a commit from one branch and applying it to another is known as cherry picking in Git.

AWS Elastic Load Balancer for Serverless Architecture

Vinoth Mohan — Wed, 02 Feb 2022 07:54:31 +0000

Load balancing is a significant part of every internet-facing software, and with Elastic Load Balancing (ELB), AWS offers a set of load balancers for every use case.

What is Elastic Load Balancer (ELB)?

ELB is a set of load balancing(LB) services offered by AWS. They include Classic Load Balancer, Gateway Load Balancer, Network Load Balancer, and Application Load Balancer.

Each of these LBs covers different use-cases.

The Classic Load Balancer is a good choice for EC2 based architectures
The Gateway Load Balancer helps with third-party VMs in VPCs
The Network Load Balancer focuses on high-performance low-level networking, think UPD based connections for games or IoT
The Application Load Balancer is a high-level solution for software that uses the HTTP protocol

In the case of serverless architectures, all services use HTTP APIs, which means the ALB is the best choice. So, this article will focus on the ALB.

What is Application Load Balancer (ALB)?

The ALB’s focus on HTTP allows it to use parts of the protocol to make decisions about caching and save you some Lambda executions. This means your Lambda functions have to set their caching headers correctly.

Pricing

While ALB can integrate with Lambda, ALB isn’t a serverless service; it has no pay-as-you-go model, which means you pay for times that aren’t used. But if you have a service with continuous steady traffic requirements, it can be cheaper than API Gateway in the long run.

Limits

Also, API Gateway has a limit on 10k connections; the ALB doesn’t. It’s an API Gateway with more minor features; bare-bones, but built for performance. If you’re going big, ALB might be your only solution.

Permissions

ALB is more of a traditional “strap in front of your public HTTP endpoint” kind of thing. So, while it integrates with Lambda, it doesn’t offer permissions based on IAM. You have to take care of this inside your serverless functions.

Transformations

This traditional load balancing approach also means ALB can’t do request and response transforms; it just pipes your data along. Again, this makes the ALB less flexible than the API Gateway and shifts more work to Lambda.

Multi-Region

You deploy the ALB to one region at a time. Again, this isn’t a serverless service, so more work on your side is required. To get your traffic balanced between multiple regions, you need Route53’s DNS-based balancing.

Configurations for Reliability or Costs

Using ALB with a Lambda target usually delivers good reliability because Lambda scales automatically. If you need more than out-of-the-box reliability, you must deploy ALB to multiple regions and put it behind Route53.

In terms of costs, Lambda can become your main offender. If you route every request to a Lambda function with a big memory config, things can get expensive quickly. So, follow the serverless best practice of keeping Lambda functions small and purpose-driven. Set up conditions for your ALB listeners, so you can use functions with a smaller memory footprint when possible.

Health Check Best Practices

While an EC2 target can easily get overwhelmed, a Lambda target has a bit more buffer because of its inherent autoscaling; there are still things that can go wrong in a serverless system.

AWS disables ALB health checks by default for Lambda targets, so you have to opt-in here.

While some issues can arise from buggy code pushed to Lambda, most problems come from upstream services your function uses. So set up your Lambdas to pipe the health check and later respond with the result from the upstream service.

If things are broken, the only quick solution is to tell Route53 to route the following requests to a different deployment.

Log Analysis in AWS

AWS lets you use Amazon Athena to analyze your ALB logs. Athena is a serverless query service. You need to activate query logs and save them to S3 to explore them with Athenas SQL queries.