DEV Community: Rahul Kumar

nest.js + TypeORM + PostgreSQL

Rahul Kumar — Tue, 08 Jun 2021 04:30:53 +0000

Agenda :

Building a rest api using Nest.JS .

Backend Architecture :

Technologies used:

Node.js --- Platform .
NestJS --- server .
TypeORM --- orm .
PostgreSQL --- database .

What is Node.js :

NodeJS is a Javascript runtime(contains everything to run javascript) for building server side applications .

What is NestJs :

Nest is a framework for building efficient and scalable NodeJs server side applications .

What is TypeORM :

TypeORM is an object relational mapper which task is to basically generates objects using object oriented programming which maps to the database virtually .

What is PostgreSQL :

PostgreSQL is an object relational database management system for building scalable and high availability applications .

Install @nestjs/cli package .and create a new project

$ npm i -g @nestjs/cli
$ nest new project-name

In a common nestJS project :

main.ts contains the bootstraping code .
.spec file contains the testing files .
nestjs usages module to organize the application structure .
controller contains the routing rules for the application .
service contains the business logic for the application .

$ npm run start

curl localhost:3000

Set up Database :

$sudo -U postgres
$psql
$create database conduit
$create user conduit with encrypted password conduit
$grant all privileges on database conduit to conduit

File structure :

src
|- app.controller.ts
|- app.controller.spec.ts
|- app.module.ts
|- app.service.ts
|- main.ts

Create a `connection` to the database .

$ touch app.dbconfig.ts

import {TypeOrmModuleOptions} from "typeorm";

export function createTypeOrmProdConfig(): TypeOrmModuleOptions{
 return({
    type: "postgres",
    username: "conduit",
    password: "conduit",
    database: "conduit"
    entities: [join(__dirname, '**', '*.entity.{ts, js}')],
    synchronize: true,
    logging: true,
    logger: "advanced-console",
});
}

Next we have to create the entities for our Database . And the individual entity to the entities array in the createConnection function .

$ mkdir src/entities

$cd entities

$nano Article.ts

@Entity()
export class Article {

    @PrimaryColumn({length: 40})
    slug: string

    @Column({length: 40})
    title?: string

    @Column({length: 100, nullable:true})
    description: string

    @Column({type: 'text'})
    body: string

Generate module, service, controller for the article route :

nest g mo article module
nest g co article module
nest g s article module

$ cd module/article

$ nano module.article.ts

imports : [Typeormmodue.forFeature([Article])]

navigate to the article service file .

@InjectRepository(Article) private readonly articleRepo: Repository<Article>,

async getAllArticles(): Promise<Article[]> {
        return await this.articleRepo.find();
    }

navigate to the article controller file .

constructor(private readonly articlesService: ArticlesService) {}
@Get()
    async getAllArticles(): Promise<Article[]> {
        return await this.articlesService.getAllArticles();
    }

Then finally in the application root module .

imports :[TypeOrmModule.forroot([Article])]

$npm start

Thank You For Reading 🔥

How to build and run Kubernetes locally .

Rahul Kumar — Mon, 07 Jun 2021 17:00:07 +0000

Install GOLang (latest version):

sudo wget https://golang.org/dl/go1.16.5.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.16.5.linux-amd64.tar.gz

Setup the environment via Bash :

echo 'export PATH=${PATH}:/usr/local/go/bin' >> ~/.profile
echo 'export GOPATH_K8S=${HOME}/go/src/k8s.io/kubernetes' >> ~/.profile
echo 'export PATH=${GOPATH_K8S}/third_party/etcd:${PATH}' >> ~/.profile
source ~/.bashrc

MAKE SURE YOU HAVE SET THE PATH FOR BOTH GO AND ETCD IN THE SECURE_PATH IN THE /etc/sudoers FILE

MAKE SURE TO EDIT IT AS VISUDO

clone kubernetes

mkdir -p {GOPATH_K8S}
git clone https://github.com/kubernetes/kubernetes ${GOPATH_K8S}
git remote rename origin upstream

install etcd

$ hack/install-etcd.sh

start docker demon

systemctl enable docker
systemctl start  docker

Build k8s

This step takes 15 minitues
cd ${GOPATH_K8S}
git checkout v1.12.3 (checkout the stable release version)
time make quick-release

Run a Local k8s cluster

$ {GOPATH_K8S}/hack/local-up-cluster.sh

verify the cluster is running in another window

export KUBERNETES_PROVIDER=local
export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
${GOPATH_K8S}/cluster/kubectl.sh get nodes

Fork the k8s repo on github

navigate to 
click fork button

setup gihub user name

export GITHUB_USER=[Insert your github user name]
echo 'export GITHUB_USER=${GITHUB_USER}' >> ~/.bash_profile

Change the k8s source code

Navigate to the directory your clone

cd ${GOPATH_K8S}
git remote add origin https://github.com/${GITHUB_USER}/kubernetes
git fetch origin

These instructions make the folllowing assumptions:

-Your upstream remote going to
http://
-Your origin remote
http://

you can double check by runnig

$ git remote -v

Create a new branch before making a change

branch should have some meaning means what you have added to the code

git checkout upstream/master
git checkout -b my-brnach-of-k8s

If you would like to make change we recommend finding some untested code and adding a few unit tests. If you don't know how to write unit tests in go read this

GOLang_Testing

Example new test

File: pkg/util/normalizer/normalizer_test.go

Run Kubernetes Locally

$ make test WHAT-k8s.io/kubernetes/pkg/util/normalizer

Contribute a pull request

Commit your changes and push to your fork

gofmt -v [insert path to changed file]
gofmt -v pkg/util/normalizer/normalier_test.go
git add .
git config --global user.name ""
git config --global user.email ""
git commit -m "commit msg"
git push -u origin/my-branch-of-k8s

create a pull request via GiHub UI

Navigate to https://github.com/kubernetes/kubernetes page
click on the new pull request
click on compare across forks link
select your fork and branch from the two dropdowns on the right
click on create pull request
Note that an existing maintainer must comment with the /ok-to-test comment the automated test for new contributer

If you have any doubt please ASK .
Thank you for reading 🔥

Building and compiling Linux Kernel :

Rahul Kumar — Sat, 23 Jan 2021 10:57:38 +0000

What is Linux Kernel ?

As we know Kernel is the core of any operating system which is the interface between computer's hardware and it's processes .
And the main task of the kernel is :

1. Memory management
2. Process management
3. System calls
4. Device drivers
5. Security

Linux Kernel is a free opensource monolithic, modular and multitasking Kernel for Unix like operating systems .
Though it is opensource we can install and manipulate as per our need .

Dependencies :

Make sure to install these dependencies :

sudo apt-get install libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf

How to installation :

Navigate to kernel.org and download the latest version .
Here I am using Ubuntu as the operating system .
Install it with wget .

$sudo wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.9.tar.xz

Unzip the downloaded file .

$ sudo tar -xvf linux-5.10.9.tar.xz

Make a directory in your home folder .

$ mkdir kernel

Copy the file after extraction .

$ cp -r linux-5.10.9/ /kernel

Navigate to :

$cd kernel/linux-5.10.9/

Start the Linux source configuration tool

make menuconfig

Edit the configuration of kernel and save-and-exit.
Compile the kernel .

$make -j 5 KDEB_PKGVERSION=1.0 deb-pkg

Install the Kernel .

sudo dpkg -i ../linux*.deb

Reboot your system and check for the kernel version .

Thank You 🤖🤖🤖

Set up MySQL with Sequelize in NodeJS :

Rahul Kumar — Tue, 15 Dec 2020 11:13:08 +0000

First of all we have to create a User of the MySQL database .

create database devdb;

create user dev identified by 'devpass';

use devdb;

grant all privileges on devdb.* to dev;

Then we have to install sequelize as an orm(object relational mapper ) and mysql2 as the database into our project .

$ npm i mysql2 sequelize 

$ touch db.js

connecting to a database :

const sequilize = require('sequilize')

const db = new Sequilize(database, user, password{

  host : 'localhost',
  dilect : mysql,
  pool: {
  min : 0,
  max : 5,
  }
});

Create our Model :

const Post = db.define('Post', {
  id: {
  type: Sequilize.INTEGER,
  autoIncrement: true,
  primaryKey: true
  },
  name:{
  type: Sequilize.STRING,
  allowNull: false
  },
});

db.sync()
  .then(() => {
  console.log('Database has been created');
})
.catch((err) => {
  console.log('error creating database '+ err);
})

exports = module.exports = {
 User 
}

node db.js

Thank You For Reading

Set up MongoDB with node.js :

Rahul Kumar — Tue, 15 Dec 2020 02:36:19 +0000

What is MongoDB :

It is a general purpose document based NoSQL database . Build for scalable application .

What is Mongoose :

mongoose is a object document mapper (ODM) used for storing and managing data in form of objects .
Which makes our code more simpler to write than the MongoDB native driver .

What is mongod:

Mongod is the demon for MongoDB which performs background management operations and actively listening for the incoming request on port 2701.

What is mongo shell :

This is the command line utility for performing CRUD operations on command line .

What is NodeJS :

Node.js is a backend framework for building backend web application .

npm i mongoose

add this URL for the database connection

mongoose.connect("mongodb://localhost:27017/(name of your db)", {useNewUrlParser:true, useUnifiedTopology: true});

Create the schema of our db

const newSchema = mongoose.Schema({
   name : String,
   rating : Int
});

Create a new mongoose model the singular version of our db .

const Post = new mongoose.model('Post', newSchema);

Create a new Document :

const Post_a = new Post({
name : "Dev post",
rating : 7
});

save the new value into the database

Post.save();

If you want to upload data into the cloud then you could upload to MongoDB atlas which is a database as a service .

Post not found or has been removed.

Thank You For Reading 🤖🤖🤖

Connect MongoDB Atlas and upload data :

Rahul Kumar — Mon, 14 Dec 2020 18:56:49 +0000

What is MongoDB atlas :

It is a database as a service where we could upload our data to the popular cloud providers such as AWS , azure, GCP .

setup :

Crate a new project .
Select the free tire and then choose the cloud provider .
Navigate to the create a new user under the security section and then add a new user with password .
and finally we can click on the connect button and then click on connect to our application .
Then integrate that URI in your application with your username and password .

Here i am using node.js with mongoose as an odm(object document mapper)

mongoose.connect('mongodb+srv://admin:<password>@cluster0.jaomd.mongodb.net/<dbname>?retryWrites=true&w=majority', {useNewUrlParser: true});

replace with your username , password and dbname .
Now we can execute CRUD operations on our database .

Cloud computing :

Rahul Kumar — Mon, 14 Dec 2020 11:36:48 +0000

Before cloud computing :

What happened in the deployment phase .

Buy a stack of servers or we could do it on our computer .
Keeping the peak traffic in mind, buy more servers .
Monitoring our infrastructure 24/7 .

Disadvantages of on-site architecture :

This setup is expensive .
Troubleshooting problems can be tedious and may conflict with your business goals .
The main disadvantage is suppose we have to keep a peak traffic and bases upon that we need to configure our hardware and our system should be up and running 24/7 . But when we monitor our application we get to know that our resources are not optimized in the night which is loss of money .

What is cloud computing :

It is the set of remote servers on the internet to store , manage, and process data rather than a local server .

Cloud Model :

Service Model :

Iass (infrastructure as a service) :
You have the complete access of the infrastructure .
Provides virtualized computing resources over the internet .
No worries about the underling physical machine .
Abstract the user from the physical machine .
Pass :
You have a platform where you can deploy your applications and you don't have the access .
Sass :
we have a access only in a form of a software .

Deployment Model :

There are three different ways to deploy cloud services: on a public cloud, private cloud or hybrid cloud. Which deployment method depends on your business needs.

There are mainly three types deployment model :

Public cloud
Private cloud
Hybrid cloud
Multi cloud

Public cloud :

This is the most common type of deployment model where the cloud resources like servers and the storage are completely stores on a third party cloud provider and delivers over the network .

Features :

Lower costs : Because they run on a pay as you go model .
No maintenance : The cloud provider manages and checks the status of the server .
Highly available : Access any time and anywhere with zero downtime .

Private cloud :

This type of model is used by business or organizations which do some confidential work and they want their data to be compromised.
Resources (like servers and storage) are stored on a on-site datacenter .
If it is stored on a third party cloud provider then it is stored on a private network .

Features :

More flexible : organization can customise according to their need .
More secure : Because it is inside a private network .

Hybrid cloud :

It is the combination of public and private cloud which allow our application to move between public and private cloud .
The hybrid cloud is evolving to include edge workloads as well. Edge computing brings the computing power of the cloud to IoT devices closer to where the data resides. By moving workloads to the edge, devices spend less time communicating with the cloud, reducing latency and they are even able to operate reliably in extended offline periods.
A hybrid cloud platform gives organizations many advantages—such as greater flexibility, more deployment options, security, compliance and getting more value from their existing infrastructure.

Features :

Control : Organization can maintain a private cloud to store it's sensitive data .
Flexibility : You can take additional resources from the public cloud when you needed .
Cost effective : With the ability you get to scale to public cloud you can maintain a pay as you go model .

Multi Cloud :

As like we have replicas running of the software same way we can configure multiple cloud providers to run the application .
As the name suggest you can configure multiple cloud providers to run the application .
You can check the availability zones and the regions provided by cloud providers .
If you want to setup a edge server in multiple counties then go ahead with which cloud has the maximum reach in that country .

Features :

Avoid Single point of failure :
Running multiple replicas in multiple instances in multiple regions in multiple cloud providers of your application can save you from single point of failure .
High performance :
Planning to set up edge servers then go ahead with multi cloud .

Thanks for Reading 🤖🤖🤖

Decoding the beauty of cryptography :

Rahul Kumar — Mon, 14 Dec 2020 08:56:40 +0000

What is cryptography :

Cryptography is a practice and study of techniques for securing communication between two hosts . Which ensures that no middle person can understand the communication going on between two hosts .

Process of cryptography :

History of encryption :

The Caesar cipher is one of the earliest known and simplest ciphers . It is a type of substitution cipher in which each letter in the plaintext is 'shifted' a certain number of places down the alphabet .
Enigma cipher :
The enigma cipher is a field cipher used by Germans during the world war 2.

---------------------------------------
Encryption : key values a = 17 , b = 20
---------------------------------------

original text : t  w  e n  t  y     f i f t  e  e  n
            X : 19 22 4 13 19 24    5 6 5 19 4  4  13
ax + b % 26   : 5  4  10 7 5  12    1 0 1 5  10 10 7
Encrypted text: f  e  k  h f  m     b a b f  k  k  h

------------------------- 
Decryption : a^n - 1 = 23
-------------------------

Encrypted Text      : f  e  k  h f  m     b a b f  k  k  h
Encrypted Value     : 5  4  10 7 5  12    1 0 1 5  10 10 7
23 * (X - b) mod 26 : 19 22 4 13 19 24    5 6 5 19 4  4  13
Decrypted Text      : t  w  e n  t  y     f i f t  e  e  n

Digital encryption standards : The digital encryption standard (DES) is a symmetric key block cipher published by NIST . DES is an implementation of Feistel cipher .

Classification of cryptography :

Symmetric key cryptography (secret key encryption)
Asymmetric key cryptography (public key encryption)

Symmetric key cryptography is further divided into two parts

A. classical Cryptography :

Transposition cipher
substitution cipher

B. Modern Cryptography :

Stream cipher
Block cipher

Symmetric key cryptography :
An encryption system in which the sender and receiver of a message share a single common key that is used to encrypt and decrypt the message . The most popular symmetric key system is the data encryption standards(DES) .

Classical cryptography :

Transposition cipher : In this kind of encryption the positions held by units of plaintext (which are commonly characters or group of characters) are shifted according to the regular system so that the ciphertext constitutes a permutation of the plain text .

1 2 3 4 5 6    4 2 1 6 3 5
m e e t m e    t e m e e m
a f t e r p    e f a p t r
a r t y        y r a t

plain text  : meet me after party 
key used    : 421635
cipher text : temeemefaptryrat

substitution cipher : Methods of encryption by which units of plain text area replaced with cipher text , according to a fixed system the "units" may be single letters, mixtures of the above , and so forth .

A B C D E F G H I J K L M     RTO13 is a caesar cipher , a type of 
| | | | | | | | | | | | |     substitution cipher in RTO13 
N O P Q R S T U V W X Y Z     alphabet is rotated 13 times .

plain text : ABCDEFGHIJKLMONPQRSTUVWXYZ
keyword    : ZEBRAS
cipher text: ZEBRASCDFGHUKLMNOPQTUVWXY

Modern Cryptography :

Stream cipher :
A symmetric or secret-key encryption algorithm that encrypt a single bit at a time . with a stream cipher , the same plaintext bit or byte will encrypt to a different bit or byte every time it is encrypted .
Block cipher :
The alternative method is block cipher in which a key and algorithm are applied to block of data rather than individual bits in a stream .

Asymmetric key encryption :

It usages a pair of keys public key which can be disseminated widely over the network and private key is only know to the host .

Some examples of Asymmetric key :

Diffie-Hellman key exchange
DSS
RSA

Diffie-Hellman key exchange
It is also known as exponential key exchange is a method of digital encryption that uses numbers raised to specific powers to produce decryption keys on the basis of components that are never directly transmitted , making the task of a would-be code breaker mathematically overwhelming .

Working of Diffie-Hellman :

step-1 :

Assume a prime number and selecting its primitive root .
Assume a prime number q .
select alpha primitive root of q (alpha < q)
alpha is a primitive root of q if :

alpha mod q , aplpha ^ 2 mod q .......... alpha ^ q-1 mod q
is implies 1, 2, 3, ..... q-1

step-2 :
Finding private key and public key of sender and receiver .

Assume xa (private key of user a)   |
----------                          |
| xa < q |                          |
----------                          |    (Sender)
Now , calculate public key ya       |
------------------------            |
|ya = alpha ^ xa mod q |            |
------------------------
Assume xb (private key of user a)   |
----------                          |
| xb < q |
----------
                                    |    (receiver)
Now , calculate public key , yb
----------------------
| yb = alpha^xb mod q |             |
-----------------------
                                    |
__Step-3 :__
Generate secret key at senders and receiver's side so sender has his private key         = xa
    receiver public key = yb
    prime number        = q

key is being generated with the following formula :
------------------
| K = yb ^ mod q |
------------------
similarly receiver has his private key = xb
                           receiver public key = ya
                           prime number = q 
* key is being generated :
-------------------
| K = ya ^ xb mod q |
--------------------
* After calculating the key on both the sides 
K = yb ^ xb mod q   ==  K = yb ^ xa mod q

There are lot more to know :

prefer old friend wikipedia for in-depth knowledge .
And cryptii for practice .

prefer site for practice

Thanks for reading 👨‍💻👨‍💻👨‍💻

Let's build a DevOps project :

Rahul Kumar — Mon, 14 Dec 2020 08:29:15 +0000

The Agenda :

This is a project in which we have to take an application through DevOps lifecycle .

Integrating the source code in a version control environment for mutual development.
The application must pass through a ci/cd pipeline .
continuous monitoring is enabled for fault tolerance .

Phases in DevOps :

Planning
Coding
Testing
Building
Release
Deploy
Operate
Monitor

Planning :

A better planning of an application can lead to a better growth of our application where we have to design our system where high level system design is involved like flow charts, features , intended users , security , Relationship in the database .

Coding :

Then we have to do the coding according to the features . where low level design is involved .

Testing :

A simple unit testing is good enough for our application if we are not building a high scale application . Since we are using DevOps so we using Selenium to automate application testing . we use the selenium web driver and TestNG for report writing and Maven for managing our dependency .

Building :

In the building stage we are going to implement packaging , error testing, basically integrate our application with Jenkins which is a ci/cd pipeline tool which can trigger our code by configuring with webhook and then automating the testing using selenium .
So there are mainly two types of servers one is for testing and another is production .
So our application first should go to the testing servers and then to the production servers .

Release :

In the release state we are releasing a MVP of our application to our intendent uses .

Configuration management :

This is a phase of infrastructure as a service where we have to write code for our infrastructure . and to automate it because we are using DevOps and this can be achieved by Configuration management tools such as ansible, puppet, Ceph , SaltStack . Let's suppose our application needed more resources so that we have to allocation new servers and manually we could do that but if we think in a large scale so it is a repeatative task so by using configuration management tools we could make sure that automated servers added to our infrastructure when ever needed .

Monitor :

Monitoring our application is really a essence like monitoring our infrastructure . Monitoring based upon different metrices
like resource utilization (Mainly CPU and Memory) .
When it find potential failure symptoms an Alert has been triggered with the log about which part of our application crashes .

Steps :

We are use two machines one is the worker and another one is the slave machine .
create two ec2 instances in AWS and connect using SSH .
Install Git and Jenkins(Java as a prerequisite) on the master Node .
Fire up Jenkins web dashboard in the usual steps .

configure setting where we have to create a new node and then configure our node machine by downloading two files and if we are downloading them we have to use the SCP command to copy them to the master node .

$scp -i /downloads/.pem (name of the files) ubuntu@ip:/home/jenkins

Configure webhooks for a triggering point whenever we are pushing our code into GitHub .
Coming to the node machine we need to install Java and git .
clone the git repos modify some change .
add it to the staging area (place where git can track all the files ) .

$git add .

$sudo git commit -m "Initial commit"

Then copy the ip address of Jenkins go to the settings of your repository add a webhook .

recognized as a payload URL

http://testingserverip:8080/github-webhook/

Implement a Job inside Jenkins
create a new item test-job

Create a docker file

The container is consists of the operating system and the required software .
It our case in this container Ubuntu is the OS and apache is configured as a webserver .

FROM hshar/webapp
ADD . /var/www/html

In the Jenkins build environment

sudo docker rm -f $(sudo docker ps -a -q)
sudo docker build /home/ubuntu/website/. -t test
sudo docker run -it -p 82:80 -d test

Now we have to test our application :

Install selenium webdriver , the singleton server and the chromedriver as per your operating system .

Install in on test-server

$sudo apt install chromium-browser

Go to this link !

$wget https://chromedriver.storage.googleapis.com/87.0.4280.20/chromedriver_linux64.zip

$unzip chromedriver_linux64.zip

Lets write a simple
/home/ubuntu/chromedriver as a path
execute it on headless mode because our ubuntu server does not have a dashboard .

public class Test {

   public static void main(String []args) {

       System.setProperty("webdriver.chrome.driver", "C:\\Selinium\\chromedriver.exe");

       ChromeOptions options = new ChromeOptions();

       options.addArguments("--headless");
       WebDriver driver = new ChromeDriver(options);

       driver.get("http://(ip address)/");

       driver.findElement(By.id("id_name"));
   }
}

java -jar hello.jar

Then transfer it to the node machine scp

Lunch a production server instance

$sudo apt install open-jdk

$sudo apt install docker

Go to jenkins
add a new node called prod server and the location is /home/ubuntu/jenkins
run the jnlp file

next create a new job as a prod branch without webhook

echo hello
sudo docker rm -f $(sudo docker -ps -q)
sudo docker build /home/ubuntu/jenkins/workspace/test-job/. -t prod
sudo docker run -it -p 80:80 -d prod

cd jenkins 
cd workspace 
cd test-job

create a new job called test-job
same github url
label expression test-server

sudo cp ./hello.jar /home/ubuntu
sudo java -jar /home/ubuntu/hello.jar

test-github-webhook
post-build-action build-website
save
build-website
remove build triggers
add test-website post-build
test-website
build-other-project -- test-job
add a new pipeline view
define the initial job other will be handeled by post-build action
test-github-webhook
Inside the test-server

sudo mv ./hello.jar /home/ubuntu/website
cd website
git add . 
git commit -m "test jar added"
sudo git push -u origin master

Lets go to master server for configuration management
cd jenkins/workspace/test-job
The ip address of production server paste in on browser .
Modify it on test server

cd website
change something
commit -m "Final commit"
sudo push -u origin master

New job named as Develop-branch
restrict it to the test-server

sudo docker rm -f $(sudo docker ps -t -q)
sudo docker build /home/ubuntu/. -t test1
sudo docker run -it -p 85:80 -d test1

Configuration management using ansible:

In this phase we have to install a Configuration management which is ansible in our case . which helps achieve infrastructure as a code basically writing code for our infrastructure .

Continuous monitoring using Nagios :

The Final step in our loop and most important phase .

Ansible setup and architecture

Nagios setup and architecture

Thank You for reading 🤖🤖🤖

configure Nagios in our server :

Rahul Kumar — Mon, 14 Dec 2020 07:35:52 +0000

Why we need Continuous Monitoring :

In a large scale distributed applications we never know what kind issue we may ran into .
some of them are :
Failure in CI/CD pipeline .
Static code analysis .
Application side issue .
Infrastructure side issue .

And in big organizations server health and application Security can not be compromised at any cost .

What is continuous monitoring :

Continuous monitoring is a phase in our application development lifecycle which make sure that our application is getting continuously monitored and debugged if any issue .

Phases of continuous monitoring :

Define
Establish
Implement
Analyze
Respond
Review and Update

What is Nagios :

Nagios monitors your entire IT lifecycle to ensures systems, applications , services and business process are functioning properly .

Features of Nagios :

Using a master slave architecture .
Easy to write plugins .
Good web interface for visualization purpose .
Automate the e-mail sending in case of any network failure .

Architecture of Nagios :

How install Nagios (Ubuntu):

$ sudo apt update

$ sudo apt install build-essentials libgd-dev openssl libssl-dev unzip apache2 gcc

Nagios user and group are used to run Nagios process

$ sudo useradd nagios

$ sudo groupadd nagcmd

$ sudo usermod -a -G nagcmd nagios

$ wget https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.4.6.tar.gz

$ tar xpf nagios-*.tar.gz

$ cd nagios-4.4.6

$ ./configure --with-nagios-group=nagios --with-command-group=nagcmd

You can see the configuration using the configure command :
Now we have to build the project using the make command and then install it :

$ make -j4 all

$ sudo make install

Now run these commands to install Nagios sample files, binary files and it's sample config files .

sudo make install
sudo make install-daemoninit
sudo make install-commandmode
sudo make install-config

We are using apache to serve Nagios web interface .

$ sudo make install-webconf

Enable the Apache rewrite and cgi modules with the a2enmod command:

$ sudo a2enmod rewrite

$ sudo a2enmod cgi

In order to execute commands via the web interface to Nagios add the web server user, www-data, to the nagios group:

$sudo usermod -a -G nagios www-data

Next we have to create a username and password for our user which can enable our user to login to the web interface .

$ sudo apt install htpasswd

$ sudo htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

Enter Your Password :

Warning: If you create a user with a name other than nagiosadmin
you will need to edit /usr/local/nagios/etc/cgi.cfg 
and change all the nagiosadmin references to the user you created.

Restart the Apache server

$ sudo systemctl restart apache2

Installing the Nagios plugin :

$ cd ~
$ http://www.nagios-plugins.org/download/nagios-plugins-2.3.3.tar.gz
$ tar zxf nagios-plugins-2.2.1.tar.gz
$ cd nagios-plugins-2.2.1

*configure their installation 
$ ./configure

* Now build and install the plugin

$ make
$ sudo make install

Now we need to install check_nrpe plugin to monitor the remote plugins :

cd ~
curl -L -O https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-3.2.1/nrpe-3.2.1.tar.gz
tar zxf nrpe-3.2.1.tar.gz
cd nrpe-3.2.1
./configure

Now build and install check_nrpe :

make check_nrpe
sudo make install-plugin

Configure Nagios :

Navigate to the Nagios configuration file

$sudo nano /usr/local/nagios/etc/nagios.cfg

Add this line to the config file

...
#cfg_dir=/usr/local/nagios/etc/servers
...

Uncomment this line

cfg_dir=/usr/local/nagios/etc/servers

Now we have to create a directory so that we can store our configuration file for each server .

$sudo mkdir /usr/local/nagios/etc/servers

open the Nagios contact file :

$ sudo nano /usr/local/nagios/etc/objects/contacts.cfg

Find the email directive and replace it with yours

...
define contact{
        contact_name                    nagiosadmin             ; Short name of user
        use                             generic-contact         ; Inherit default values from generic-contact template (defined above)
        alias                           Nagios Admin            ; Full name of user
        email                           your_email@your_domain.com        ; <<***** CHANGE THIS TO YOUR EMAIL ADDRESS ******
...

Navigate to the directory and add a new command which is for defining check_nrpe command .

$ sudo nano /usr/local/nagios/etc/objects/commands.cfg

...
define command{
        command_name check_nrpe
        command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

This defines the name and specifies the command-line options to execute the plugin.

$ sudo systemctl start nagios

Access the Nagios web interface :

http://nagios_server_public_ip/nagios

Enter the login credentials .

Monitoring :

ssh into the server you want to monitor.

$ ssh ubuntu@server_ip

$ sudo useradd nagios

cd ~
curl -L -O https://nagios-plugins.org/download/nagios-plugins-2.2.1.tar.gz

tar zxf nagios-plugins-2.2.1.tar.gz
cd nagios-plugins-2.2.1

then configure and build it

$ ./configure

$ make

$ sudo make install

Install nrpe daemon find the latest url at github pages:

cd ~
curl -L -O https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-4.0.0/nrpe-4.0.0.tar.gz

tar zxf nrpe-4.0.0.tar.gz
cd nrpe-4.0.0

$ ./configure

Now build and install NRPE and its startup script with these commands:

make nrpe
sudo make install-daemon
sudo make install-config
sudo make install-init

$ sudo nano /usr/local/nagios/etc/nrpe.cfg

server_address: Set to the private IP address of the monitored server.
allowed_hosts: Add the private IP address of your Nagios server to the comma-delimited list.
command[check_hda1]: Change /dev/hda1 to whatever your root filesystem is called.

...
server_address=second_ubuntu_server_private_ip
...
allowed_hosts=127.0.0.1,::1,your_nagios_server_private_ip
...
command[check_vda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/vda1
...

$ sudo systemctl start nrpe.service

$ sudo ufw allow 5666/tcp

Now you can check the communication with the remote NRPE server. Run the following command on the Nagios server:

$ /usr/local/nagios/libexec/check_nrpe -H second_ubuntu_server_ip

On your Nagios server, create a new configuration file for each of the remote hosts that you want to monitor in /usr/local/nagios/etc/servers/. Replace the highlighted word, monitored_server_host_name with the name of your host:

$ sudo nano /usr/local/nagios/etc/servers/your_monitored_server_host_name.cfg

Add the following host definition, replacing the host_name value with your remote hostname, the alias value with a description of the host, and the address value with the private IP address of the remote host:

define host {
        use                             linux-server
        host_name                       your_monitored_server_host_name
        alias                           My client server
        address                         your_monitored_server_private_ip
        max_check_attempts              5
        check_period                    24x7
        notification_interval           30
        notification_period             24x7
}

Add some sevices
First, add this block to monitor load average:

define service {
        use                             generic-service
        host_name                       your_monitored_server_host_name
        service_description             Load average
        check_command                   check_nrpe!check_load
}

Next, add this block to monitor disk usage:

define service {
        use                             generic-service
        host_name                       your_monitored_server_host_name
        service_description             /dev/vda1 free space
        check_command                   check_nrpe!check_vda1
}

Now restart and navigate to the web interface :

$ sudo systemctl restart nagios

Thanks for reading 🤖🤖🤖

AWS multi-tire architecture Project :

Rahul Kumar — Wed, 09 Dec 2020 17:26:08 +0000

As we all know in today's era everybody is moving towards cloud because deploying and maintaining applications in our local environment could be compound and costly .

Here is the architecture diagram below .

Explain :
The architecture comprises of a private network and a public network in which the load balancer resides where we can basically implement a path based routing . The separation of private network is to strength the security of our application where we make sure the integrity of our application .
This is a multi-tire architecture project which comprises of individual business logics individually scalable and monitorable .

Steps :

This project is build on AWS so you must have a AWS account and ssh-client like putty if you are on windows or for mac or Linux use openssh .

First we have to create a VPC :

VPC is an isolated space inside our region More info for click me .

creating a VPC with a name and with a ipv4 CIDR(classes inter-Domain Routing is a method used to allocate IP address) block 10.1.0.0/16 range where the application is going to deploy .

Creating a subnet for the access :

Partitioning the network according to the public and private access .
Let's define 4 regions (You can choose the number according to your need) and later in the configuration part we can decide which one to deploy in the public and in the private network .

In our network all the internet access and non-internet access part is going to deploying in public and private subnet respectively .

So the public subnet is having a public IP address and the private subnet is having a private IP address .

Configuration for our subnet :

Let's have a look at the configuration of four of our subnets .

tag - public-2a
region - 2a
VPC - main network
ipv4 - 10.1.1.0/24 this range is a part of the main network range

tag - public-2b
region - 2b
VPC - main network
ipv4 - 10.1.2.0/24

tag - private-2a
region - 2a
VPC - main network
ipv4 - 10.1.3.0/24

tag - private-2b
region - 2b
VPC - main network
ipv4 - 10.1.4.0/24

Create an Internet gateway :

suppose you want to have a internet access in your home then you contact to the internet service provide to bring up a connection to your home . similarly here we have to create a internet gateway for the same purpose .

steps :

create a new internet gateway .
attach it to your VPC

when you deploy a resource in the cloud it is always deployed in the subparts . just like you have WIFI in the home and you do not have a active connection to that then your laptop does not know weather to connect to the WIFI or not .

Creating route tables :

So we have created 4 subparts of our main network then we just have to define which one to give the internet access and which one to not .

create two one is Active and another one is non-active
Go to the actions -- connect the public subnets to the Active route-table and then the private subnets to the -- non-active route-table .
Make sure you have deployed both of them in your main network (VPC) .

Now we have to specify the route to the gateway .

Steps :
Select the gateway which we want to give the internet access and in our case it is named as Internet .

Edit routes --> Add a route
0.0.0.0/0 internet gateway

we connected two subnets to the route table and they can connect to the internet by using the internet gateway .
All the resources that we are going to communicate with each other because they are the part of the same network .
So consider a scenario where in your home network your laptop and mobile is connected which leads the communication between two .
The subparts can communicate with each other whether they have the internet access or not . Which make sure the communication to our private network . This kind of division ensures the security of our infrastructure .

The resources which are public subnet we have to assign a public IP address to them in order to talk to them .

Just select your public network and then go to actions modify auto assign IP and then tick the enable public assign IP .
steps:
subnets
public-2a
modify auto....
enable auto assign
save

Public IP address is the address which can be reach out to the network .

All the address we have specified earlier are the local IP address.

Lets Move to the application deployment part :

Steps :
Create a new EC2 instance
in the configure instance
network -- main-network
subnet -- public subnet
go with the default firewall as of now and then we have to restrict the firewall later .
Then lunch it .

select the security group
edit inbound rule -- > my IP

Connect it to your machine :

ssh -i pem file and host@ipaddr

Installing the webserver for the access to our webpage .

sudo apt install apache2

$ cd /var/www/html

So this is a sample blogging project . where i have two routes as home and the about route . Obviously same steps can be followed in different real life projects .

create a new index.html page with the content as follows :

<h1> Welcome to the homepage </h1>

Create an AMI of the home-page :

home page is now ready to deploy in the private subnet .

Steps:
And then name it as you wish here i name it as blog-home
Lets again go to the same path where our home page id resides and then change the index.html .

<h1>This is the about page</h1>

Then create an AMI out of it . By following the same steps but here we have to name it as about-page .

Configure both of the images :

We have to give two of our web pages public-2a and public-2b as the subnet . And make sure they reside in our main-network(VPC) .

Steps :
Lunch Instances
My AMIs

Select main-network(VPC)
private-2a (For the home page)
default security group .

And for our about page select the subnet in the private-2b and rest of all are same .

Note :

If you see it not has a public IP and it has a private IP address,
which is only reachable from our public instance . which in our case the proxy-server .

After creating both the AMI's we now can have a look at our next architecture .

As per the diagram we have to define a load-balancer which can indeed talk to our AMIs . As per the path based routing specified routing .

Make sure we can connect to our private networks through the proxy-sever :

Steps:
Copy the pem file which is in our local system to the proxy server .

$ scp -i pemfile ubuntu@publicip:/home/ubuntu

Next connect to the private server(blog-home) through private IP .

$ ssh -i pemfile ubuntu@privateip

If you are trying to connect to one of our machine which is deployed in the private subnet it could not because it is not a part of our main network .

check in both home and about server if the apache server is running or not .

Configure our load balancer to connect to the server .

Steps :
load-balancer
create a load balance
application load balancer (which helps in path based routing)
Configuration :
name --> blog-demo
Protocol --> HTTP
VPC--> main
availability jones public-2a and public-2b
default firewall

Make sure you have the security group same as your servers . otherwise it will throw an error says server is not healthy

routing give it a name and then select the blog-demo and blog-about .

target group (set of severs that you can connect to)

1.name --> home-page-server

next --> select main-home
home target group should have the home server attached if not then edit it .
create one target group for about-page-server

Then edit rules for the Load-balancer :

Steps :
load balancer
in the listeners
edit rules
configure the path /about* --> about-page-server --> save
configure the path /home* --> home-page-server --> save

Then copy the load balancer DNS URL and paste it in the browser .

ssh to stream server .

cd /var/www/html
mkdir watch
cp index.html /about

ssh to home server . 

cd /var/www/html
mkdir home
cp index.html /home

Now our servers are inside a private subnet and we can only connect to them through using a proxy server that we have deployed in our public region .

NAT Gateway :

In case we need to update our server or the application then we could not do in our current architecture . We have to configure NAT (Network address translation) which converts the private IP to public and then we can communicate to the Internet .

Steps to attach a NAT gateway

Navigate to the Nat Gateways
create a NAT Gateway
Edit the private route table routing configuration .
Destination : 0.0.0.0/0
Target : NAT-gateway

Now your private instance is attached to the internet .

$ping www.google.com

Thank You For reading 🔥🔥🔥

Prometheus architecture and use cases :

Rahul Kumar — Sun, 29 Nov 2020 15:10:17 +0000

What is Prometheus :

Prometheus is an open source automated monitoring and Alerting tool used for analyze the logs and send the alerts in case of any failure .

Why use Prometheus :

As we all know monitoring our application as well as the infrastructure is really an essence and monitoring distributed applications is a real challenge .

Prometheus helps us in that area where it scrapes the metrics data from the endpoint using query language(PromQL).
Can be configured with Grafana which is a data visualization tool used for analyze the metrics data .
Trigger the alters in case of any misconfiguration in our application .
And the best part that it is free so community support is good .

Prometheus Components :

Prometheus server :
Is the core component of Prometheus consists of three components .
Client Libraries :
Provides language support for monitoring different applications .
Push gateway :
Push all the collected data to the database but only for short-lived jobs .
Exporters :
It is a script needed for configuration like exposing the /metrics endpoint .
Alert Manager :
Send alerts in case of misconfiguration to different channels like slack, email, notification .

Prometheus Architecture :

Prometheus server :

Data retrieval worker
Time series database
HTTP server

Data retriever helps us to retrieve the data from the application endpoint .
Time series database stores the metrics data so later they can be retrieved and analyzed .
HTTP server accepts PromQL query language push the collected data to the dashboard for visualization purpose .

push gateway :

In case of pull mechanism data retrieval worker pull all the metrics data from the endpoint .
As a result multiple Prometheus instances can pull metrics data .
But for short lived jobs like batch jobs which can be pushed to the Prometheus database . In that can we can use the push gateway .

Alert Manager :

Responsible for sending the alters in case of any failure in the system .

Prometheus web UI :

Prometheus have it's own UI to visualize the logs but can be configured with Grafana .

What does Prometheus monitor :

Linux/windows server
Web server
Stateless application
Statefull application

which units are monitored of those targets ?

It totally depends if we are analyzing log of an application or an operating system .
CPU states
Memory and disk usages
request count
request Duration
Exception count

These all are known as metrics which got saved into time series database which then can be visualized .

How does Prometheus collects the data :

The third component in our Prometheus server which is a HTTP server which uses PromQL query language and then scrapes the metric data form worker nodes through a HTTP endpoint - hostaddress/metrics

Target endpoint and exporters :

Some services are exposing to /metrics endpoint by default but many services have not .
exporter is a script and that fetches metric from the target so that Prometheus can understand that .

Example of a Linux server exporter :

download node expoter
untar and execute
converts metrics of the server
exposes /metrics endpoint
configure Prometheus to scrape this endpoint

Client libraries :

Client libraries have different language support for monitoring your application .

Data storage :

Prometheus stores its on-disk time series data under the directory specified by the flag storage.local.path.
But can be configured to store the data on a remote location .

Configuring Prometheus

In Prometheus.yaml file you can define your desired configuration .
global config defines how often Prometheus will scrape it's targets .
rules for aggregating metric values or creating alerts when condition met .
Scrape config for what resources Prometheus monitor .

DEV Community: Rahul Kumar

nest.js + TypeORM + PostgreSQL

Agenda :

Backend Architecture :

Technologies used:

What is Node.js :

What is NestJs :

What is TypeORM :

What is PostgreSQL :

In a common nestJS project :

Set up Database :

Create a connection to the database .

Generate module, service, controller for the article route :

How to build and run Kubernetes locally .

Install GOLang (latest version):

Setup the environment via Bash :

clone kubernetes

install etcd

start docker demon

Build k8s

Run a Local k8s cluster

verify the cluster is running in another window

Fork the k8s repo on github

setup gihub user name

Change the k8s source code

These instructions make the folllowing assumptions:

you can double check by runnig

Create a new branch before making a change

If you would like to make change we recommend finding some untested code and adding a few unit tests. If you don't know how to write unit tests in go read this

Example new test

Run Kubernetes Locally

Contribute a pull request

create a pull request via GiHub UI

Building and compiling Linux Kernel :

What is Linux Kernel ?

Dependencies :

How to installation :

Thank You 🤖🤖🤖

Set up MySQL with Sequelize in NodeJS :

Thank You For Reading

Set up MongoDB with node.js :

What is MongoDB :

What is Mongoose :

What is mongod:

What is mongo shell :

What is NodeJS :

Create the schema of our db

Create a new mongoose model the singular version of our db .

Create a new Document :

save the new value into the database

Thank You For Reading 🤖🤖🤖

Connect MongoDB Atlas and upload data :

What is MongoDB atlas :

setup :

Cloud computing :

Before cloud computing :

Disadvantages of on-site architecture :

What is cloud computing :

Cloud Model :

Service Model :

Deployment Model :

There are mainly three types deployment model :

Public cloud :

Private cloud :

Hybrid cloud :

Multi Cloud :

Thanks for Reading 🤖🤖🤖

Decoding the beauty of cryptography :

What is cryptography :

Process of cryptography :

History of encryption :

Classification of cryptography :

Classical cryptography :

Modern Cryptography :

Asymmetric key encryption :

Some examples of Asymmetric key :

There are lot more to know :

Thanks for reading 👨‍💻👨‍💻👨‍💻

Let's build a DevOps project :

The Agenda :

Create a `connection` to the database .