DEV Community: Mohan Manavalan

Comparison of HTTP and WebSocket Protocols

Mohan Manavalan — Sat, 29 Jul 2023 14:24:10 +0000

Introduction

The Hypertext Transfer Protocol (HTTP)is the most widely used protocol over the internet. It falls under the application layer in the Internet Protocol suite. Hypertext documents are the building blocks of the World Wide Web, which uses HTTP as its primary data communication protocol. Hypertext documents contain hyperlinks that points to the location of other resources over the World Wide Web and makes it easy to access them by simply performing events like clicking or tapping on the screen. The HTTP protocol was primarily designed as a request-response protocol. The request originates from the user’s browser and a response is sent by the server for this request.
While this is great for most applications, in today’s world there are certain scenarios where the server needs to push the data to the client and vice-versa asynchronously. While it is possible to achieve this using HTTP, the approaches for achieving this are often not optimal and has some drawbacks to it. The WebSocket protocol was designed to solve this problem. It enables two-way full duplex communication between a controlled client environment and a remote host server with minimum performance overhead while also using the existing HTTP infrastructure.

What is HTTP

Before we deep dive into what HTTP is, we need to have some understanding of the common terminology used throughout the document.
HTTP Terminology

Resource: A service or piece of network data that can be recognised by a URI. Resources could come in different forms (it can be an image, text, or some other form of data), or they might be something completely different.
Client: A controlled environment, typically a web browser, that establishes a connection to submit requests to retrieve resources from the internet.
Server: A software application that accepts connections and responds to queries by sending back information. Any program may be both a client and a server.
Origin Server: The server where a specific resource is located or created.
Proxy: A program that takes in a request from another client and forwards it to the server while also giving the response from the server back to the client thus behaving like both a server and a client. Requests are handled either internally or by forwarding them to other servers for possible translation. The HTTP specification's client and server requirements MUST be implemented by a proxy. A proxy is said to as "transparent" if it alters requests and responses only to the extent necessary for proxy authentication and identification. If a proxy modifies the request or response, then it is called a "non-transparent proxy. Unless explicitly stated otherwise, the HTTP proxy requirements apply to both types of proxies that use transparent or non-transparent behaviour.
Gateway: A server that acts as a go-between for many servers. Unlike a proxy, a gateway receives requests as if it were the origin server for the requested resource, the requesting client may not be aware that it is communicating with a gateway.
Tunnel: A third-party application that serves as a blind relay between two connections Once the tunnel is active, it is not considered a part of the HTTP connection, even though the tunnel may have been initiated by an HTTP request. When the relayed connections are closed at both ends, the tunnel disappears.
Cache: Local response message storage of a program and the subsystem that manages message storage, retrieval, and deletion. To speed up response times and conserve network capacity for similar future requests, a cache stores cacheable responses. Both the client and the server may have a cache. A server acting as a tunnel cannot have a cache.

Basic aspects of HTTP:

Simplicity: Even with the extra complexity created by enclosing HTTP messages in frames in HTTP/2, HTTP is still typically intended to be straightforward and easily readable. Humans are capable of reading and comprehending HTTP messages, making testing for developers easier and reducing complexity for novices.
Extensible: This protocol is simple to modify and experiment with thanks to HTTP headers. Even a straightforward understanding of the semantics of a new header between a client and a server can result in the introduction of new functionality.
Stateless but not Sessionless: Due to the statelessness of HTTP, there is no relationship between requests that are sent one after the other over the same connection. This has the potential to cause issues where users engage coherently with certain pages. HTTP cookies, however, enable the usage of stateful sessions even if HTTP's fundamental structure is stateless. Using header extensibility, HTTP Cookies are introduced to the workflow, allowing session generation on each HTTP request to share the same context or information.

Http Flow

The following steps are performed by the client to communicate with the server

Open a TCP connection: To send one or more requests and receive responses, a TCP connection is used. The client can establish multiple TCP connections to the servers, reuse an existing connection, or establish new ones.
HTTP Message is sent to the server: Before HTTP/2, HTTP messages can be viewed by humans. These straightforward messages are now encased in frames with HTTP/2, making it hard to read them directly, but the idea is still the same.
Response from the server is read by the client
Connection is closed

Realtime web with HTTP

HTTP was originally designed as a request/response protocol. Hence, it’s unidirectional by nature, which means that the client always asks for the information that is sent by the server. But there are many cases where the server needs to push data to the client. For such cases, we can use the following techniques in HTTP.

Http Short Pooling

In HTTP polling, a series of request-response messages are sent. When the server receives a new request from the client, it replies with the response if it is available or responds with an empty message. The client queries the server once more after a brief period (referred to as the polling interval) to check for any new messages. HTTP polling is used by many programs, such as chat, online games, and text messaging.

Http Long Pooling

Every HTTP request is made with the intention of "holding open" (not replying immediately) and only responding when there are events to deliver. This minimises message delivery delay since there is always a pending request that the server may respond to, in order to transmit events as they happen.

In contrast to "short polling," "long polling" tries to reduce both the consumption of processing and network resources as well as the latency in server-client message transmission. To accomplish these efficiencies, the server only responds to a request when a specific event, condition, or timeout has taken place. Normally, the client immediately sends a fresh long poll request when the server sends a long poll response. This effectively means that the server will always have a lengthy poll request open, to which it will respond as soon as fresh data becomes available for the client. As a result, the server is allowed to "initiate" communication asynchronously.

Http Streaming

A request remains open indefinitely thanks to the HTTP streaming technology. The connection is kept open even after the server responds to the client’s request. Since the connection is always open, HTTP streaming allows the server to send responses in bits of packages. The responsibility of collecting and joining the packages is given to the client. Because the client and server do not have to open and close connections, this approach dramatically lowers network latency. Below image shows video streaming using http streaming.

Drawbacks of Realtime web with Http

Overhead: A HTTP request is always associated with headers. These headers are useless for a real-time application. So, using HTTP for Realtime has a severe overhead of useless headers data.
Unidirectional: Since HTTP is unidirectional, the connection should still be opened by the client and needs to be maintained in its open state, else the server cannot push the message to the client

WebSockets

As we have seen above, using HTTP for Realtime applications has increased overhead in communication. Because the WebSocket protocol offers a full-duplex bi-directional communication channel that is accomplished with a single socket, it enables a robust real-time online application.
For achieving a WebSocket connection, first, a two-way handshake should be achieved between the server and the client. The client sends an Upgrade request to the server through HTTP. This is the only time HTTP is used and this is just used to initiate the handshake. If the server is configured to use WebSockets, it will respond with a connection upgrade response. With this, the handshake is completed and a WebSocket connection is created. Once the connection is made, any data like images and text can be transferred simultaneously in any direction in full duplex mode.

Advantages of WebSocket

Allows bidirectional communications between the client and the server without significant performance overhead.
WebSocket connection can bypass proxies and firewalls which can be challenging for many applications using the HTTP protocol. Whenever a WebSocket connection detects a proxy, it automatically creates a tunnel.

Security Model in WebSocket

WebSocket Protocol does not restrict access from clients with different origins. But it is possible to specify the clients during the HTTP handshake process using the origin header. Using this we can limit which web pages are permitted to talk to WebSocket Server.
This protocol is designed to fail when trying to connect to servers of other protocols like SMTP and HTTP, however, it does provide HTTP servers the option of supporting it if they so choose. This is accomplished by restricting the amount of data that may be added to the connection before the handshake's completion (therefore minimising the amount of server influence that can occur).
WebSocket Payload size should be restricted to a maximum limit. There are no inbuilt restrictions for this, and a payload of any size can be sent using WebSocket which might be dangerous. Hence maximum payload size should be restricted by the WebSocket server.
WebSocket does not provide any mechanisms for identifying the user. Hence it is always better to authenticate the users before the handshake and allow them to perform the handshake only when they are authenticated.
We can also use SSL over WebSockets just like we do in HTTP. WebSocket servers should always be configured to use SSL encryption.

Application of WebSocket

The following lists several instances of real-time applications that have used WebSockets:

Financial Tickers
Because of how quickly the financial world is evolving and how quickly human minds can digest information, algorithms are deployed. Data can be streamed without delays using WebSocket when it is necessary to track businesses that are worth using a dashboard for right now rather than a few seconds ago.

Sports updates
A further advantage of WebSocket is its applicability to sports updates, which enables users to stay informed if sports news is integrated in Web applications.

Multimedia Chat
When everyone can't be there in person, video conferencing is employed, yet in-person meetings cannot be replaced. As a better alternative, WebSocket is utilised with HTML5 video components, and obtain User Media APIs because the plug-in used in videoconferencing is quite hefty.

Location Based Apps
Applications are increasingly employing GPS to locate users and follow their movement along routes, allowing for the collection of relevant data.
Because HTTP is overly complicated, WebSocket is being utilised in place of it.

Online Education
As the internet is becoming more affordable and accessible and education becomes more and more expensive, online education is quickly becoming a great alternative. A fantastic online learning platform for communication between students and teachers can be provided by using WebSocket.

Cons of using WebSocket

Although WebSocket runs mostly on TCP, the initial handshake is still done over HTTP. So WebSocket servers need to support both TCP and HTTP implementations.
The security in WebSocket is still an un-explored territory and might be open to vulnerabilities.
The information from a WebSocket server cannot be cached by an intermediary (such as a proxy) as it is supposed to be changing continuously. Also, it’s not possible to perform pre-fetching of data from a CDN for WebSockets.

Conclusion

HTTP is the most widely used protocol on the internet. While it is suitable and optimized for many applications, when it comes to real-time data fetching, the traditional HTTP approach such as HTTP long pooling and HTTP streaming is not much effective. For these scenarios, we need to use the new HTML5 WebSocket protocol, which was designed to use the existing HTTP infrastructure to achieve real-time data fetching from the server.

Understanding JWT

Mohan Manavalan — Thu, 02 Jun 2022 05:05:38 +0000

What is JWT

JWT stands for JSON web tokens. It is an industry-standard, for passing user claims between client and service and between services. The claims in a JWT are encoded as a JSON object. A claim is just information of the user which is needed by the server to verify their identity and assign appropriate roles.
A JWT consists of 3 parts

Header
Payload
Signature

Header

The Header in JWT indicates that it is a JWT token and identifies which algorithm is used to generate the signature.

{
  "alg": "HS256",
  "typ": "JWT"
}

Payload

The payload contains a set of claims.

{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022
}

Signature

The signature validates the JWT token. The signature is generated by encoding the header and the payload along with a secret and concatenating the two together with a period separator.

HMAC_SHA256(
  secret,
  base64urlEncoding(header) + '.' +
  base64urlEncoding(payload)
)

The three parts are encoded and combined using a period separator to form a JWT token. This is what a sample JWT looks like. (The red part indicates the Header, the purple indicates the Payload and the blue indicates the Signature)

Why JWT

Most of the application we use requires us to be logged in to access some additional functionality. Normally user logs in by giving his credentials. It is not good practice to use the credentials provided by the user throughout the lifetime of the user's session.

Traditionally, the user's state and related information was maintained by servers. They would store this information in external storage like Redis. A session cookie was used to get the user's data from the external storage.

This solution would still work, but modern applications rely on RESTful services and RESTful services are stateless. They don't store any state information in them. This makes them highly scalable. So JWT was needed. JWT can store the required state-level information in them in a somewhat secure manner. The server will send this JWT token to the client and the client has to send this token in every subsequent request.

What are the pros and cons of using JWT

Pros of using JWT

Since the tokens contain all the necessary information, you do not have to query the DB all the time
The server does not need to maintain an external storage to handle users state
Easy to validate across multiple services. You just have one server which does the authorization and then passes the token across multiple services that require the information

Cons of using JWT

If someone gets the JWT, they may be able to impersonate the person
You cannot store lots of information into the JWT token as it will create a data overload. It is advised to store only the basic information and fetch other data by querying DB as required. Also for web applications, you will use JWT with cookies can store only a certain bytes of data.

Securing your JWT

As mentioned above, if some malicious user gets access to your JWT, they might be able to impersonate your user. So you must be careful when using a JWT.

Never store JWT in local storage in browser. Any malicious javascript code can easily access this.
If possible, create JWTs with short life. But be careful, if you invalidate your JWT, your users will be treated as logged off. You don't want your users to be logging in all the time.
Make sure you use SSL as it will encrypt your network traffic which will prevent your JWT from being stolen by anyone who has access to your network.

What to do if you loose your JWT

A compromised JWT is same as a compromised password. You should follow the same procedure as you do for a compromised password. This might include

Blocking the user's account
Asking the user to change his password immediately
Revoking all permissions from the user

How to know if you have lost a JWT

Acting quickly on a compromised JWT can be vital in preventing the loss of your customer data. We can do the following to know if we have lost a JWT

Monitor the client's location. Most likely, if someone has breached your security, he will try to use the token from some other location than the one which your client was using. This can be a good indicator to know that your JWT is stolen. But masking location is a simple task.
Analyze the pattern of the user. If your attacker has stolen your JWT, he will try to make use of it as soon as possible. So you will have to check the behavior of the JWTs. For example, if a user who normally sends 10 requests per minute, starts sending 50 requests then that is a sign of lost JWT.

As you see, securing a JWT is a complex task in itself and would require a significant effort if you care about the data of your users. That's why it is always advised that you delegate the task of handling authentication to an external service like Auth0 or Okta or FusionAuth. For a nominal charge, these services will take care of all your authentication problems.

Understanding password storage

Mohan Manavalan — Sun, 27 Mar 2022 11:02:04 +0000

Storing passwords can be a challenging and painful task. With this post, I hope to highlight some of the challenges. Keep in mind that managing password by yourself is never a good idea and it's always best to hand it over to a security expert by making use of one of the several auth providers out there.

Storing Passwords in plain text

So what's the deal with storing the passwords. Can't they just be stored as it is like other things in DB. Well, we can, but we should not do that due to the following 2 reasons

Anyone who has access to the database can then easily see and use the password. Sure, they might not do it, but nothing is stopping them from not doing it.
If your table is compromised by someway by a hacker, they can easily get access to the login details of your user. They just have to get access to the one table in the database. This might cause serious data breach and even legal problems.

Now we know that we should not store passwords as plain text. Lets look at some of the way, data can be stored safely.

Encryption Vs Hashing

Encryption is a mathematical algorithm that converts some text to something completely different by using an encryption key. Anyone who has access to the key can decrypt the encrypted data. If the encryption key is compromised, the passwords are likely compromised as well. Since we can find out the original password using an encryption key, it is not safe to use encryption for storing passwords.

Hashing is another kind of cryptographic mathematical function that converts a string into a completely random combination of characters of a fixed length. Unlike encryption, it is not possible to find the original string easily once hashed. Due to this reason, hashing is the preferred way of storing passwords. Some common hashing algorithms are MD-5, SHA-1, and SHA-2 family algorithms.

Hashing works well for storing the password for the below reason

It is impossible to know the length of the original password by looking into the hashed password.
It is close to impossible to reverse the hash.
Even a small change in input will produce a dramatically different hash.

input - password
hash - 5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8

input - Password
hash - e7cf3ef4f17c3999a94f2c6f612e8a888e5b1026878e4e19398b23bd38ec221a
// Just by changing p to caps produces totally different hash

So if we don't know the original password, how will we validate if the password entered by the user is correct? We will validate it by hashing the password entered by the user and comparing it with the stored hash in the table. We will get the same hash for the same input. Hence if the hashed password matches, that means the user has entered the right password. This is also the reason why it is not possible to retrieve your password once you have lost it (but still you can validate it with previously stored hashed passwords by comparing the hashed values)
So now even if your database is compromised, all the attacker will see is a collection of hashed passwords. Great! that means our client's data is safe right? Well, hashing protects data to an extent, but it's not completely safe. Let's put on our hacker's cap and try to see what we can do with the hashed password list.

Limitations of Hashing

Now we have a list of hashed passwords. You go through the list and find that there are a few identical hashes. That probably means that the password is a commonly used word. As a hacker, you can then just use the commonly used passwords as input and see if you can crack it. Most probably you would. That's the reason it's always advised to use a complicated password. But not all users are going to use it. And if you lose the user's data, it's going to be your responsibility.
There is also a list of commonly used passwords and there hashes available that get updated with every data breach. These are called the rainbow table which the hackers can use to figure out the password. So how can we overcome it?

Seasoning with Salt

As storing the password with hashing alone might not be enough, we have to combine the password with a randomly generated string which we call as the "salt" and then hash them together. Since we generate the salt randomly, we will get different hash even from the same passwords. So hashing makes it much more secure to save the password. The salt is then stored usually in a different database.

input1 - password
unsalted hash - 5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8
salted input - password6m:f?F!cZal-
salted hash - d60ec14145dcf8631e604b6db2a2fc1b0ef18460f8ecba77e79b2662ba36c9be

input2 - password
unsalted hash - 5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8
salted input - passwordtJgeu:nJSgvf
salted hash - 6a8cdd9c910aa70ed8bbd58eb03b1d46bc524cc0d85800b58c9bcf801b15c1d9

As you can see, with salt even for the same input, we generate completely different hashes. This makes salting passwords much better even for not so strong passwords.
To validate the user input, we will have to get the salt stored for that user, add the given password with the salt and hash it and see if it matches the hash stored in the database.

Spicing with pepper

It is not unheard of that even the database containing the salt of the user is compromised. If you want to secure our passwords further, we have to use a random string that we add to all passwords along with the password and salt. This random string is called "pepper" and it's usually stored right inside the source code. Just make sure you don't push this into a publicly hosted repository.
Using pepper might not be necessary all the time. And it might cause an issue if you compromise the value in the pepper cause you have hashed all the passwords with this single value and it is not easy to change. So keep that in mind.

Linked List and how dynamic collection of data is handled in modern languages

Mohan Manavalan — Wed, 23 Feb 2022 03:19:33 +0000

During programming, we often need to use collection of data. Collection of data can be handled in an array or in a linked list. Lets see the advantages and disadvantages for both.

Arrays

Arrays are probably the most commonly used data structures. They are fixed length data structure. That means when you create an array, the language compiler will allocate a fixed length of memory for you to use. You can add elements till that length. If you need to add more, you will have to create a new array. One advantage of an array is that we can access any element in the array if we know the index. This is one of the reason why they are the most common data structure. So in short

Arrays has limited size allocated to them
If we know the index, it is very easy to access the element of the array.

Linked List

Often times we don't really know the size of the collection that we use. We might need to add elements to the collection to satisfy our use case. This is where Linked List come into use. Linked List is a collection of data where each data is linked to each other. Depending on the type of linked list each of your data might know the details of its next value in the collection (singly linked list) or it might know both the next and previous value in the collection (doubly linked list). They don't need to have a fixed length and it is possible to add new data to the a linked list. So to summarize

Linked list is a collection of data where each data knows about its adjacent data
The length of a linked list is dynamic. We can easily add data to a linked list.

Singly Linked List

Doubly Linked List

Solutions offered by modern languages

As I said, arrays are probably the most used data structures. On the others hand linked list are almost never used. Even though we find that in many scenarios we don't really know the length of the collection of data. Still we use arrays most of the time. This is possible because of two reasons

Most languages offer simple solution to add data by abstracting the creation of new array and providing simple methods to add data

For example C# provides List collection for handling collection of data whose length is not fixed.

Internally when we create a new List object, the compiler assign creates and array with double the size of the given length. It adds data to the same array when we use the List.Add method. When the length exceeds the initial allocated memory, the compiler then creates a new array with increased size and allocates all the data to this array.

// The c# compiler creates an array of length 2 * given length
var a = new List<int> {1,2,3,4};  // creates an array of length 8
a.Add(5);
a.Add(6);
a.Add(7);
a.Add(8); // all the allocated memory of array is filled
a.Add(9); // this now creates a new array with the increased length

If there is no way to figure out the initial length, then the compiler creates an array of length 4.

var a = new List<int>(); //this creates an array of length 4

This is the reason, we are not allowed to access the length property for a c# List object. Instead we are given a Count property which gives us the count of data stored in the array instead of the length of the array.

Most of the languages are now garbage collected, so the unused array is removed with very little loss of memory.

We might end up having to create large number of new arrays by using the inbuilt language methods. It would not be efficient and might take huge space if those memories locations are not freed. Fortunately, most of the modern languages comes with garbage collector. The garbage collectors takes up the responsibility of freeing up the unused memory. So we don't have to worry about clearing the memory manually, which might become a tedious process.

Conclusion

While creating a new array for each element added might take a bit more processing power, its usually always compensated by the ease which arrays allows us to access its elements. This makes arrays the most popular data structures and almost always more efficient to use then the linked list. While linked list might not see much use, understanding the working of linked list will help us understand other data structures more easily.

Understanding Dependency Injection in JavaScript

Mohan Manavalan — Mon, 06 Dec 2021 06:02:32 +0000

Dependency Injection is a fairly complex topic for beginners. It might not be necessary to know the concept but knowing it will help you make better decisions on your code.

Lets start by the definition.
Dependency Injection - Dependency Injection is the technique in which an object receives other object in which it depends on (source: Wikipedia).

Now lets try to break down the definition a bit. Lets start by object. An object is an instance of a class. For Example

// lets define a class dog
class Dog{
  speak(){
    console.log("wuff");
  }
}

//now lets create object dog
const fluffy = new Dog();

In the above example, we have a class Dog and fluffy is the object of the class Dog. When we new up a class we create an object of that class. This is one of the ways of creating an object in Javascript (and its the common way to create an object in languages like c# and java).
Lets now see an example where 2 objects are dependent on each other.

class Pet{
  whatDoesMyPetSay(){
    const pet = new Dog();
    pet.speak();
  }
}

const fluffy = new Pet();
fluffy.whatDoesMyPetSay();
// response will be "wuff"

Here, as we see, the class Pet is dependent on class Dog. So to get what we want, we need to create an instance of Dog inside our Pet class. Now this class is not reusable as it is tied to Dog class. If someone has a cat as Pet, they wont be able to use this class. This is what is called as tightly coupled code.
Now lets change this code and try to satisfy all other pet owners with dependency Injection. But first, lets create a cat class

class Cat{
  speak(){
    console.log("meow");
  }
}

The cat class must also implement the same method for Dependency Injection to work. In languages like C# and Java this is ensured by using an interface. But we have no such method in JavaScript, so it is up to the developer to remember it. Now lets see the new implementation of the pet class.

class Pet{
  //usually we have a private variable that needs 
  //to be accessed only in this class
  #pet;

  //create a constructor that recieves the dependent
  //object
  constructor(pet){
    this.#pet = pet;
  }

  whatDoesMyPetSay(){
    //as long as pet class implements speak method we are fine
    this.#pet.speak();
  }
}

//what does fluffy the dog say?
const fluffy = new Pet(new Dog());
fluffy.whatDoesMyPetSay();
//The response will be "wuff"

//what does milo the cat say?
const milo = new Pet(new Cat());
milo.whatDoesMyPetSay();
//The response will be "meow"

Now, we have removed the dependency from inside the pet class and have given it to the caller of the class. This promotes the reusability of the pet class. This is a very simple example and the purpose is to only understand dependency injection and not to implement it. In real world, the dependency is abstracted even from the caller and given to a new object, which is usually called an Injector.

Why not to use Dependency Injection in JavaScript

If you have read till here I hope you are clearer with the concept of dependency injection. Now lets see some reasons we might not want to use dependency injection

Unlike purely class based languages like c# and java, JavaScript provides us lots of flexibility when it comes to grouping functionalities. Many times, you won't even need to use a class and just using a function might be enough. In those cases, trying to implement Dependency injection will only create unnecessary complexities.
JavaScript also happens to be dynamic in nature. You can overwrite and use any function implementation in JavaScript. So we should make use of these rather then dependency injection for unit testing our code.

At the end of the day, as a developer, we should realize that there is no one solution that fits all problems. I hope with this article, you would be able to take a better decision on your solutions.

Web Developer environment setup for a new Ubuntu/ Ubuntu based distro

Mohan Manavalan — Sun, 07 Nov 2021 05:15:09 +0000

In this post I will share the common developer environment setup steps once you have a new Ubuntu/Ubuntu based distro setup. This can be a clean OS installation or a Windows subsystem for Linux setup.

So the first thing we need to do is update the existing packages currently present. To do that we need to run the following command in the terminal. This will make sure that you have the latest security patches updated in your instance.

sudo apt-get update
sudo apt-get upgrade

Now we will install NodeJs. We will install NodeJs using the apt package manager which is the default package manager for Ubuntu. We need to use the following command on the terminal to install the latest LTS version of NodeJs.

sudo apt install nodejs

Verify that node is installed using the below command

node -v

Node will also install node package manager. If you don't see node package manager, then you can install it separately using the below command.

sudo apt install npm

You can verify the installation with

npm -v

I also like to install Node Version Manager (NVM) tool. NVM makes working with different versions of NodeJs so much easier. To install NVM you need to use the following command

curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash

Once nvm is installed, you will need to restart your terminal to be able to use it.

Most of the Linux distro installations should come with Git. But if git is not present in your installation, you can easily install it using the below command.

sudo apt install git

Once you have installed git, you need to configure it with your user name and email. This can be done with the following command

git config --global user.name "John Doe"
git config --global user.email johndoe@example.com

I like to configure some global git alias as well to make my life easier to work with git. The most common one if to get a better log info. This is what I use.

git config --global alias.lg "log --graph --all --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit --date=relative"

Next thing you need to do is install a text editor of your choice. I like to install VSCode. It works especially well with WSL.

With this you should be good to start with your development journey. But I like to do the following things to improve my developer experience. Feel free to skip this part if you are new or not interested in this.

The default editor for Ubuntu is nano. I don't really like to use nano. So I usually change it to vim. Ubuntu comes with vim pre-installed as well. You can change the default editor using the below command

sudo update-alternatives --config editor
#Type the number which corresponds to vim and press enter

This should set your default editor to vim. You can change it to something else as well with the same command if you don't prefer vim.

Next think I like to change is the default shell. By default Ubuntu comes with Bash shell which is not the best shell option to have. I like to install ZSH. To do that use the following command

sudo apt install zsh

Once ZSH is installed, you need to change the default shell from bash to ZSH. Use the following command to do that.

chsh -s $(which zsh)

You will need to restart your terminal window to see the change. To verify that the default shell is set to ZSH, use the following command

echo $SHELL

This should print out the following output.

/usr/bin/zsh

If everything is fine, then we move on to install Oh My Zsh which is a framework for managing your ZSH configuration. It superpowers your terminal experience and a must have according to me. You can install Oh My Zsh with the below command.

sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Once its installed you should see the following in your terminal.

Now you need to select a good theme for your oh-my-zsh installation. You can browse the list of available themes from the below links.

https://github.com/ohmyzsh/ohmyzsh/wiki/Themes
https://github.com/ohmyzsh/ohmyzsh/wiki/External-themes

Once you have selected the theme, you open your .zshrc file and change the theme name. .zshrc file is located in your home folder which can be accessed by using cd ~

ZSH_THEME="intheloop"

Once this is done you will need to restart your terminal fo rthe change to take place.

Now you are really good to start your development journey in your new Ubuntu based instance (This time honestly).

Happy Coding!

Change the directory of WSL

Mohan Manavalan — Sun, 31 Oct 2021 06:29:31 +0000

Windows sub-system for Linux (WSL) is an extremely useful tool for developers to improve their development experience in windows operating system. It allows the users to run Linux commands and apps directly in windows without having to use a VM or dual booting your computer. But one issue I faced was that by default, It is installed in c drive of windows. You can use WSL in other drives as well, but to make full use of the speed and functionality offered by WSL, it is recommended to use it in its home drive, which by default is set to C with no option to change it during the time of installation. Now if you are someone like me who has a small C drive with windows installed on it (stored in SSD to speed up the OS boot-time, or some other reason), you might find yourself going out of space in C drive pretty quickly. To overcome this we can change the directory of WSL to some other drive.
To do that we need few things. First you need the WSL command line tool. Please note that WSL command line tools windows version 1903 or later. Chances are, if you have installed WSL, you must have already be having it. If you don't have WSL, then you need to open a command prompt or PowerShell window with admin permission and just use the following command

wsl.exe --install

This will take care of installing WSL for you along with the WSL command line tool.

Once done you can verify the installation by using the below command

wsl.exe --list

This should show you the list of distributions currently installed in your computer.

Now that we have WSL installed, we will look into how we can change the default directory. First you need to create a .tar file of your existing Linux distro. To do that you must use the below command

wsl.exe --export <Distro Name> <tar file name>

Here Distro Name will be the name of the Linux distribution whose directory you wish to move. will be the path along with file name where your .tar file be generated. If you don't give the path in it will be created in the current directory. For example -

wsl.exe --export Ubuntu20.04 d:\ubuntu.tar

Once this is done verify that the .tar file is generated in the path. Once that is done, you need to take a backup of all the data from your current home path of the distro, cause after this you will loose access to your distro and might loose the files. Once that is done, use the below command to unregister your distro and bring back all the precious space that your distro had hogged up.

wsl.exe --unregister <Distro Name>

Again the Distro Name will be the name of the distro you wish to unregister.
Now the final step is to import the exported .tar file into your desired directory. To do that use the following command

wsl.exe --import <New Distro Name> <Distro Path> <tar file name>

The is the name you wish to identify your distro as in your computer. will be the path you wish to store this distro in. Typically this will be some path in the directory you wish to move your WSL. will be the full path and name of your exported .tar file. An example of the above command will look like

wsl.exe --import Ubuntu d:\wsl\Ubuntu d:\ubuntu.tar

Once the command is executed, your distro will be created in the new path. By default, we will be logged in as the root user. Its not the ideal scenario as the root user has access to change things without needing the password. So it is better to change to a new sudo(sudo user is like admin user in linux) user. You will still have the user and password you created at the time of installing WSL available. You can switch to that user by using the below command

su <username>

where username is the user name of the user you created while installing WSL. You can verify the user details by using the below command in terminal

whoami

If this shows the user name instead of root. Then you have successfully switched to the account. Now you will have to give password for making changes with sudo command.
Its not ideal to have to switch to the new user every time you open your terminal. It would be great if we could set it as the default user just like how it was before we moved the disk. To do that we will need to open a command prompt or windows powershell and add the following command based on your distro

<distro-name> config --default-user <username>

where username is the name of the user you wish to configure and distro-name will be the name of your distro.

for Ubuntu it will be "ubuntu"
for openSUSE Leap 42 it will be "opensuse-42"
for SUSE Linux Enterprise Server 12 it will be "sles-12"

Now each time you open up your terminal, you will be set as this user by default. You can verify that by running whoami command in your terminal.

Phew, quite a bit of work. You can finally use your WSL without worrying about space.

Hope this article was helpful. Thanks for reading.

What is RESTful Service?

Mohan Manavalan — Sat, 24 Jul 2021 17:28:35 +0000

Web is a place where there are tons of computers talking to each other. As the web grew, it became clear that we needed a common and secure way to allow others to interact with our application. This need gave rise to Representational State Transfer Architecture or more commonly known as REST architecture. REST defines a set of constraints for how the web should behave.
REST defines a client-server communication model. The device making the request is called the Client and the device which is listening for the requests is called a Server The server is in charge of the resources. A resource can be anything like a HTML file, a JSON object, an image file or event a JavaScript file. The client-server communication happens with HTTP/HTTPS protocols. The actual logic for fetching the resource is abstracted behind the server and the client has no idea about what happens in the server. The resource might be fetched from the server the client requested or it can be fetched from some other server. The only thing client is concerned is that if they are getting the resource or not, and if they don't receive the resource, what is the reason for that? REST is also Stateless, which means that the server has no idea about the previous calls from the client. The server does not hold on to information's for the client. A web service that obeys the REST constraints is called a RESTful service.

The Client calls the server using http methods and gets a http response code and http response status along with the resource as the response. The most commonly used HTTP methods are

GET: This the the most widely used request method. When the client makes a GET request, it means that the client is asking for the resource mentioned in the URI. Multiple GET request for the same resource will result in same response from the server.
POST: The second most common request method. A POST request from the client comes with a request body that contains a structure of the resource itself. POST request is used to create the resource enclosed in the request body of a POST request. Every time we make a POST request, the server will create a new request even if the body of the request doesn't change.
PUT: A PUT request, just like the POST request contains a structure of the resource in its request body. A PUT request tries to find of the resource is already present and if its present then update the resource and if its not present it will create the resource. Hence PUT request will always give same response for same request.
DELETE: A DELETE request from the client means the client wants to delete resource.

The response codes from the server are in the following categories

Successful responses
When we get the expected response from the server. This includes the following status codes

200 OK: This means that the request was success. This can be the result for any HTTP methods.
201 Created: This means that the resource was created. This will be the response for either PUT or POST request.

Client error responses
When there is something wrong with the request made by the client we get one of the client error response.

400 Bad Request: The server was not able to understand the request due to as the structure of the resource in request body is not same as the one server expects
401 Unauthorized: The client is not recognized by the server and this is a resource only for recognized clients.
403 Forbidden: The client does not have access to the resource. Unlike 401, here the server is aware of the client but the client does not have access to the resource.
404 Not Found: The server cannot find the requested resource.

Server error responses
When there is something wrong with the server, then we get one of the server error response.

500 Internal Server Error: The server encountered a situation it doesn't know how to handle.
501 Not Implemented: The request method is not supported by the server.
503 Service Unavailable: The server is not available to handle the request. This might be due to the server being down for maintenance.

Serverless Computing

Mohan Manavalan — Sat, 17 Jul 2021 16:53:30 +0000

What is Serverless?

The concept of cloud is to move the infrastructure our application is managed by the cloud vendor. Serverless takes it one step further. With serverless you don't have to worry about anything except for your code. Everything else like the OS of the server, or even the framework of the application is taken care by the cloud vendor. This allows the developer to concentrate only on the code. With serverless, the developer has to write a function that gets executed by a trigger. The trigger is an event that can be configured as per the requirements. Since serverless is mostly functions that respond to triggers, its also called serverless functions. Even though it's named serverless, it doesn't really mean that we don't have any servers, it just means we don't have to worry about managing the servers.

Why select serverless?

Now you might be wondering, why do we need a serverless function? The advantage of a server less function is that you will only be charged for the amount of time your function is hit. You don't need to pay anything if your serverless function is not being used. The others advantage is that you can easily scale up and scale down your function based on its load easily.

Popular Serverless Offerings

Serverless functions can be used in lots of scenarios. Because of this, most popular cloud vendors are offering serverless functions. AWS lamda is the first and probably the most popular serverless functions offering. The other famous function offerings are

Google Cloud Functions from Google Cloud platform
Azure Functions from Microsoft Azure
Oracle Cloud Fn from Oracle
OpenWhisk from IBM

Working with AWS Lamda.

AWS lamda is the serverless offering provided by amazon web service. Aws provides 1 million requests for free forever which is a great offering. But please note that AWS lamda needs to be called by an another AWS service which might not be free, so please do check AWS pricing for that.

We will see a sample AWS lamda funtion created using node js. You can create an AWS lamda function from the AWS console. When you create a new function you will see something like this

AWS lamda function is called by a trigger. This trigger is by one of the other services offered by the AWS, it can either be a api gateway, S3 bucket, dynamo DB etc.

The function may or may not call other AWS services like S3, or SNS service which will be the destination if it does calls them.

A simple lamda function looks like below in node js.

exports.handler = async (event) => {
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

Every lamda function will have a handler function. The handler is the function that gets called by the trigger. You can write whole application in this handler function like call a DB and send mail etc. AWS makes it very easy to integrate with other AWS services.

Thats it for today. This was a very basic introduction for serverless computing. Please let me know your thoughts in the comments.

Deep dive into Git Rebase

Mohan Manavalan — Sat, 10 Jul 2021 11:57:00 +0000

Whenever we have two branches and they are being worked on by separate members and we want to integrate the changes from one branch to the other, we have two options, merging and rebasing. These two commands work in a very different way. Merge will take the changes from the other branch and create a new commit with those changes into the branch. In a way, merge happens as if we did those change ourselves and committed those changes. But merge preserves the reference from the main branch in its log.

/// checkout the branch your are working and merge the branch with latest changes into it.
git checkout branch1
git merge branch2

Git rebase however performs this differently. When you rebase your branch onto another branch, git changes the whole base of the branch to the new commit. Its is as though you have started working from the latest changes of the other branch. It takes all the commits from your branch and starts applying them one by one on the new branch, thereby modifying the git branch history.

///  checkout the branch your are working and rebase the branch with latest changes into it.
git checkout branch1
git rebase branch2

Why Git Rebase?

Rebase is an operation that is not used or not understood properly. If we can just merge the changes, then why would you need to think of rebasing? But rebase has its own uses. One of the most important advantages of git rebase is, it keeps your history in a linear fashion. There will not be any unnecessary branchings when you use rebase. Git rebase also allows us to rebase the commits in an interactive way. This allows us to perform lots of other operations

Git Rebase interactive

If we want to rebase with the interactive prompt, we need to use the following command

git rebase branch2 -i

Some of the functionalities we can achieve through interactive rebasing are

we can squash multiple commits into a single commit
normally git rebase will apply all the commits. We can choose to skip the commits we don't want to include.
change the order in which commit occurs in your git history.
change the commit messages of previously committed changes

Rebase only part of the commit

If you want to rebase only a specific number of commits then you can do so by using the below command

git rebase branch2 -i HEAD~3

This command will now rebase only the last 3 commits.

These are some very powerful functionalities that will affect the whole branch. We should be careful and use rebase only when we know it's safe to do so. Once you have changes git history it is impossible to recover that.

When should you not use git rebase

Usually, it is advised to use rebase only on the branches in which only you are working. It is useful to use rebase in this case and it also helps to keep the local commit tree clean from unnecessary commits. But once you have raised a PR, we should not rebase any further as it will change the commit tree which will cause confusion for the reviewer.
Another thing to keep in mind is that we should never rebase a public branch. We should never rebase a feature branch, in which lots of people are working. This will cause lots of issues for those who are using the branch currently. So it's not a great idea to rebase on public facing branches.

Demystifying Web Jargons Part - 2

Mohan Manavalan — Sun, 04 Jul 2021 10:58:48 +0000

Welcome to the second part of my demystifying web jargons series. If you have not checked the first part, you can do so from here
Demystifying Web Jargons Part - 1

Today we will be covering the below topics.

API
DOM
Web Service
Database

So let's start straight away.

API

API stands for Application Programming Interface. According to Wikipedia “an API is a type of software interface, offering a service to other pieces of software”. Let's try to understand this definition by breaking it down. First of all, it's a software interface, which means it is a bridge that allows the software to talk to another software. Unlike a graphical user interface (GUI), which allows a person to communicate with a computer or a computer program, an API is not designed to be used by common people. It is specifically designed to be used by a software of software developer.
The second line state that it offers services to other software. That means it allows others programs to talk to software through certain methods. These methods are called the endpoints or API methods that the API is said to expose to the outer world. This is one of the purposes of API, which is to hide the underlying implementation of a program. When a program uses these exposed methods it is said to consume the API method or endpoint.

DOM

We now know that the browser renders a webpage with the help of HTML (check part 1 of this post for details). HTML along with CSS allows us to create beautiful web pages. But if you see any modern websites, you will notice that they allow users to interact with them in multiple ways other than just navigating from one page to the next. This capability to be interactive with the user is done using javascript which is a programming language (although nowadays you can also use web assemblies to achieve this). But to interact with an HTML element javascript will need to understand and parse HTML in a way that makes it easy for it to work with it.

The browser engine converts the HTML document to a tree structure, wherein each node is an object representing a part of the document. Each branch of the tree ends in a note. This Tree structure is called Document Object Model or DOM.

The DOM API exposes methods that allow programmatic access to the tree and one can change the structure, style, and/or content of the HTML document using these APIs. They also expose APIs, that allow us to attach Event handlers to the DOM nodes. An Event Handler is a function that is triggered when a certain event is fired. An event can be any of the inputs even like a mouse click or button press. There are various events available for us to work with. Using these Event Handlers It becomes possible for us to build an interactive website.

Web Service

An interactive site is not very useful if it doesn't have any data. Every site needs data from the server. A web service is an API through which a client or another server can talk to a server. A web service can be called from a client or another server. It allows us to communicate with other devices through the world wide web. In other words, a web service is an interface that allows us to communicate over the internet by transferring data.

Database

Now we got the data from the client, but this data needs to be stored somewhere. Storing data is a complex and sensitive task. Over the internet, data is everything. A small data inconsistency might be the doom of a company if not worse.

A Database is a special server dedicated to storing and updating data. A database is usually controlled by a database management system (DBMS). There are Primarily two kinds of databases.

Relational Database
Document Database

Relational Database

Relational database stores the data in an optimized fashion, to minimize the duplicate storing of the data, by splitting the data and storing them into different tables. When we need to manipulate this data, we need to connect these tables and update the data. This is a complex task and usually performed by using Structured Query Language or SQL. SQL is used by almost all relational database management systems for handling the connected or relational nature of the tables. Since the data is split and stored in multiple tables, the data needs to be looked up and fetched from multiple tables. Due to this, a relational database tends to be a little slower in its capacity to read and write data.

Document Database

A document database stores the data in unstructured documents or collections. These database systems do not depend on tables and hence they don't need special language like SQL to work with the data. They are also extremely fast in their reads and writes as they don't need to look up and connect multiple tables. The drawback of a document Database is that since they store data in unstructured form, the data is often duplicated and hence they require more space. In a way, document databases are similar to the traditional storage of data in a paper document.

Understanding Git a little better.

Mohan Manavalan — Sun, 27 Jun 2021 16:24:35 +0000

I see that a lot of people are confused and even scared of git, especially the beginners. To me, git is an awesome and powerful tool. I will try to explain the common use cases of git. I will not be telling about the commands, but the concepts, so that it will be easier for everyone even if you are using a GUI tool to work with git.

Let's start by explaining a common word used across git, repository

Repository

A repository or a repo (as it is commonly called) is a folder containing all your source code, the build scripts, dependency information, unit tests, environment variables, etc. Basically, when you say you are working in an app, everything for that app will be under a single repository (although not necessary).

Branch

This is probably the most basic use case of a version control system. When you have an app running in production, typically you would not want to change its code as it might impact your live app. So, if you want to make any changes, first you need to create a copy of the original repo, and then work on it. Git allows us to easily create this copy by using Branches. The original repo is usually called the main branch or master branch. The copied branch can be named anything, feature, beta, v1.0, v2, depending on the naming scheme of your team. From here onwards, I will just call it the feature branch for simplicity's

Cloning

It's not a good idea to keep your code only in your local machine. If something happens to your computer, you will lose all your code. To prevent this from happening, we normally host our repos in servers like Github, Bitbucket, Codecommit, etc (Github being the most popular one). If your fellow developer wants to work on the code, they will need to download a copy of your code into their local machine. This process of downloading the code into our local machine from a server is called cloning.

Commit

You have made some changes for your feature. Now you need to save your changes into your git repository. We can do this by using commit. But commit not only saves the files but also creates a copy or snapshot of the original file and saves the file every time you perform commit. Commit is one of the core features of git. Git does not take any changes that we make into consideration unless we explicitly mark the change for commiting. This process of marking the change for commiting is called staging in git. After you stage a file, you will need to give some message on what the change is. This will be useful in case you ever want to come back to this commit. So it is always advised to give a proper and meaningful commit message. Each time we commit a change, git tracks it and adds a unique id for that commit. You can get the full information on all the commits by using git log or any nice GUI tool that shows the git graph.

Log will show you where your current code is usually by marking them as head. It also shows other information's such as the commit message, author information, the date and time of the commit, etc. If you are using a GUI tool, it might show the information in a different way. If you are using VS code, I would recommend installing the git graph extension. It shows the information in a really nice way as below.

As you see the data is shown in a table here. The columns denote the commit messages, Date, author and commit ids. each row is actually a git commit. "origin/Head" shows at which commit the code is in our local, which in this case happens to be at "origin/master". "origin/docs", "origin/feat/automatic-esm-file-generation" and "origin/feat/show-sent-data-in-rest-errors" are the other branches in this repo (as you see in the graph column, they are also denoted with a different color). Here we can see that "origin/feat/automatic-esm-file-generation" and "origin/feat/show-sent-data-in-rest-errors" are branches with changes that are not there in the master branch as they start from the master branch.

Push

When we make a change and commit the code, the change is only saved in our local machine. To share this change with other team members you will need to upload these changes into your git repository on your server. To do this we need to perform push. It's like we are pushing the code from our local machine back to the server.

Pull

In a team, there will be more than one developer working on the code. The code in the server constantly keeps getting updated. But these changes will not be automatically downloaded into your local machine. To get the code into your local machine you need to pull. Pull will get the latest code of your selected branch into your local machine.

Fetch and Fetch All

Pull will get the latest changes from the server into your local machine and update it in your branch. If you just want to get the latest changes of the branch, then you need to fetch. If you want all the latest changes from the server to your local, you have to perform Fetch All.

Merge

If you want to get the latest changes from another branch into your branch, then you will have to merge the other branch into the branch. If you are working on a file that was also changed in the other branch, then you will be shown conflicts while merging. You need to resolve these conflicts before you can proceed with the merge.