DEV Community: Sumedh Nimkarde

Understanding the Node.js event loop phases and how it executes the JavaScript code.

Sumedh Nimkarde — Sat, 08 Feb 2020 14:43:00 +0000

I believe if you are reading this, you must have heard about the famous event loop that Node.js has, how it handles the concurrency mechanism in Node.js and how it makes Node.js a unique platform for event driven I/O. Being an Event driven I/O, all of the code that is executed is in the form of callbacks. Hence, it is important to know how and in what order are all these callbacks executed by the event loop. From here onwards, in this blog post, the term 'event loop' refers to the Node.js' event loop.

The event loop is basically a mechanism which has certain phases through which it iterates. You must also have heard about a term called 'Event Loop iteration' which implies an iteration of event loop over all of its phases.

In this post, I will be going a bit at showing you the lower level architecture of event loop, what all its phases are, which code is executed in which phase, and some specifics and lastly some examples which I think will make you understand better about event loop concepts.

Following is the diagram of what all phases an event loop iterates through as per their order:

So, the event loop is a mechanism in Node.js which iterates over a series of in loop. Following are the phases that the event loop iterates through:

Each of the phases has a queue/heap which is used by the event loop to push/store the callbacks to be executed (There is a misconception in Node.js that there is only a single global queue where the callbacks are queued for execution which is not true.).

Timers:
The callbacks of timers in JavaScript(setTimeout, setInterval) are kept in the heap memory until they are expired. If there are any expired timers in the heap, the event loop takes the callbacks associated with them and starts executing them in the ascending order of their delay until the timers queue is empty. However, the execution of the timer callbacks is controlled by the Poll phase of the event loop (we will see that later in this article).
Pending callbacks:
In this phase, the event loop executes system-related callbacks if any. For example, let's say you are writing a node server and the port on which you want to run the process is being used by some other process, node will throw an error ECONNREFUSED, some of the *nix systems may want the callback to wait for execution due to some other tasks that the operating system is processing. Hence, such callbacks are pushed to the pending callbacks queue for execution.
Idle/Prepare: In this phase, the event loop does nothing. It is idle and prepares to go to the next phase.
Poll:
This phase is the one which makes Node.js unique. In this phase, the event loop watches out for new async I/O callbacks. Nearly all the callbacks except the setTimeout, setInterval, setImmediate and closing callbacks are executed.
Basically, the event loop does two things in this phase:
1. If there are already callbacks queued up in the poll phase queue, it will execute them until all the callbacks are drained up from the poll phase callback queue.
2. If there are no callbacks in the queue, the event loop will stay in the poll phase for some time. Now, this 'some time' also depends on a few things:
  - If there are any callbacks present in the setImmediate queue to be executed, event loop won't stay for a much longer time in the poll phase and will move to the next phase i.e Check/setImmediate. Again, it will start executing the callbacks until the Check/setImmediate phase callback queue is empty.
  - The second case when the event loop will move from the poll phase is when it gets to know that there are expired timers, the callback of which are waiting to be executed. In such a case, the event loop will move to the next phase i.e Check/setImmediate and then to the Closing callbacks phase and will eventually start its next iteration from the timers phase.
Check/setImmediate: In this phase, the event loop takes the callbacks from the Check phase's queue and starts executing one by one until the queue is empty. The event loop will come to this phase when there are no callbacks remaining to be executed in the poll phase and when the poll phase becomes idle. Generally, the callbacks of setImmediate are executed in this phase.
Closing callbacks: In this phase, the event loop executes the callbacks associated with the closing events like socket.on('close', fn) or process.exit().

Apart from all these, there is one more microtask queue which contains callbacks associated with process.nextTick which we will see in a bit.

Examples

Let us start with a simple example to understand how the following code is executed:



function main() {
  setTimeout(() => console.log('1'), 0);
  setImmediate(() => console.log('2'));
}

main();

Let us recall the event loop diagram and combine our phase explanation with it and try to figure out the output of the above code:

When executed with node as an interpreter, the output of the above code comes out to be:

1
2

The event loop enters the Timers phase and executes the callback associated with the setTimeout above after which it enters the subsequent phases where it doesn't see any callbacks enqueued until it reaches the Check (setImmediate) phase where it executes the callback function associated with it. Hence the desired output.

Note: The above output can be reversed too i.e

2
1

since the event loop doesn't execute the callback of setTimeout(fn, 0) exactly in 0ms time. It executes the callback after a bit of delay somewhat after 4-20 ms. (Remember?, it was earlier mentioned that the Poll phase controls the execution of the timer callbacks since it waits for some I/O in the poll phase).

Now, there are two things which happen when any JavaScript code is run by the event loop.

When a function in our JavaScript code is called, the event loop first goes without actually the execution to register the initial callbacks to the respective queues.
Once they are registered, the event loop enters its phases and starts iterating and executing the callbacks until all them are processed.

One more example or let's say there is a misconception in Node.js that setTimeout(fn, 0) always gets executed before setImmediate, which is not at all true! As we saw in the above example, the event loop was in the Timers phase initially and maybe the setTimeout timer was expired and hence it executed it first and this behaviour is not predictable. However, this is not true always, it all depends on the number of callbacks, what phase the event loop is in, etc.

However, if you do something like this:



function main() {
  fs.readFile('./xyz.txt', () => {
    setTimeout(() => console.log('1'), 0);
    setImmediate(() => console.log('2'));
  });
}

main();

The above code will always output:

2
1

Let us see how the above code is executed:

As we call our main() function, the event loop first runs without actually executing the callbacks. We encounter the fs.readFile with a callback which is registered and the callback is pushed to the I/O phase queue. Since all the callbacks for the given function are registered, the event loop is now free to start execution of the callbacks. Hence, it traverses through its phases starting from the timers. It doesn't find anything in the Timers and Pending callbacks phase.
When the event loop keeps traversing through its phases and when it sees that the file reading operation is complete, it starts executing the callback.

Remember, when the event loop starts executing the callback of fs.readFile, it is in the I/O phase, after which, it will move to the Check(setImmediate) phase.

Thus, the Check phase comes before the Timers phase for the current run. Hence, when in I/O phase, the callback of setImmediate will always run before setTimeout(fn, 0).

Let us consider one more example:



function main() {
  setTimeout(() => console.log('1'), 50);
  process.nextTick(() => console.log('2'));
  setImmediate(() => console.log('3'));
  process.nextTick(() => console.log('4'));
}

main();

Before we see how the event loop executes this code, there is one thing to understand:

The process.nextTick comes under microtasks which are prioritised above all other phases and thus the callback associated with it is executed just after the event loop finishes the current operation. Which means that, whatever callback we pass to process.nextTick, the event loop will complete its current operation and then execute callbacks from the microtasks queue until it is drained up. Once the queue is drained up, it returns back to the phase where it left its work from.

It first checks the microtask queue and executes the callbacks in it(process.nextTick callbacks in this case).
It then enters its very first phase (Timers phase) where the 50ms timer is not yet expired. Hence it moves forward to the other phases.
It then goes to the 'Check (setImmediate)' phase where it sees the timer expired and executes the callback which logs '3'.
In the next iteration of the event loop, it sees the timer of 50ms expired and hence logs down '1'.

Here is the output of the above code:

Consider one more example, this time we are passing an asynchronous callback to one of our process.nextTick.



function main() {
  setTimeout(() => console.log('1'), 50);
  process.nextTick(() => console.log('2'));
  setImmediate(() => console.log('3'));
  process.nextTick(() => setTimeout(() => {
    console.log('4');
  }, 1000));
}

main();

The output of the above code snippet is:

Now, here is what happens when the above code is executed:

All the callbacks are registered and pushed to their respective queues.
Since the microtasks queue callbacks are executed first as seen in the previous examples, '2' gets logged. Also, at this time, the second process.nextTick callback i.e setTimeout(which will log '4') has started its execution and is ultimately pushed to the 'Timers' phase queue.
Now, the event loop enters its normal phases and executes callbacks. The first phase that it enters is 'Timers'. It sees that the timer of 50ms is not expired and hence moves further to the next phases.
It then enters 'Check (setImmediate)' phase and executes the callback of setImmediate which ultimately logs '3'.
Now, the next iteration of the event loop begins. In it, the event loop returns back to the 'Timers' phase, it encounters both the expired timers i.e 50ms and 1000ms as per their registering, and executes the callback associated with it which logs first '1' and then '4'.

Thus, as you saw the various states of event loop, its phases and most importantly, process.nextTick and how it functions. It basically places the callback provided to it in the microtasks queue and executes it with priority.

One last example and a detailed one, do you remember the diagram of the event loop at the beginning of this blog post? Well, take a look at the code below. I would like you to figure out what would be the output of the following code. Following the code, I have put a visual of how the event loop will execute the following code. It will help you understand better:



 1   const fs = require('fs');
 2
 3   function main() {
 4    setTimeout(() => console.log('1'), 0);
 5    setImmediate(() => console.log('2'));
 6 
 7    fs.readFile('./xyz.txt', (err, buff) => {
 8     setTimeout(() => {
 9      console.log('3');
10     }, 1000);
11
12     process.nextTick(() => {
13      console.log('process.nextTick');
14     });
15
16     setImmediate(() => console.log('4'));
17    });
18 
19    setImmediate(() => console.log('5'));
20
21    setTimeout(() => {
22     process.on('exit', (code) => {
23      console.log(`close callback`);
24     });
25    }, 1100);
26   }
27
28   main();

Following gif indicates how does the event loop execute the above code:

Note:

The numbers in the queues indicated in the following gif are the line number of the callbacks in the above code.
Since my focus is on how event loop phases execute the code, I haven't inserted the Idle/Prepare phase in the gif since it is used internally only by the event loop.

The above code will output:



1
2
5
process.nextTick
4
3
close callback

OR, it can also be (remember the very first example):



2

5

1

process.nextTick

4

3

close callback

Misc

Microtasks and Macrotasks

Microtasks

So, there is a thing in Node.js or say v8 to be accurate called 'Microtasks'. Microtasks are not a part of the event loop and they are a part of v8, to be clear. Earlier, in this article, you may have read about process.nextTick. There are some tasks in JavaScript which come under Microtasks namely process.nextTick, Promise.resolve, etc.

These tasks are prioritised over other tasks/phases meaning that the event loop after its current operation will execute all the callbacks of the microtasks queue until it is drained up after which it resumes its work from the phase it left its work from.

Thus, whenever Node.js encounters any microtask defined above, it will push the associated callback to the microtask queue and start the execution right away(microtasks are prioritised) and execute all the callbacks until the queue is drained up thoroughly.

That being said, if you put a lot of callbacks in the microtasks queue, you may end up starving the event loop since it will never go to any other phase.

Macrotasks

Tasks such as setTimeout, setInterval, setImmediate, requestAnimationFrame, I/O, UI rendering, or other I/O callbacks come under the macrotasks. They have no such thing as prioritisation by the event loop. The callbacks are executed according to the event loop phases.

Event loop tick

We say that a 'tick' has happened when the event loop iterates over all of its phases for one time (one iteration of the event loop).
High event loop tick frequency and low tick duration(time spent in one iteration) indicates the healthy event loop.

I hope you enjoyed this article. If you have any questions regarding the topic, please feel free to ask in the comments. I will try to answer them with the best of my knowledge. I am, by no means, an expert in Node.js but I have read from multiple resources and combined the facts here in this blog. If you feel I have mistaken at any place, please feel free to correct me in comments.

Thanks a lot for reading.
Feel free to connect with me on Twitter/GitHub.

Have a good day! 👋

An introduction to the Domain Name System.

Sumedh Nimkarde — Sat, 04 Jan 2020 04:44:48 +0000

You all might have heard about or know about the Domain Name System (DNS) if you understand how the internet works or how computer networks work. If you aren’t familiar with DNS, I would recommend that you go and check out my previous blog post which is focused on computer networks here.

Hostnames alone cannot tell us where the particular machine/hardware that we are trying to communicate with is located in the world. Hence, all communication is done with IP addresses.

Domain Name Servers are the devices that map the hostname to the IP addresses of the machine/hardware on which your services are running.

In this post, I will be explaining in detail the types of DNS queries, types of DNS servers, and types of DNS records.

DNS Resolver

DNS Resolvers are the computers used by Internet Service Providers (ISPs) to perform lookups in their database for the particular hostname requested by the user. They then redirect that user to the mapped IP address. They play a vital role in DNS Resolution.

DNS Resolvers also cache the data. So for example, my websiteexample.com is currently hosted on a machine with the IP address 35.195.226.230 . So, the caches of the DNS Resolvers all over the world have mapped the following:

example.com -> 35.195.226.230

Consider, in the future, if I want to host my website on any another server across the world with an IP of, say, 35.192.247.235. The DNS caches of all the DNS Resolvers across the world will still have the old IP address for some time. This may lead to unavailability through conventional means of the website until the DNS propagation happens completely.

The record in the DNS Resolver cache remains there for some time, which is called time to live (TTL for short).

This is the time a record is cached in the DNS Resolver. This can be set in the registrar’s dashboard from where you have purchased the domain.

Note: from now on, I will refer to the DNS Resolver as Resolver only in this blog post.

Types of DNS servers

Root DNS server

The Root DNS servers are the ones who have the addresses of all the TLD domain servers. A request first encounters the Root DNS servers while on its journey to obtain the IP address from the hostname.

There are 13 root domain name servers across the world as of 2016. This does not mean that there are only 13 machines handling the load of the requests coming from all over the world — there are multiple servers at ground level handling the load.

Different organizations manage the Root DNS servers:

TLD domain server

These are the ones classified according to the Top-Level Domain. They are usually the next ones which the iterative query hits after the Root DNS server. They store the TLD specific records for the hostname.

Let’s say if we are requesting an IP address of medium.com , then the TLD domain servers for “.com” TLD are queried. The TLD domain servers return the address of the Authoritative DNS servers to the Resolver.

Now, the question arises: how does the TLD name server know the address of the Authoritative Name server? The answer is simple: when you purchase any domain with the registrars like Godaddy or Namecheap, the registrars also communicate the domains to the TLD name server. So it is able to contact the Authoritative Name servers.

Nowadays, some of the registrars provide the ability to use third party Authoritative Name servers. As shown in the above figure, you can set up the Authoritative Nameservers in the registrar’s dashboard.

Authoritative DNS server

These are queried iteratively in the end by the Resolver. They store the actual records for type A, NS, CNAME, TXT, etc.

Thus, they return the IP address of the hostname if available. If it is not available even in the Authoritative DNS server, then they throw an error with the particular message and the process of searching IP addresses across the Nameserver ends.

Types of DNS queries

There are three types of DNS queries:

Recursive: Recursive queries are made by users to the Resolver. It is actually the first query made while doing any DNS lookup.

The Resolvers can be your ISP or your network admin, but usually, it is the ISP in almost all cases.

Non-recursive: in non-recursive queries, the Resolver knows the answer and responds immediately without making any further queries to any other name servers. This happens because the local DNS server has the IP address stored in its local cache or it just queries the Authoritative name servers directly. They happen to definitely hold the record and this eventually avoids the recursive queries.

Iterative: Iterative queries happen when the Resolver cannot return the results since they may not have cached it. So, it makes a request to the Root DNS server. And the Root DNS servers know where to find the particular TLD domain server.

So, for example, if we are trying to obtain the IP address for say medium.com , then the Root domain server will have the address of the .com TLD server stored in it and will then send it back to the Resolver. The Resolver then asks the TLD server for the IP address. The TLD domain server may not know it, but it knows the address of the Authoritative DNS server for medium.com .

Okay, enough of the theory. Let’s understand it by a flow diagram:

Let’s break down the above diagram in steps:

The user makes a request to the Resolver with the hostname for which it wants the IP address. This is a recursive query.
The Resolver does a lookup in its cache to see if it is present in it.
If it is, it returns it back to the user.
If it does not have it cached, it makes an iterative request to the Root DNS servers that are present globally. As of 2016, there are 13 Root DNS servers named from A — M. Now, the Root DNS server looks up for the TLD of the requested domain. For example, if the hostname is medium.com , then the TLD becomes “.com” and the Root DNS server has the entry for “.com” domain servers and it returns the results back to the Resolver. The Resolver must have the addresses of all the Root domain name servers. If it doesn’t, the DNS lookup may fail in the first place.
Now, the Resolver again makes an iterative request to the TLD domain server asking for the IP address of the domain. The TLD domain server then returns back the address of the Authoritative server for the requested domain.
As of now, I believe, you may understand what are Authoritative DNS servers. They contain the actual records where the hostname is mapped to the IP address and hence the IP address is returned back to the Resolver (which in turn returns it back to the user).
If no matching record is found in the Authoritative Name servers, then an error with a message saying “DNS_PROBE_FINISHED_NXDOMAIN” is thrown indicating there is no record for the requested hostname.
In all the Nameservers the request passes through, the results for the requested hostname are cached so that when any other user requests the same domain, the record will already be present in the DNS cache.
All in all, it takes at the max four queries to perform the DNS lookup. But, it hardly takes a few milliseconds to perform the lookup.

The concept of DNS Propagation

Consider, you have your website hosted with some provider like Digital Ocean on any machine with IP “x”, and you want to shift the website hosting to any other machine with different IP address say “y”. You will have to change the IP address in the Authoritative records so that traffic navigates to the new IP address.

Even if you update the records in your registrar’s/ name server’s dashboard, it takes some time to reflect in all the Resolvers’ caches in the world. DNS propagation can take 24–72 hours, but usually it happens sooner than that since most ISPs keep the TTL low.

And that’s it!

Thanks for reading the article. If you have any questions, please feel free to ask them in the comments below and share this post with whomever you want.

See you in the next one. Thank you!

Feel free to reach out to me on Twitter.

What are computer networks and how to actually understand them

Sumedh Nimkarde — Sat, 04 Jan 2020 04:34:08 +0000

Whether you are new to the world of development, or have been building things for a long time — or even if you’re a person who just likes computers and uses the internet daily — you’ve got to know the basics of networking and specifically Computer Networks.

If you like digging more into servers, their security, and how you connect to your servers from a remote client, all of this requires some knowledge of computer networks and their components. I have tried to cover most of the topics concerning computer networks in this article.

Also, from here, I will refer to “computer networks” simply as “networks”.

Let us first look at my working definition of computer networks:

Computer networks can be defined as the exchange of network packets between computing machines across the world with the help of data lines like wire cables, optical fibers, etc.

The Internet is a kind of computer network. Sorta.

We will take a look at some commonly used terms and components and how they function in a computer network, some of which are in the above diagram.

Commonly used terms in Computer Networks

Nodes

Nodes in computer networks mean any computing device such as computers, mobile phones, tablets, etc which try to send and receive network packets across the network to another similar device.

Network Packets

Network packets are nothing but the information or units of data that a source node wants to send/receive to/from the destination node. In this article, network packets/data packets all convey the same meaning.

Internet Protocol (IPs)

Consider you want to send a birthday gift to your friend on their birthday, where will you send it? To their street address right?

Same is the case here. The early computer scientists wanted to identify computers on the internet with a unique number, something like a telephone numbers today. So, they came up with the concept of TCP/IP.

An IP of a computer device is the address of that device in a computer network. Technically, it is a 32-bit number used which identifies devices in a network. All the communication to and fro from the device in that network will be done in terms of its IP address.

Consider that you are uploading a file to any site or say to Google drive.

Talking at the lowest level of network communication, your file is converted to packets and each packet has the destination node address with it which is nothing but the IP address.

On a higher level, IP addresses are classified into two types:

IPv4: IPv4 addresses are 32 bits (four bytes) as explained in the definition. An example of the IPv4 address would be 104.244.42.129 which is the IPv4 address of twitter.com. They are stable to use and hence are used today to identify machines in the world.
IPv6: IPv6 addresses are pretty new to the world and are basically eight hexadecimal numbers separated by “:”. An example of IPv6 address would be 2001:0cb8:85a3:0000:0000:8a2e:0370:7334. They are unstable and hence not used widely yet. The web is still using IPv4 due to its stability and there is no estimate when we will start to use IPv6 since it is not stable for now.

IPv4 is classified into five classes named Class A, B, C, D, E.

Class A: As shown in the third column of the above image, for a Class A IP addresses, the first bit of the first octet of the IP address is constant and is “0”.

The Second column indicates the Network bits and the host bits of the corresponding class of IP address. Consider in case of a Class A IP address, we have the following formula:

Number of networks/subnets = 2^(# of network bits).

Number of valid hosts in each subnet = 2^(# of host bits) — 2.

The number of network bits and host bits are decided by the default subnet mask of the class of IP address.

The default subnet mask for a class A IP addresses is 255.0.0.0, that is 11111111.00000000.0000000.00000000. Thus, for class A:

Network bits = 8, and Host bits = 24.

Since Network bits = 8, Host bits = 24, their sum has to be 32, since IPv4 addresses are of 32 bits. But, since we are using the one bit (first bit in the first octet) to identify the class:

Number of usable network bits = Number of network bits — Number of constant bits = 8–1 = 7

Thus, the Number of possible networks in Class A = 2^7 — 2 = 126 and,

Number of possible hosts (that is devices that can be connected to the network) per network in Class A = 2^24-2 = 16277214.

Now, here, for class A, you may wonder why I subtracted an extra 2 from the number of possible networks. It is because, for class A, 127.x.y.z was kept reserved. For other classes, the usual formula is used.

Thus, IP addresses in class A range from 1.x.x.x to 126.x.x.x.

Class B: the case is similar with Class B. The only difference is 2 bits of the first octet are constant (10) and they identify the class of IP address that is class B. All other calculations are same, and I am not mentioning them here since they are easy to grab from the table above. They range from 128.0.x.x to 191.255.x.x.

Class C: 3 bits of the first octet are constant (110) and they identify the class as class C. They range from 192.0.0.x to 223.255.255.x.

Class D and Class E: Class D and Class E are used for experimental purposes.

IPv4 addresses are mainly of two types:

Static: These IP addresses are the ones which remain constant for a device over time. Examples of these are the remote servers that we use to host our apps, websites, etc. where we use the ssh client to ssh to our server.
Dynamic: Generally, these are the IP addresses that a common computer in an Internet network is assigned. Try switching your router off and you will see a change in the IP address of your computer! (But only after reading this article 😛). Now, you may be thinking who allocates these IP addresses? It is the DHCP (Dynamic Host Configuration Protocol) server which is explained briefly further in this article.

Note: A device can have multiple IP addresses at the same time. Consider a device connected to two networks, wifi as well as any LAN network — it will have two IP addresses. This implies that the IP addresses are assigned to the interfaces and not directly to the computer.

Okay, so far so good. Let’s continue.

Routers

As its name suggests, a Router is a hardware component that takes care of routing packets. It determines which node the packet came from and which destination node the sender node want to send it to. No computer knows where other computers are located, and packets are not sent to every computer. A Router identifies the destination node address to which a network packet has to be sent and it forwards it to the desired address.

Routers have a specific “Routing Protocol” which defines the format in which they exchange data with another router or networking nodes. In other words, routing protocol defines how routers communicate with each other.

Routers build up a “Routing Table” which identifies the most optimized paths to be taken in the network while sending packets.

Technically, a routing table is just a table with the list of “routes” from one router to other. Each route consists of the address of the other routers/nodes in the network and how to reach them.

Destination	Gateway	Genmask	Flags	Metric	Refs	Iface
default	192.168.0.1	0.0.0.0	UG	1024	233	eth0
192.168.0.0	*	255.255.255.0	UC	0	0	wlan
192.168.0.0	*	255.255.255.0	UH	0	2	eth0

Above is an example of a routing table. The key points to take a note of here are:

Destination: This is the IP address of the destination node. It indicates where the network data packet should end up.
Gateway: Gateway is the component which connects two networks. Consider that you have a router connected to another router. Each of the routers has devices connected to it. So, the address of the last router (say R1 here) after which the network packet enters the other network (say R2’s network) is called the gateway. Usually, the gateways are nothing but the routers. Let me give one more example: say that your room is one network and your sibling’s room next to yours is another network, then the “door” between the two rooms can be considered the gateway. People sometimes refer to the “routers” as the gateway, because, that’s what they are, “a gateway to another network”.
Genmask/Subnet mask: It is nothing but the net/subnet mask. A subnet mask is a number which when combined with an IP address allows you to divide the IP space into smaller and smaller chunks for use in both physical and logical networks. The explanation of how subnet mask calculations happen is beyond the scope of this article.
Flags: Different flags have a different meaning. For example, in the first route, “U” in “UG” means the route is UP, whereas “G” in “UG” means GATEWAY. Since the route signifies a GATEWAY, it is a door to the other network. Whenever we send any data through this route, it gets sent to another network.
Iface (Network interface): Network interface refers to the network that the route defined in the routing table is having the destination computer in. That is if you are connected to Wifi, then it would be “wlan” and when you are connected to a LAN, then it would be “eth”. So this is the way a router works, with the help of Routing Protocol and Routing Table.

All good up to now. But, you must be thinking —

“Okay! But hey, we are learning about components here. I need to stitch them together and get to know how the internet works.”

Cool! Some more terms and you will have a proper understanding of how everything goes.

Network Address Translation (NAT)

Network address translation is a technique used by routers to provide internet service to more devices with less usage of public IPs. Thus, a router is assigned a single IP address by the ISP and it assigns the private IPs to all the devices connected to it. NAT helps the ISPs provide internet access to more consumers.

Thus, if you are connected to the router of your house, your public IP will be visible to the world, but the private one will not. Whatever network packets are communicated will be addressed by your public IP (that is the public IP assigned to the router).

Consider the above figure. Let’s say that in your home network, you are trying to access medium.com (remote static IP: 72.14.204.147), from your computer (private IP: 192.168.1.100).

So, for your computer, the connection looks like:

192.168.1.100:37641 → 72.14.204.147:80.

“37641” is the random port number assigned by NAT router to your device/computer. (When there is network communication between daemons running on different ports on a computer, the respective port is used by NAT). Each outbound connection gets an assigned port by the NAT router.

The connection is established in NAT like:

Private Ip	PrivatePort	PublicIP	PublicPort	Remote	RemotePort
192.168.1.100	37641	104.244.42.129	59273	72.14.204.147	80

But, since the outside world of the network doesn’t know about your private address, the connection looks like the following to medium.com:

104.244.42.129:59273 → 72.14.204.147:80.

That way, we achieve assigning a higher number of IP addresses without wasting many public IPs.

Now, when medium.com sends the response back to 104.244.42.129:59273 , it travels all the way to your home router which then looks up for the respective private IP and private port and redirects the packet to your device/computer.

Note: NAT is a generalized concept. NAT can be achieved as 1:1, 1:N where 1, N are the number of IP addresses in the network. A technique called as “IP Masquerading” is a 1:N NAT.

Dynamic Host Configuration Protocol (DHCP)

Dynamic Host Configuration Protocol or DHCP is responsible for assigning dynamic IP addresses to the hosts. The DHCP server is maintained by the ISP or previous router if there is a chain of routers to reach the host.

Thus, allocation of IP addresses is carried out by the DHCP server. Generally, ISP maintains a DHCP server and the routers in our houses get assigned a public IP from the DHCP server.

Note: Whenever a router or say a DHCP server maintained by an ISP or router restarts, the IP address allocation starts again and devices are allocated IPs which are different than the previous ones.

Domain Name System/Server

We have already discussed that any machine is identified by the IP address.

Okay, so you are running a web server on your localhost on your machine. If you have dug around in the hosts on any Linux machine, you would have encountered something like this:

127.0.0.1	localhost
255.255.255.255	broadcasthost
::1	localhost

which means that even if you type 127.0.0.1 in your browser’s URL bar, it would mean the same as localhost .

Similar to the above, the websites you use daily are web servers running on some remote instance/node having a static IP address. So, typing that IP address in your browser’s URL bar will take you to the website?

Yes, surely it will. But, are you a superhuman to remember the IP addresses of thousands of sites?

NO.

Thus, there come the domains that we use, say medium.com, twitter.com, behance.net, codementor.io, etc.

A Domain Name Server is a server having huge records of domain name mapping IP addresses which searches for the domain input and returns the respective IP address of the machine on which the website you want to access is hosted.

How does DNS work actually?

DNS is managed by your ISP (internet service provider).
When we type an URL in the address bar, the data packets travel through your router, maybe multiple routers to your ISP where your DNS server is present.
DNS server present at the ISP looks up for the domain in its database. If an entry is found, then it returns it.
If any entry is not found in its primary database that it maintains, the DNS server will travel through the internet to another DNS server maintained by another ISP and check if the entry is available in that another DNS server’s database. Along with returning the IP address taken from another DNS, it will update the primary database with this new entry also.
Thus, sometimes (very rarely) a DNS server may have to traverse to multiple DNS servers to get a matching entry.
If after traversing a lot of DNS servers across the internet, it doesn’t get a matching entry, then the DNS server throws an error indicating that the “domain name is invalid or doesn’t exist”.

Note: The Internet Corporation for Assigned Names and Numbers (ICANN) is a consortium (a non-profit corporation) that manages the assignment of domain names and IP address ranges on behalf of the community.
A domain is divided into three parts as shown in the following figure.

Protocol: The protocol used to access the website, for example, HTTP, HTTPS, etc.
Domain name: The main domain name in our domain. This can be anything that is available as per the ICANN registry.
Domain extension: This is one which is considered important while buying a domain. Generally, it is classified into two types:

Generic Top-level Domains (gTLDs): This includes most popular domain extensions like .com, .org, .net, .edu, .co, etc.
Country Code Top-level Domains(ccTLDs): These indicate that the domain is related to the country code specified in the domain extension. For example, “.in” indicates that the website is originated from India. Also, some of the ccTLDs require that the person purchasing the domain should be from the same country. Most of the small country code extensions are not searchable from outside that country.

Internet Service Providers (ISPs)

Internet Service Providers are the companies that provide everyone Internet. The article you are reading now is because of the internet that your ISP provides you.

ISPs provide internet, handle routing your requests to the correct destination, resolve domain names with the help of DNS cache that they maintain, and handle all this network infrastructure which enables us to use the internet.

ISP is a hierarchical thing working across the internet. There are certain types of ISPs namely Tier 1, Tier 2, Tier 3 ISPs.

Tier 1 ISPs are the ones which connect major networks on the internet. Consider them as the major highways of the internet. They are connected to almost every network on the internet. Also, they provide internet access to the Tier 2 ISPs. ex. CERFNet, UUNet, PSINet. They are also called Network Service Providers. These ISPs are connected to each other by means of large cables going beneath the sea.
The Tier 2 (Regional) ISPs are the ones who primarily provide Internet services to organizations, consumers (that is “us”) or the Tier 3 ISPs. The internet connection you are using is from a Tier 2 ISP. However, organizations can also get Internet access from Tier 1 ISPs.
Tier 3 (Local) ISPs are just like Tier 2. It’s just one more level of hierarchy out there that purchases bandwidth from Tier 2 ISP and sells it to consumers.
The traffic that goes through your router also goes through Tier 3 (if present), Tier 2, and ultimately through Tier 1 ISPs all the way to another network.

Woot Woot! I am happy that you are still with me. We will put all the things together now.

Putting all of the above things together

Up until now, we have learned about all the components needed to make everything work. Now, we will glue them together.

Let’s summarize all the things we’ve learned:

When a computer/device comes online, it gets a private IP assigned by the router. The router gets a public IP from the ISP. Other devices in the network are allocated unique private IPs.
ISPs are the ones who are present across the world and are connected to each other. They sell Internet services to the regional and local ISPs, from whom we, the consumers, purchase Internet.
Thus, when a device tries to establish a network connection with some other device on some other network, it does it with the identity of its gateway (the router). The router then maps the private IP and private port number with the public IP and random high integer public port number.
The router then sends the packets to the desired destination where some other router or gateway does the same thing as the previous router and analyses which computer/device that packet came from.
The remote computer/device responds by sending the destination as the public IP and public port of the router.
The router then again checks for the private IP and private port and forwards the network packets. So, this is how the Internet aka A kind of Computer Network using TCP/IP protocol works.

Thanks for reading the article. If you have any questions, please feel free to ask them in the comments below.

If you liked this article, just message me on twitter(dms are open) and share it with others. See you in the next one. You are awesome!

Thanks a lot again for reading! Feel free to connect with me on Twitter, GitHub.