DEV Community: Nicolas Kirchhoffer

Expose your Kubernetes services from your home network

Nicolas Kirchhoffer — Sun, 07 May 2023 21:24:55 +0000

The ultimate goal for a Cloud Engineer and a Sysadmin who is interested in DevOps is to setup a Kubernetes cluster on their servers to manage applications cycles.

On bare-metal infrastructure (and homelabs), this can be quite a challenge, as we do not have any public cloud service to help us.

Using Load Balancers inside of our Kubernetes cluster seems impossible without a proper infrastructure. We do not have any BGP router, nor do we have a physical Load Balancer to guarantee failover.

In this article, I will be trying to give you a correct solution to expose your Kubernetes cluster to the Internet safely and for free.

The problem

Every person that has tried to setup a Kubernetes cluster on a bare-metal infrastructure has encountered the <pending> status under the External-IP field on their LoadBalancer services... and the headaches that followed.

Actually, on Public Clouds, the Load Balancing products (Cloud Load Balancing on Google Cloud) assign IP addresses to your Kubernetes services. Without one, Kubernetes doesn't actually know which IP address to assign to a specific Service.

The solution

Fortunately, the Kubernetes ecosystem is quite vast and a solution to this specific problem has been developed, it is called MetalLB.

Note that the L2 mode of MetalLB doesn't guarantee a zero downtime failover, because of the ARP protocol. While the ARP cache is not refreshed, the users would hit a specific node that is no longer up. It can sometimes last many minutes.

To install MetalLB, please refer to the documentation here : MetalLB, bare metal load-balancer for Kubernetes

MetalLB (L2 mode) uses Address Resolution Protocol (ARP) to associate a Service to a specific IP address. This allows Kubernetes to balance the load using the internal kube-proxy component that establishes a connection to a specific Pod.

In practice, we tell MetalLB to use a specific pool of IP addresses, 10.0.0.100-10.0.0.200, to expose services. The nginx service would have its External IP set to 10.0.0.100.

Precision : An ARP request is broadcast (given that we don't know who is at the specific IP address), but the previous diagram has been simplified. The concerned MetalLB Operator (not all MetalLB operators) responds to the ARP request by giving the MAC address of the Service and the Pod of the application. The user can now access the application. Although, the user and the Kubernetes cluster must be located in the same broadcast domain in order for it to work.

Read more about MetalLB L2 (and its limitations) here.

To indicate an IP Address pool to MetalLB, we can use the following manifest :

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: home-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.100-192.168.2.200

We must also tell MetalLB to advertise this pool using Layer2 (and not BGP) :

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: home
  namespace: metallb-system
spec:
  ipAddressPools:
  - home-pool

The given IP addresses must not be in use. It would lead to a conflict, and thus, connectivity issues. To avoid any problem, I recommend you to exclude this pool from your DHCP range.

Now, MetalLB will assign an IP address to your service, which you can get with kubectl get svc :

Your service is then exposed on your local network. You can access it directly, but you might want to expose it on your domaine name.

You can also install an Ingress Controller to route your different domains and subdomains !

Expose your service to the Internet

Historically, a simple solution was to setup a PAT, which means to bind a port of your public IP address to a specific port of a private IP address.

This solution has different limitations :

You would be exposing your home network
Your public IP address might change (depending on your ISP)
You would not always be able to bind a port to a virtual IP address

A possible solution to avoid these limitations would be Cloudflare Tunnels. It is an agent that you can install on a machine of your home network, which will establish a connection between your network and Cloudflare, like an IPSec VPN for instance.

This method has many advantages :

Cloudflare would be acting as a reverse proxy, thus, would not expose your home network
Cloudflare would be managing your public TLS certificates (no need to renew them every 3 months !)
No configuration is required, the agent install being quite easy

To set this service up, you would need to configure the NS fields of your domain name to the Cloudflare servers. Follow the steps on Cloudflare Dash. You can then go to the "Zero Trust" category and choose a plan (the free one preferably). Then go to the Tunnels page under the Access category.

Click on the Create a tunnel button, give a name to your tunnel (for example: home), and choose an install method for the agent. I recommend using Docker and to host it outside of your Kubernetes cluster, so that, if your cluster goes down, the tunnel remains active. Please note that if the agent is down, so is your website.

When the tunnel status becomes HEALTHY, you can associate a subdomain (or your main domain) to a private IP address (and even to a specific port if necessary). Cloudflare will host a proxy to your local address via a subdomain, then configure a CNAME field to expose it to your (sub)domain.

If you use an Ingress Controller, you would need to set the value of the Host HTTP header manually, under the HTTP Settings category. Your controller will then know which service you try to access.

Alternative solution

Normally, I would have talked about a free and open-source solution to resolve this specific problem. However, given the current Internet state and the difficulties to get IPv4 addresses, the easiest way of solving it was to rely on a 3rd-party provider. Here, Cloudflare. I do not have any specific relation with them, other than being a customer.

If you do not want to rely on a "Giant tech company", you could use the dynamic DNS principle to assign, even if it changes, your public IP address to a DNS A field, using PAT. This does not cancel the other limitations.

Note that you would always rely on a provider for your domain names. This is only a question of opinions. Cloudflare being an omnipresent company on the Internet, and defending freedom of expression, I feel okay using their services.

But that's your choice !

Make your personal project a real product

Nicolas Kirchhoffer — Sun, 28 Aug 2022 00:18:00 +0000

Personal side projects are popular in 2022, they allow to build portfolios and add value to our job applications, or they simply became a hobby for people, as it did for me, who like to build things for fun in their free-time. But, sometimes, we feel like we've had a good idea, to the extent that we want to deploy it and sell it to potential customers. This series', illustrated with examples and my own personal experience, will tell this process from A to Z !

This series of articles is dedicated to developers who like to play with many technologies. Every technical concept will not be entirely covered (like the actual application programming, for example).

Landing Zone
Step 0 : Believe in your project
Step 1 : Write a "State of the Art"
Step 2 : Define specifications
Step 3 : Conduct a market study
Conclusion

Landing Zone

In the beginning of this year, for my birthday, I had this crazy idea, but easy to make : I wanted to allow my friends to add songs to my Spotify queue, as I was tired with this choir during our parties. After building a prototype, I rolled it out for this specific party. I was very enthusiast and happy, I felt like I have made something useful, and I got many positive feedbacks to this early version, and also many ideas, which was the most important.

After discussing with potential customers and other enthusiasts, I decided to deploy this project in order to sell it online. This series of articles will relate my feedback on this process, as I do so !

It mainly addresses the technical challenge behind a production deployment, specifically when you have a "draft" prototype of your future product. But I wanted to share a global point of view around this process, so this first article will be about the conception of a perimeter around the application in order to put it on the market.

Example

In order to have specific examples, while not actually messing with my main project, I built a basic app which will be useful to illustrate the different concepts and arguments. It is a website that allows its users to create to-do lists, and to share it with others in order to collaborate over it. Very simple, in fact. Here is a preview of the first version.

This application consists of a front-end and an API. The front-end has been built with Vue, Vue Router and Bulma. The API has been designed using NodeJS with Express and uses MongoDB to persistently store user data.

The code source is entirely free and available at the following repositories :

We are now ready to begin the process of deploying our to-do lists prototype !

Step 0 : Believe in your project

This step really isn't one, that's why it's called "step 0". However, it is essential to successfully deploy your project.

Believe in your ideas !

If deploying this project crossed your mind at some point, it is legitimate, you believe in its potential and in its capacity to solve your customers' issues. You must not give up !

Seek for people who you can trust and that will help you to complete this challenge. Having constructive, not necessarily positive, feedback is really helpful. It allows you to enlarge your point of view, and to not fall in the confirmation biais. Having good friends and people to trust is essential to have the energy to stay focus in your objectives.

Step 1 : Write a "state of the art"

The State of the Art allows you to understand the current situation of your project and to where you want to go.

Even if, when you decide to deploy this project, you don't have completed much technically, it is always interesting to do this little exercise in order to have a global point of view of your future product, and of the steps to complete before you could really begin to work on it.

In my case, given that my personal project was already in a "prototype" state, I have a code base that I can use and a set of features that will be the application's backbone.

Here is the framework that I followed for my project :

What are the features currently implemented on my prototype ?
What are the technical limitations of my prototype ?
What are the positive aspects of my prototype ?
Do I have an identity for my product ?
- A name ?
- A brand ? (graphical)

Example

Applied to our "to-do lists" example project, here is an example of State of the Art :

Features already implemented on the prototype
- As an user, I can create my own to-do list with a name
- As an user, I can share a to-do list with its unique URL
- As an user, I can create a task by referencing its textual objective
- As an user, I can mark a task as read
Technical limitations
- There is no authentication management, nor users, nor permissions
- The API is not versioned
- There is no test nor codebase standards set
Positive aspects
- My code source allows for a simplified scaling framework
- My code structure allows to make important structural changes (for example, a database change)
This product doesn't have a name, nor a logo, but the emojis is a defined apsect of the design.

Answering these questions will allow you to establish precise specifications of your final product and will thus help you to position yourself on a potential market.

Step 2 : Define specifications

Specifications of your product reflects your own vision of it. It must indicate every little detail that define your product, to whom it is targeted, who will use it, etc.

Specifications must be the result of a challenged thinking about every aspect of your final product. What is the use of this specific feature, what is the targeted audience for the application ?

Don't hesitate to think about every question you have in mind in order to define your specifications. The exercise is even more legit when it is done with other people, you will avoid confirmation bias.

Here is how I redacted my specifications :

Product's name
Quick description
Targeted audience (social category, for example, students, a video games forum, or even certain types of professionals, for examples, coffee shops, bars/pubs and restaurants).
Users types
- Your team (who will manage the product - the back-office)
- The customer (the one who pays for it)
- The end user (could be your customer - they use the application)
Define every feature as User Stories (put yourself in the shoes of a specific user)
- As an end user, I want to add a Spotify song to the distributed waiting queue
- As a customer, I want to be able to see my subscription status, the billing and to manage my offer
- As a team member, I want to see the turnover of the product in a given time period

Example

The specifications for the example project is given here : Todoing.pdf

There are no limits to true specifications, everything you want your product to be should be in those. They are the written copy of your final product, it will guide your teams through its conception and is what you will be selling to your customers.

It is even recommanded to include your graphic brand in your specifications, but your market study is the step that will really help you build your brand. However, it is always helpful to consider a potential brand name and to buy the domain name accordingly fast.

Step 3 : Conduct a market study

Now that the perimeter of your final product is defined, you can begin the most determining step of your project. Without it, it would be like sailing a ship without compass, in troubled waters in which you can easily get lost.

It is important that your specifications are defined and precise enough to conduct a market study, otherwise your conclusion might be based over incomplete deductions. With non-complete specifications, you can easily confuse your product with another. For example, by defining very brief specifications of Twitter, that I simply describe with "Post content on the Internet", we can easily deduce the same specifications of Facebook or any other social media.

Internet is already really vast, and your product probably already exists in another form if you specify brief specifications. Everything resides in nuances, that will allow you to distanciate yourself from your potential competitors.

Don't engage with projects that are too specific though, you must be able to evaluate the number of people that you will target by previously defining your target audience. If it is too broad, implementations of your product probably already exist, if it is too precise, you won't be able to make a consistent change on the market.

To conduct a good market study, don't hesitate to browse public databases for brands and patents (like Google Patents). Make Google searches of products that are close to yours, by searching with keywords and concepts that you can affiliate your product with.

If you don't want to spend much time yourself on such a thing, you can always seek for external help as many companies will conduct it for you.

If you do conduct it by yourself, here are the points that I think are important to make a good market study :

Analyse the actors of your potential market, write profiles about them, about the products they sell and the audiences they target
Compare their products to yours, clearly establish the nuances by writing them
Compare the target audiences and which type of population they don't target, so that you can distanciate yourself from them on specific markets

If, in every aspect, your product is similar to an existing one from a competitor, and that you do not have prior revolutionary technique to sell at a lower price (we will come back to this later), it might not be advised to launch your own product if you don't provide with further features or imlementations.

If you do find a legit nuance that can become a marketing argument, then your product may be legit ! You can continue the deployment process without making important changes to your specifications.

Example

For the to-do lists, the competition is tough ! There are many implementations of them that cover the same specifications as we do.

The actors : Todoist, Trello, Microsoft To-Do, Google Tasks, and many others...
Comparisons with our product
- Todoist : users can create diverse lists, collaborate. It has more features than what our specifications cover. It is available as a web browser extension, on mobile devices, as a website... (also has a professional version)
- Trello : allows people to organize ideas as many lists on a board, allows them to attribute tasks to other collaborators, allows them to set a deadline and to put them in categories, and much more..., for example its vast plugins marketplace
- Microsoft To Do : allows the same things as Trello and Todoist (has a business version)
- Google Tasks : Google implementation of to-do lists, having more or less the same features as its competitors, allows users to create tasks from the Google Workspace chat

Every solution has a business alternative, however, these products offer so much features that they can not be considered "minimalist". This is the selling point we will stay focus on for the following articles.

In all seriousness, it would be quite a bad idea to still decide to launch a to-do lists product after analyzing its market... pretty much every implementation of them have been done

Once you have determined (or not) your position in a precise market, of which you know the actors and the stakes bound to each one of them, you can developed a true brand : a name, logo, visual identity - colors, photos... - catch phrases, etc...

The order to first define specifications then conduct the market study seemed logical to me. However, it can be interchanged if you are not sure about the pertinence of your own implementation. Both must use one another to be useful for your decision making.

Conclusion

We have now completed some steps that will define a perimeter for our project before launching it. Even if they can be boring, I tried to make you understand the stake of doing them so that you can elaborate a true profile of your product.

At the end of this article, we have :

An elementary prototype (that we had at the beginning of the article...)
A State of the Art indicating elements already in our possession
Specifications that define what we want our product to be, it will influence - and be influenced by - our market study
A list of actors, of their products, of their relations inside a specific market, which will define the marketing strategy for our product

The next article will be about the application conception, of a technical and architectural point of view. We will be speaking about critical aspects such as the distributed architectures, scaling, user management and security throughout an elaborate permissions system.

Why can't Serverless really be serverless ?

Nicolas Kirchhoffer — Sat, 21 Nov 2020 10:56:01 +0000

The Serverless architecture has brought some attention to its terminology while it became more and more popular. It went for a time before someone discovers that, actually, serverless applications run on servers !

To that discovery, we observed 2 major reactions :

The ones who have known since the beginning, and didn't tell
The ones who didn't know and whose life meaning is now compromised

Before I begin explaining why, really, Serverless can't be serverless, you should be familiar with this cloud concept and service. If you are not, here are a few articles on DEV that I strongly recommend you to read :

Building RESTful APIs on Serverless architectures

Build a RESTful API with the Serverless Framework

Sagar ・ Oct 19 '18 ・ 6 min read

#serverless #node #javascript #beginners
10 ways to use Serverless functions

⚡️ 10 Ways to Use Serverless Functions

Nader Dabit ・ Jan 8 ・ 10 min read

#serverless #javascript #webdev #aws

Please also note that Serverless is a type of architecture, which means that it is not basically standardized between providers, serverless solutions on Heroku differ to App Engine on Google Cloud Platform or Lambda on Amazon Web Services.

However, the discussion around clouds and deployment ends here, because this post is about systems and networks, and how the Internet doesn't really allow massive deployments without servers (blockchains and torrenting apart).

The "server" principle

I think you are mostly aware of the concept of servers, and what they are. In fact, they really are computers designed to run efficiently to deploy services to end-users and also to run 24/7 with less downtime as possible.

Servers are massively used, every website you browse to is hosted on a server, the DNS system you use to resolve hostnames is hosted on servers, and so on...

But can we actually deploy a service without them ?

I mean, can't we find architectures to share data without having centralized servers ? Can we think, for example, of a distributed network to expose services ?

We will try to answer those questions on the next sections, but first, let's talk about the already known distributed services (and networks).

Distributivity and the web

The most common example for distributed networks is the old famous BitTorrent protocol, which allows for file sharing without having servers ! That is awesome, right ?

Yes, it is. But let's run through the architecture of a typical BitTorrent network.

A BitTorrent network, with the uploader in the center and the downloaders around it

To share a file on BitTorrent, a computer (then called "the uploader") must have the whole file on its system. It will be originating the file sharing, by "seeding" it to the other computers. Once a computer has a bit of the file, it can also share it to the other computers, reducing the trafic on the uploader, and augmenting the seed of the file.

Of course, when downloaders have successfully downloaded the whole file, they are asked to do the same. BitTorrent has a "mesh" logical topology, which means that every PC on a BitTorrent network is able to have a communication with every other PCs connected to it.

The dream comes to an end, with no regulation whatsoever, this network isn't possible. You need to have what's called a "tracker", to coordinate the computers in their communications. And even, on some networks, to regulate the ratio download/upload, to avoid having computers that download files and then leave the network once they got it, avoiding the other computers to get it from them. Which is kind of the whole purpose of a Peer-to-Peer file sharing system. Peer-to-peer means that you talk directly with the computers, with no server between the two.

But, it is still possible to distribute the tracker with the DHT (Distributed Hash Table), but a bootstrapper server is still needed. So, while you don't have a direct communication with a server when you are sharing or receiving a file, you still need servers to organize the whole network.

Besides, the BitTorrent network requires a dedicated application that is built especially for this purpose.

The dream definitively comes to an end, because the web really is not designed for distributed architectures, well, in a way of having clients being servers and being able to reverse roles when they want.

The HTTP server, on top, distributes data to the clients, on the bottom

The HTTP protocol relies strongly on the server/client architecture, having an "unique" (or a distributed cluster) server that is responsible for sharing the data to the clients, sometimes with access restrictions or specific policies.

That is the key with HTTP and having central servers, you can have policies and access control. While we can imagine redesigning HTTP to be distributed on a "BitTorrent-like" network, it is not possible to have control over the data anymore, and it is causing a lot of security issues, like :

Checking the original data integrity
Regulating payments or financial informations over the distributed network
Not divulgating the source code to everyone
Restricting access to staff or specific users
And so on...

One way to get over some of these aspects would be to use blockchain to check the integrity of the data shared over a distributed network, one very good example is the IPFS (InterPlanetary File System). I am not an expert of IPFS and will not try to debate over the fact that it can, or not, replace HTTP securely and keep all the aspects listed before, but it is to consider.

The main key for the whole Serverless architectures to be still deployed with servers is that you can use simple old protocols like HTTP and still be universally compatible with the browsers everyone uses, avoiding to have billions of people installing new softwares just to serve its purpose 😄

A final word

We had a brief, but quite explained, look over the simple architecture of the web that does not compare to a distributed one, and thus, proving that serverless, on today's WWW, is not possible.

But, by publishing this post, I break the whole Serverless principle : to not care over servers. It comes in a simple and humble way to remove server complexity for the DevOps engineers or even developers in smaller teams, they don't have to care about the server, the cloud platform does it for them.

I don't know if people really wanted to know the reason why we still use servers for Serverless, but here it is if you wanted it 😆

If you want to learn more about networks, topologies or communications between computers, check out my network series :

Building networks from A to Z - Part 1 : the basics

Nicolas Kirchhoffer ・ Oct 25 ・ 4 min read

#networks #basics #computerscience #beginners

Building networks from A to Z - Part 6 : Transporting data

Nicolas Kirchhoffer — Fri, 13 Nov 2020 20:35:14 +0000

This post is the part 6 of the "Building networks from A to Z" series, I would really recommend you to go read the 5 other parts if you haven't, you would not understand all concepts explained here if not. Thank you !

We are now talking about OSI model layer 4, which is called Transport. On the TCP/IP model, most known by developers, Transport is the 3rd layer, and is also very important to the programming world, for reasons we will explain later.

From this layer comes the very known term "port", which is a logical address used by your system to know to which program the distant computer wants to talk.

Two protocols are really used in this layer, which are TCP (Transmission Control Protocol) and UDP (User Datagram Protocol). There are main differences between the two of them, that I will explain later, let's first speak about computer ports.

In every Operating System, the port system is a really important notion that is defined by the network, meaning that you will have no major differences between the different OS.

Ports addressing

Ports are assigned by the IANA (Internet Assigned Numbers Authority), that also assigns IP addresses to companies around the world. It is a branch of the wider organization of the ICANN (Internet Corporation for Assigned Names and Numbers), that regulates the worldwide Internet. They are also responsible for the Domain Name System root servers, which will be introduced in the next part.

We have different ranges of ports, explained below :

Ports 0 to 1023 are the well-known or reserved ports, which are assigned to specific programs or protocols by IANA. Among them, you have :
- 80 and 443 ports reserved to, respectively, HTTP and HTTPS. They serve the web.
- The 53 port is widely used by the name-resolution protocol, called DNS. It is used to perform queries.
- Ports 25/465 and 143/993 are used, respectively, for SMTP/SMTPS and IMAP/IMAPS. Both of these protocols are essential to e-mail communication.
- Port 21 is used for FTP control (File Transfer Protocol). For FTP control over SSL/TLS, port 990 is used.
- Port 22 is used by SSH (Secured SHell), this protocol allows you to run a distant shell to a machine that hosts a SSH server.
Ports 1024 to 49151 are user ports, they can be used freely by the users without risking conflicts with other programs. 8080 is the unofficial admitted alternative port to HTTP when programming websites locally, also, Minecraft servers often use the 25565 port to establish connections.
Ports 49152 to 65535 are dynamic ports that are used by programs to create sockets (or connections), they are almost always by the client-side program to connect to a server. They are dynamically attributed and tend to change a lot, they are not stable so you shouldn't use them manually !

As you can see, ports are encoded on 16 bits (2^16 - 1 = 65535). They are unique to all TCP protocols, so UDP and TCP have to share them.

Data transport

The main principle of the Transport layer consists in a router inside your computer that will distribute data to concerned programs :

Two programs communicating with each other over the local network

Please note that the "router" seen here is only used as demonstration purposes. It has not the same features than a Layer 3 router, for example, the source port is not changed when going through the outgoing router. Its only role is to find the program associated with the destination port.

Transport reliability and connection

The main difference between UDP and TCP is what we call the "connection". On the transport layer, a connection is a mechanism that will ensure that the two hosts are able to communicate by making sure they receive the messages each time they are supposed to. It seems quite difficult, so I will explain the whole connection mechanism over a diagram.

TCP connection establishment mechanism

The host willing to communicate over TCP sends a "SYN" (synchronize) packet over the network. When the receiver receives it, it sends a "SYN-ACK" packet to the first host (synchronize, acknowledgement) which sends an "ACK" packet to the second host.

This mechanism allows for both hosts to be sure that their packets will be received and that the network is reliable. Internet Protocol DOES NOT provide any reliability at all, which means that TCP is the only responsible of this feature. Lower layers ensure that the data is electronically emitted and that the data integrity is preserved, but they will not ensure the IP packets aren't lost, which TCP monitors.

When the two hosts are 100% sure that the data can be safely transmitted, then the program data is sent over the network.

TCP also is responsible of the effective data bandwidth over IP, which means that it has to regulate the bandwidth depending of the network speed. For that aspect come different mechanisms that rely on the ACK duplications (when a lot of ACK messages are duplicated, it means the data is well received and the network is reliable) to adapt the bandwidth. The different mechanisms are Reno, New Reno, tahoe, Vegas and are called "congestion control". As they are a very specific aspect of networking, I will not discuss the details of theses mechanisms on this series but we can talk about them in the comments.

For TCP to be sure that the data that is sent is well received, it uses a "serial number" that is assigned to each and every packet sent over the network. The serial number is incremented by the data length received. For example, when a data packet with a serial number 1000 is sent with a length of 1500 bytes, the ACK message that will come after has to be 2500. When an ACK is received, the serial number used in the connection is incremented by 1, when receiving SYN messages, by 10.

By this simple operation, a lost packet will result by an incoherence of serial numbers between the two hosts, which will cause a retransmission of the lost packet(s).

TCP or UDP for my application ?

TCP is THE big chunk of the Transport layer, and used when data is crucial to the applications (and so to the users). But UDP is used really often, quite as often as TCP.

By its features, TCP is used for "signalling", which means establishing the connection between two hosts, then UDP is used in paralled to stream data. UDP ensures no reliability of data over the network, which means that it is sent with no ACK needed nor confirmation whatsoever.

Even if it lacks a big feature of reliability, UDP is used when speed is required over reliability. Example applications can be video streaming or even phone communications (paired with a TCP connection for the signalling).

Now you have the basics of the two big protocols of the Transport layer, which are no mystery to you anymore.

On the next article, we will begin talking about applications, but I have to briefly discuss the layers 5/6, especially for SSL/TLS, which is used everywhere (fortunately).

See you soon :)

Building networks from A to Z - Part 5 : Routing

Nicolas Kirchhoffer — Tue, 27 Oct 2020 21:13:15 +0000

This post is the part 5 of the "Building networks from A to Z" series, please read the 4 previous parts so that you can understand this one. Thanks very much 😁

Before beginning this post, I would like to warn you, routing is a big chapter of networking and this will be a long post, as you're being used to now in this series. I will try to illustrate as much as possible every concept that we will be talking about to make the reading lighter.

Local and foreign networks

In the last part, we spoke about the concepts of networks and netmasks, but we don't know yet how to identify if an host is part of the local network or not.

To do so, you must obviously know the netmask and the address of your local network, and also the IP address that you want to analyze.

As an example, our PC is part of the 192.168.0.0/24 network (which means that the netmask is 255.255.255.0) and we want to know if the 10.23.4.129 host belongs to our network. Just follow this little operation :

         10.23.4.129
AND (&)  255.255.255.0
         -------------
         10.23.4.0

AND bit-by-bit operation just copies both 1 values in the result, as we are talking of a basic netmask here, I just copied the bytes where the corresponding byte of the netmask is equal to 255 (because it is 1111 1111 in binary). Of course, with more complex netmasks, you would have to do the bit-by-bit computing.

For the 10.23.4.129 host to be in our local network, the computation result is supposed to be 192.168.0.0, which it is not. Then this host does not belong to our local network, which means that it is a host from a foreign network. This notion is very important because it will decide whether or not we should route trafic based on the destination IP address. This is the very base of the routing process.

Before talking of routing, I would like to go further and present to you the first network service of the series, Address Resolution Protocol (also known as ARP), which is an application (layer 7) that will intervene between OSI layer 2 and OSI layer 3.

Address Resolution Protocol

In order to send data to a computer, you must know both its IP address and its MAC address, when discussing over the Internet. Internet Protocol apart, you must at least know the MAC address which is required to choose the physical path to which send the data.

ARP has been created in order to solve this problem. Imagine that we want to send a request to a specific IP address, and we know (after doing the calculations shown in part 2!) that the host is in our local network, so we just have to determine its MAC address.

For that specific role, ARP links a specific IP address to a specific MAC address. To know the MAC address of an host identified by an IP address, we just have to do a little ARP request.

ARP requests are sent via the MAC broadcast address (ff:ff:ff:ff:ff:ff, which means that the message is sent to all local hosts) asking Who has [IP address] ?, then the host who effectively has this IP address answers with saying [IP address] is at [MAC address].

After seeing ARP request and associated response, all hosts seeing the dialog are copying the pair of addresses into their ARP table (or cache), which they will be using to send data to each other without asking again.

Here is an example of the ARP process :

Example of a typical ARP exchange

Now you know how computers do to know to which IP address corresponds which MAC address. They use ARP !

Fun fact : You can see your ARP table by running the command arp -a (both in Windows - using cmd or PowerShell, GNU+Linux and macOS). Run arp --help to know more about ARP and also to add, remove and update the entries of your ARP table 👀

On the previous post, I wrote that we lack of IP addresses and that, then, the network engineers had to find a way to avoid switching the protocol version (because of the cost of the operation). We spoke about VLSM and having subnetworks, but there is another solution that has been adopted that's called NAT (Network Address Translation).

Network Address Translation

We saw, on the previous posts, that there are some reserved networks, and some of them are reserved to what's called private networks.

Private networks are not routed through the Internet, it is impossible to have a host on the Internet with a private IP address.

Private networks are the following :

10.0.0.0/8
172.16.0.0/12
192.168.0.0/16

If we think, to avoid having too much hosts compared to the pool of IP addresses, we can only address them using private networks addresses and have the messages relayed to them using the "translator", that has both a private IP address and a public one (= an address that is valid on the Internet).

Here is a little diagram with NAT working between a local network and the Internet :

192.168.0.2 sends data to 8.8.8.8 (public), the translator changes the source IP address to its own public (77.23.124.35) and then sends the data to 8.8.8.8

Another basic concept of routing is that, when the computer has to send data to a foreign IP host, it sends it to the gateway (generally, the router, here, the translator), that will try to find the path to the host. Note that only the MAC address is modified to be the gateway's, so that the gateway can still read the destination IP address and route it until finding it, each time we will pass through a router, the MAC source/destination addresses will be changed to fit the router ones. It is a good demonstration that we can not use MAC to route data, because it is needed for electronical pathfinding and IP is an abstraction that allows us to find hosts throughout the Internet !

The destination IP address stays the same, but all layer 2 informations are changed to fit local transit network

A transit network is a network in which data just goes to one edge-router to another edge-router. The concept of edge-routers will be explained later in that post.

The translator IS NOT a router, it just edits the source IP address so that the foreign host can answer to him. The translator remembers the first query and its source/destination addresses, when it sees a response from the previous destination address, it basically changes the destination address from the response (which is its own IP public address) to the local host that first emitted the query.

This is it for what is called NAT. 2nd fun fact of this post, the routers you have at home are not actual routers, they are just translators for your local network into the Internet. In fact, your home "router" has a gateway that is a router, on the ISP network. But we will be wondering how the ISPs networks work on the latest posts of this series.

Routing information through the Internet

The routing process is extremely varied, we have a lot of routing protocols that all have a specific goal, but before that, we will see that we have 2 types of routing :

Static routing : when you say specifically where to find the host, for example, to find host 8.8.8.8, pass through the local router 192.168.0.24. Each path is entered manually into the router, which means that we have to know and configure every network on the routing table. We will not be discussing this method that much, this description is quite self explanatory.
Dynamic routing : you just tell the routers what are the connected networks (which means the networks that the router is directly connected to, via cables) to route and it will spread the information to the edge routers (= the other routers in its local network). Here is a simple scheme :

3 routers with 2 connected networks each

Router A has been configured in the 2 networks it is connected to, and knows that they are here, so it tells Router B and C that the networks are behind him.

If a host from the network 10.0.3.0/24 wants to speak to a machine in the network 10.0.1.0/24, it sends the data to Router B that knows that this network is behind Router A, then it sends the data to Router A by changing the MAC layer as seen before. Router A just switches the query to the host in question.

Router B and C also tell Router A their networks, so that all networks on the diagram are accessible from every host connected to them.

In this case, we can see that we have at least 2 ways to go from a network to another, because the 3 routers are inter-connected in such a way that we have a redundant cable. Each possibility of connecting two networks together is called a path, and it is the main difference between routing protocols, which we will see just after.

The link between Router B and Router C can be removed and the networks from Router C could still be available from Router B. Yes, by passing through Router A, but going through Router A is longer than directly between Router B and C. That is called "cost" and it is a protocol-specific metric.

Here are 2 basic protocols that have been used for decades :

RIP (Routing Information Protocol)
OSPF (Open Shortest Path First)

RIP has been declined into two different major versions,
RIPv1 and RIPv2. RIPv1 is classful, meaning that routing was done with specific netmasks defined by each class, whereas RIPv2 is classless, meaning that you could specify, for each network, a specific netmask, so RIPv2 also supports subnetworking.

RIP is a very simple routing protocol, it will just count the number of hops between 2 networks. Let's apply the computation to our diagram above.

10.0.3.0/24 -> Router B -> Router C -> 10.4.3.0/24 - Hops : 0 (direct connection from router-to-router)
10.0.3.0/24 -> Router A -> Router B -> Router C -> 10.4.3.0/24 - Hops : 1 (through Router A)

In RIP, the cost metric is calculated based on the number of hops between two networks, as you can see, for the route 10.0.3.0/24 -> 10.4.3.0/24, we have 2 different paths that have two different costs.

RIP will disable the most expensive path, even if it is still physically connected. The maximum number of hops using the RIP protocol is 15. The most expensive path will be used if and only if the less expensive paths are all down.

With OSPF, the cost of the routes is calculated with "weights", that are standard or can be configured by the network administrator. The cost of each route will depend of its bandwidth, for instance, a route that has 2 hops but is linked entirely in 10Gbps lins will be prefered over a direct 100Mbps link, which will be used for redundancy and rescue.

OSPF also allows every scale of network and has "zones" in which you can have multiple routers. Zone 0 is called "backbone". All routers inside a zone share all their routes, whereas between zones, default gateways are passed so that every router doesn't have to store every route possible. It allows CPU resources saving.

OSPF and RIP are called Interior Gateway Protocol (IGP), as opposed to Border Gateway Protocol (BGP), which is a peering aggregator gateway protocol mostly used for WAN routing, it is used by ISPs to interconnect big networks together. We will see how it works on the part dedicated to ISP methods.

You have now come through this part on routing, it is non-exhaustive and will be completed on the future parts. If it isn't clear for you, don't panic, we will be demonstrating all notions together in a giant exercise at the end, I promise that you will understand everything from the Ethernet addressing to IP routing.

As for now, I will find you back on the next part for the Transport layer and then, we will be exploring the wonderful land of the Application layer, with all the networks services needed to build a great network.

See you soon, hope you didn't ragequit this post 😆

Building networks from A to Z - Part 4 : Network !

Nicolas Kirchhoffer — Mon, 26 Oct 2020 22:06:40 +0000

This is the part 4 of the series "Building networks from A to Z", don't forget to read the 3 previous parts in order to fully understand what I will demonstrate here. Thank you 😄

We are coming to the BIG part of networking, the one that is usually more known to developers and "large audience" as it involves the Internet Protocol, also known as IP.

We are now on layer 3 of the OSI model, and layer 2 of the TCP/IP model. This layer offers wide network properties that allow routing, which is one of the most important aspects of networking, and we will be covering it by every angle possible.

When talking about the Network layer, you can be sure that 99% of the time, Internet Protocol is involved. IP is currently available in 2 different versions : IPv4 and IPv6. The differences will be discussed later on this post.

Basical addressing

The main feature of IP is its addressing scheme, that allows segmentation to locate both the network and the host. Such a feature is the key-component to routing. We will first discuss the IPv4 addressing.

IP addresses are composed of 4 bytes, written in decimal this time, and usually in a well-known format that is 10.0.0.1. As every data sent on a network, the address is hard-coded in binary during transmissions.

The main goal of the Internet Protocol is to cover every size of networks possible, and we will see that it works perfectly well, it is a very powerful protocol in doing so.

Each IP network has a network address (the first address possible) and a broadcast address (the last address possible), so keep it in mind before we demonstrate it, all networks must have at least 2 addresses to work (but none left to hosts, so not very useful).

Historically, IP addressing was segmented by classes. Classes allow choosing the first byte to correspond to the good network size. For the sake of demonstrating how classes work, we will work with a little bit of binary, but don't worry, it is very simple and we will be only focusing on the first byte.

Here are the different classes :

Class A : The first bit must be 0

It then goes from 0000 0001 (1) to 0111 1111 (127), so every network in this range is a class A network. 127.x.y.z are reserved addresses, they are used for loopbacks, which we will be also talking about later. Network address is 0.0.0.0 and broadcast address is 126.255.255.255 (remember, 127.* addresses are reserved). 0.0.0.0 to 1.0.0.0 also reserved, so the first network address of the class A is 1.0.0.0. The first byte is the network part in class A, so, as a simple binary calculation, we have 2^(8-1) networks, the minus 1 is due to the first bit that must be 0, 2^7 = 128 networks that each can have 3 bytes of hosts, which correspond to 2^24 - 2 = 16 777 214 hosts. We remove 2 because of the network and broadcast addresses, that are reserved. The class A networks are the largest networks possible.

To check if an IP address belongs to the class A, just check the first bit, if it is a 0, then it does.

Class B : The two first bits must be 10

The range then goes from 1000 0000 (128) to 1011 1111 (191). The first address is 128.0.0.0 and the last one is 191.255.255.255. The first two bytes are the network identificator, and the two last are the hosts identificator. We then have 2^14 (not 16 because of the 2 restricted prefix bits) networks (that is 16 384 compared to 128 in class A) that each can have 2^16 - 2 hosts (that is 65 534 compared to 16 777 214 as previously).

Class C : The three first bits must be 110

Class C addresses go from 1100 0000 (192) to 1101 1111(223). The 3 first bytes are reserved to networks and the last byte is the one dedicated to hosts. Same calculations as before, 2^21 networks, which is 2 097 152 networks, and only 2^8 - 2 hosts, which is 254. The class C networks were the most common ones because we could have a lot of networks and still have a proper amount of hosts.

D and E classes are not widely used, the class D is reserved to multicast (we will maybe talk about multicast on another post) and E class is dedicated to R&D.

A notion that is linked to classes is the network mask, also called netmask, that we use to differentiate the bits that are used by the network and the ones used for the hosts. This is where we are going to use a little binary arithmetic operation called AND bit-by-bit operation.

A netmask is formed using 4 bytes in the same format as an IP address, but the network part is only 1s and the host part only 0s. Meaning that an AND operation between a "real" IP address and a netmask gives us the network address.

         11111111.11111111.11111111.00000000 (255.255.255.0)
& (AND)  00111001.00000010.00000101.10000001 (57.2.5.129)
         -----------------------------------
         00111001.00000010.00000101.00000000 (57.2.5.0)

Netmasks came after the classes, and I will explain on the next paragraphes why, but we can associate a class to a specific netmask :

Class A : 255.0.0.0
Class B : 255.255.0.0
Class C : 255.255.255.0
Class D : 255.255.255.255

Classes were designed so that we can designate to which class belongs a network by only looking to the first bits, but they created netmask for the same purpose, but also widening the concept. As you can see, the structures of netmasks put in this exact order remembers us the pyramidal scale of the first bits of each class.

255.0.0.0           0000 0000
255.255.0.0         1000 0000
255.255.255.0       1100 0000
255.255.255.255     1110 0000

The problems begin

Considering the number of hosts in total (2^32 = 4 294 967 296), we rapidly lack of addresses to attribute to each device on the world, as we count PCs, mobile phones, tablets, game consoles, in fact every device that is connected to the Internet, even a connected fridge, a smart watch, everything.

Network engineers then had to find fixes in order to continue using IPv4, considering upgrading would have been a huge cost to companies and organizations around the world, so they came to many solutions that are now widely used.

Variable-Length Subnet Mask

Variable-Length Subnet Mask (VLSM) is born on the initiative to save IP addresses by divising the existing classes in subnets. As we saw before, until then, a byte of a netmask could either be 0 or 255. With this principle, we can have many more networks by allowing other values for each byte.

VLSM uses the CIDR (Classless Inter-Domain Routing) notation to ease the writing of netmasks and the network addresses coupled.

Here are examples by transposing classes networks to CIDR notation :

10.0.0.0 (class A) : 10.0.0.0/8
191.100.0.0 (class B): 191.100.0.0/16
192.168.1.0 (class C): 192.168.1.0/24

The number indicated after the / is actually the number of 1s from the first bit of the netmask. For example, 10.0.0.0/8 is the network 10.0.0.0 associated with the netmask 255.0.0.0 (1111 1111 is equal to 255). For /16, obviously, 255.255.0.0 is the associated netmask, and for /24, 255.255.255.0 is the associated netmask.

Now imagine you are being assigned the 52.2.0.0/16 (this network has already been divided by the ISP, how to know it ?!) network from an ISP, and you have multiple sites to address with this one and only network. To avoid too large IP broadcasting domains and vulnerabilities being spread really fast, you choose to divide this network into multiple /24 networks, each for another physical site.

This example is pretty simple because we are converting one class into another, but it can be pretty when using non-perfect values (we will do another example later ☺️).

To go from 16 to 24 network bits, we just have to add 8 bits that we call subnetworks bits. These bits will identify the subnetworks we will be using. Each combinaison of the 8 will give us a subnetwork. A few examples :

52.2.0000 0000.0/16
->   0000 0001.0/24
->   0000 0010.0/24
     ...

You have, in total, a possibility of 255 subnetworks, which should be enough for our case. Of course, a subnetwork is a network, so each one of these have both a network and a broadcast address. 52.2.1.0/24 is the network address and we will compute the broadcast address very simply by changing the hosts bits value from 0 to 1, by doing so :

52.2.1.0000 0000/24 -> 52.2.1.1111 1111
                       52.2.1.255
52.2.2.0000 0000/24 -> 52.2.2.1111 1111
                       52.2.2.255

Aside those 2 addresses, you should have 254 addresses dedicated to your subnetworks hosts. And you have successfully designed subnetworks to split your network in case to use it smartly between your multiply physical sites.

Here is a quick diagram that abstracts all concepts we have seen :

Subnetworking the 52.2.0.0/16 network

Here you have your subnetworks !

Using a non-8 multiplicator for the netmask length is also possible and trickier to setup, but we will covering one of them in another post, where we will use all knowledge we have acquired to create a network on which we have services and users !

The Internet protocol is one big part of the networks, and we are not yet to cover it entirely !

The next part will be talking about routing and all its aspects. This chapter was a big part to the series and has quite complex notions.

Rendez-vous at the next part speaking about routing and relative concepts !

Building networks from A to Z - Part 3 : Data link and topology

Nicolas Kirchhoffer — Sun, 25 Oct 2020 21:41:32 +0000

This is the part 3 of the series "Building networks from A to Z", I strongly suggest that you read the two previous parts before reading this one, as we will be using knowledge from them. Thank you.

Now that we have seen the Physical layer, it is time to discover Data link. This is the first abstraction layer of the OSI model, and is part of layer 1 of TCP/IP model for reasons that we will see later on this post.

I would, first, like to talk about Network topology. The architecture of networks isn't quite arbitrary, and it is often designed by the layer 2 protocol, the Data link.

This layer is very interesting and we can see it everywhere, not even on Internet networks, but even on Bluetooth, where addressing is exactly the same. But first, let's talk about building our local network !

Network terminology

Before going further, I would like to speak about networks terminology. We have used some acronyms to qualify different networks, and it is something you might want to know when talking to a net guy (nerd).

From tiniest network to biggest :

PAN (Personal Area Network) : mostly used for 2-devices communication, for example, USB is a wired PAN, Bluetooth is a wireless PAN (WPAN). Its range is usually a few meters (1 or 2, not much more). PAN networks do not need network infrastructure !
LAN (Local Area Network) : the network we have at home ! Usually composed of a few devices, it is centered around a router and a switch. A data link wildly used for LAN is Ethernet (wired) and WLAN (802.11 "Wi-Fi"), they both use MAC addresses (as well as Bluetooth), which we will be discussing later.
MAN (Metropolitan Area Network) : a network slightly bigger than LAN, used in cities and universities. Generally used when multiple LAN are used by the same entity, older protocols like ATM and SDDI were used for its purpose, now replaced by Gigabit and 10-Gigabit Ethernet as well as MPLS (which will be discussed at the VERY end of the series!). Structured around multiple routers and multiple switches, both in the same organization and composing the backbone of an Intranet ("network of networks of an organization" - connecting multiple networks of the same entity).
RAN (Regional Area Network) : usually refers to broadcasting TV or radio, using VHF (Very High Frequencies) and UHF (Ultra High Frequencies) using OFDMA multiplexing. We will very probably not discuss those technologies at all, can be interesting in a Telecommunications series (maybe one day?).
WAN (Wide Area Network) : the biggest of them all, the best example ? The Internet is a WAN, connects at the same time RAN, MAN and LAN together to form the WAN.

Here is a quick illustration :

A MAN network is composed of LANs, but the WAN isn't always composed of MANs only, but also LANs (they even are the most common)

With that being said, we skip network terminology and basic architecture to enter topology related to Data link protocols !

Data link protocols and network topologies

Network engineers and administrators have known multiple protocols during many years, many companies were trying to sell their own implementation of a Data link protocol, but, thankfully, this trend converged into adopting Ethernet as a whole. We will discuss how the Data link protocols are responsible for Network topology and how it evoluted during time !

1. The Ring networks

This is the first network topology that we will discuss. The main property of a ring network is that every computer is connected to the previous one but also the next. A little illustration :

Every computer is connected to the one before him and the one next to him

The most known protocol for ring networks as LAN is Token Ring, developed by IBM in the mid-80s. The principle of Token Ring is that in order to communicate, the computer must have the token, which distributes the communication time. A big advantage of this technology is that we have no collision whatsoever, but the efficient bandwidth is majorly reduced when we add more and more computers, as we divide the communication time. As far as other Data link protocols by the time, infrastructure was only sold by the companies which have the patent of the protocol, and they were controlling the licensing price.

Another well-known ring network is FDDI (Fiber Distributed Data Interface), often used to connect many LANs together in order to create a MAN. FDDI uses fiber cables to have a 200km+ range and 100Mbps speed, two routers were then connected to each other using 4 fibers (SM - Single-mode). 2 fibers were used for full-duplex in one way and the 2 others for the other way. This architecture is really stable as it is given a natural redundant path.

2. The Bus networks

Bus network consist in having all the computers on a single medium, which is known as a "bus" in IT. It is not really used nowadays, but still has some advantages, for example the cost of the overall network, the simplicity of connecting devices to it. However, if the cable serving the bus is cut or broken, the whole network fails, which is a critical criteria to avoid when building networks. Here is an illustration if you don't see what it is :

The larger line is representing the bus, to which each and every PC is connected.

3. The Mesh networks

Mesh networks are not really known from the "outer-IT world" but are quite common. Its principle is to have every device of the network connected to all the others. Here is a little illustration :

Every computer is connected to all the others (not the case in the example but you got the idea 😁)

It is mostly used in Wi-Fi where in some networks, all the wireless routers (or APs - Access Points) are connected in a mesh network. A derivate example to the Mesh networks are the connections between neighbor eNodeB (antennas in LTE and LTE-Advance networks - Access networks, also discussed in a future post), that for sake of "handover" (the fact is switching antenna when moving without losing signal) are connected to the neighbors and handing connections to them via a protocol that's called X2.

4. The Star networks

Star networks are by far the most common ones. The principle is to have every device of the network connected to a central point, either a switch or a hub. We will be discussing the differences between them at the end of this section !

So you get it, Ethernet is the most famous protocol that describes a Star network. As usual, here is an illustration of a Star network :

Each PC is connected to the same central switch

This architecture is the most used one because Ethernet was only free, and as noted above, it wasn't the case of all Data link protocol. Ethernet was quite a poor protocol compared to Token Ring for example, it has a high collision factor and requires a lot of cables to work. In IT, often time, the quality of the product isn't the most important factor, but the cost really is relevant.

Ethernet has been fixed many times in hopes to reduce collisions, and it works as well as required to run it throughout the world nowadays !

Each Data link protocol has its own implementation of the MAC addressing system, but the Ethernet one is the most famous, obviously.

A MAC address is an unique physical address, linked to a Network card (NIC) and determined during the manufacturing process. For Ethernet, each MAC address is composed of 6 bytes, quite always represented in hexadecimal, like that ee-f6-ab-d7-cd-07 (MS-DOS approach) or like that ee:f6:ab:d7:cd:07, sometimes also with a ..

The hub is used to concentrate all the data in a single point and redistribute it at all the connected devices. Here is a quick illustration of the hub principle :

The upper PC is sending data to the hub in the center, which redistributes the data to all connected PCs.

A hub is a quite dumb device. It doesn't really care about MAC addresses. With the hub broadcasting data, you can imagine that the other PCs trying to emit while the upper one is still communicating can cause problems. That's why Ethernet logical topology, opposed to physical, is a Bus network, because the media is shared between hosts.

Opposed to the hub, the switch will be reading the destination MAC address from the Ethernet header and sending the data only to the PC concerned. The other devices will not even be able to see the data.

The upper PC sends data to the lower-left PC through the switch, which reads the destination MAC address and sends it through the right port to the PC that corresponds to that address.

Ethernet is called a switching protocol because of this property, and also the fact that it would be totally unusable without this addon.

Although, even if collision is fixed with unicast (PC-to-PC communication, one device talks to one another), it is still a big issue while broadcasting. Indeed, Ethernet supports broadcasting to all PCs using a specific MAC address, ff:ff:ff:ff:ff:ff (which corresponds to all bits value being 1). The switch will then take the broadcasting request and redistribute it as a hub, to all hosts connected. This is called the broadcast domain. It might not be an issue on little networks, but when connecting switches to another switches, you can easily reach large amounts of hosts.

One solution to this problem has been CSMA/CD (Carrier Sense Multiple Access with Collision Detection), that will be listening to the media to determine if another host is talking, when not, it will emit the data it has to. However, if a collision is detected, the host will stay silent for a random period of time (a multiplicator of the RTT (Round-Time Trip) value - arbitrary) before emitting again. Hitting a certain number of collisions may cause an error that makes the host destroy the data. Note that this technology has been transposed to wireless medias, where collisions are more likely to happen, as the CSMA/CA (Carrier Sense Multiple Access with Collision Avoidance). Observing a collision on a wireless infrastructure is not something that we can allow.

Well ! It seems you're all set for the understanding of the Data link layer, mainly focused on local networks, but we will widen those concepts when talking about the Network layer (layer 3 of OSI and 2 of TCP/IP) on the next post !

Thank you for reading this long series (and this post also is particularly long), I hope to find you back on the next ones 😇

Building networks from A to Z - Part 2 : let's get physical!

Nicolas Kirchhoffer — Sun, 25 Oct 2020 17:34:20 +0000

This article is the part 2 of the series "Building networks from A to Z", if you havent't read part 1, please do so by accessing the links of the series.

We will now be focusing on the lowest layers of both OSI and TCP/IP models. The physical OSI layer actually refers to cabling, modulation, error correction, even electronical bit-by-bit processing. I will not go into the details of this layer that is a separate world on its own, but we are going to discuss the different cables we may encounter, and their connectors.

Cable Work time !

The cable used on a network actually changes based on the purpose of it. At home, you will only be encountering 1 to 2 cable types. Whereas, in enterprises networks or ISPs, you may encounter many types of fibers and copper cables.

The most common cable (still!) remains the copper cable. It is used since the analogic phone era, originally with 1 pair of wires, now with 4 for data transmission.

For one-pair wires, the connector used is called RJ11 (or another compatible variant, RJ12, that has 2 pairs) and connect mainly phone systems. It is also used for DSL (Digital Subscriber Line), which is a type of network provided by ISPs as what we call an "Access network" (between the ISP and subscribers houses).

For 4-pairs wires, the connector used is called RJ45, it is similar to the ones exposed above, but have, obviously, more wires and is slightly bigger. RJ11/12 connectors can be used in RJ45 ports, whereas the opposite is not possible. It can be wired in 2 different configurations, EIA/TIA-568-A and EIA/TIA-568-B.

A quick comparison between EIA/TIA-568-A and EIA/TIA-568-B

The purpose of having 2 distinct configurations is to solve a problem that was encountered with routers. Basically, the placement of the different pins is changed so that the Rx (reception) pin of one end of the cable corresponds to the Tx (transmission) pin of the other end. Twice, because we have a full-duplex communication (meaning that the communication can be simultaneous in the 2 ways, one device can emit while receiving data on the same time). This is not used on newest devices because they are able to sense/designate what pins to use for which communication, although, inverted-ends cables are still usable as devices, such as switches, are able to detect the change of standard.

Efficiency and signal strength using copper cables can be seen and measure with a Time Domain Reflectometer.

That's approximately it for copper cables, now let's talk about optical fibers !

Fibers are well-known to the large audience because of their performances, able to perform 1 Gigabit per second for the most common ones (in 2020, not the case before, yes, fibers are used since 1977!). When we talk about fibers, we have different types of cables, and many connectors as well !

We have two main types of fiber cables :

Multi-mode fibers : Allow the use of multiple wavelengths, meaning that we can have multiple communications using a single medium (or bidirectional communications). They are typically used for short distances. The kernel of a Multi-mode fibers is quite large (from 50 to 65µm) which allows light to bounce on the edges, shifting the light and avoiding interferences with other light beams.
Single-mode fibers : Allow the transmission of a light signal on a long distance. They have a much smaller kernel, that will avoid light from bouncing on the edges, and having a straight light beam, with less loss on long distances. The inconvenient is that they are more expensive and you must have 2 of them if you want a full-duplex communication.

The main difference between Single-Mode and Multi-Mode fibers

When it comes to connectors, we also have 2 big families, which are squared connectors and round connectors. The main difference between the two is the density of connections.

Squared connectors (SC) are preferred to round for most desktop, PC and multimedia applications. They are also declined in 2 distinct types, UPC (Ultra-Polish Connectors) that stands for the ceramic end of the connector being perpendiculary polished (90° angle between the end of the connector and the kernel of the fiber) and APC (Angle-Polish Connectors) that ensures an 8° angle between the ceramic-end of the connector and the fiber kernel.

UPC connectors are less used than APC because parts of the light beam will be reflected away while inserting the other connector, causing a big loss doing so. APC connectors are especially designed to avoid that loss, and are more therefore more efficient.

Illustration of UPC and APC connectors differences

The signal strength when using a fiber cable can be seen and measured using an Optical Time Domain Reflectometer.

As said at the beginning, the Physical layer is also in charge of the modulation and correction of the signal, but as we are only talking about the basics on this series, I will not be discussing those aspects, but maybe in another article if you want me to !

That's now it for the Physical layer of the OSI model, which can be very complex due to its close relation with physics, and is absolutely not abstract at all. But it's over, you've come through this tough part, and we're now moving on to layer 2, which is Data link, and we will have completed TCP/IP layer 1.

See you on the next article 😁

Building networks from A to Z - Part 1 : the basics

Nicolas Kirchhoffer — Sun, 25 Oct 2020 17:34:02 +0000

Welcome to my first series about networks !

From my experience as a Network technician, I know for a fact that some developers lack competence in networking and can sometimes be stuck on those aspects.

I would like to do a complete review of all the networking knowledge I have acquired and I will try to guide you through those notions with illustrations and schemes !

Firstly, I will present to you the several architecture concepts that we use in networks and how this works, then, we will browse each layer one by one and speak about the technologies used !

Introductory speech about networks

What exactly is a network ? How does it work ? Why was it built that way ? What is the Internet ?

Basically, in Computer Science, a network is a group of computers that communicate with each other. A network is managed and configured by a Network administrator, and besides of being composed from computers, most of the time, if you want more than 2 computers communicating, this network will have several network devices that we will talk about later.

Nota bene : When the notion of network was created, the IoT really wasn't even a thing, so when I speak about "computers", really I am talking about all kinds of devices that have a Network connectivity.

A basic network composed of a switch and 4 PCs

The Internet is actually an aggregator network. It is not much than all the networks on the world connected together. Internet Service Providers (ISPs) are the ones that connect the networks together. If you have an Internet connection in your house, then it is a whole separated network, that would perfectly work on its own, except that most of websites aren't hosted in your house, so you have to go explore the world !

The Internet is often described as "the network of networks", it has been founded in 1969 to make its predecessor, ARPAnet, accessible to universities and laboratories in the US first, then around the world.

But before travelling on the Internet, let's return to your homes and discover concepts on their basic level !

Introducing the OSI model

How to talk about networks if we don't talk about the OSI model ?

The OSI model has been created to standardize all communications in the IT world. It is composed of 7 layers that each has a well-determined role to play in the delivery of data.

Starting from the bottom, we have :

Layer 1 : Physical

The cables, connectors, everything that is used to physically connect devices together.

Layer 2 : Data link

Used to address devices on a local scope, we give physical devices an address to know how to talk to them.

Layer 3 : Network

This layer is used to address devices as network entities. The address is built so that every segment of it redirects us to a specific part of the network. This is the Internet Protocol (IP) layer, on which we will be talking a lot on another article.

Layer 4 : Transport

Used to encapsulate data from program to program. This layer is primarily used to locate source/destination inside of a communicating device.

Layer 5 : Session

Session layer will be holding informations on hosts, its role is to enable the communication between hosts. It is also called "Inter-host communication".

Layer 6 : Presentation

This is used to format data in particular ways. It is, for example, at this layer that SSL/TLS (for communication ciphering) intervenes.

Layer 7 : Application

The most abstract layer, the one on which the data is displayed to the end-user, and the interaction is enabled. Most common application protocols are HTTP (the web), FTP (for file sharing between hosts) or DNS (for domain name resolution).

Don't be afraid of it ! It is a purely theoretical representation. It only helps developers of the protocols to know their environment, where it happens and for what it is used.

For more convenient understanding, and also to have a more "field" approach, we will mainly be using the TCP/IP model, which is easier.

The TCP/IP model is an emerging concept that is used to represent the communications that the Software developers face, or even Sys/Net admins.

As far as this series, we will be discussing layers 1, 2, 3, 4 and 7.

TCP/IP simplifies the OSI model by merging layers 1 and 2 together, and that is TCP/IP layer 1 (called Network access or MAC), then layer 2 which is OSI layer 3, called Internet or also Network. Layer 4 stays the same as TCP/IP layer 3, Transport, and layer 5, 6 and 7 are merged together as Application layer, which is TCP/IP layer 4.

4 layers for TCP/IP that represent the exact same architecture as the OSI model, but not with the same focuses as far as working on it. Usually, the TCP/IP model is the one that talks more to the less "network-technical" people, because we hear a lot about IP and TCP/UDP.

Here is a brief view of what we just talked about :

Brief comparison scheme of the two models

Now you have the basic concepts we will be using in the next articles ! As said before, we will be browsing through the layers to understand how networks work and what have been their needs through time. Sit back and relax for some anecdotes and, I hope, a quite complete review of basic networking knowledges.

Stay tuned and don't hesitate to tell me if you didn't get something in this article, or if you think I said something wrong.

You will have all resources used to write these articles at the end of the series, it is easier for me to compile everything once I have finished writing them.

DEV Community: Nicolas Kirchhoffer

Expose your Kubernetes services from your home network

The problem

The solution

Expose your service to the Internet

Alternative solution

Make your personal project a real product

Table of Contents

Landing Zone

Example

Step 0 : Believe in your project

Step 1 : Write a "state of the art"

Example

Step 2 : Define specifications

Example

Step 3 : Conduct a market study

Example

Conclusion

Why can't Serverless really be serverless ?

Build a RESTful API with the Serverless Framework

Sagar ・ Oct 19 '18 ・ 6 min read

⚡️ 10 Ways to Use Serverless Functions

Nader Dabit ・ Jan 8 ・ 10 min read

The "server" principle

Distributivity and the web

A final word

Building networks from A to Z - Part 1 : the basics

Nicolas Kirchhoffer ・ Oct 25 ・ 4 min read

Building networks from A to Z - Part 6 : Transporting data

Ports addressing

Data transport

Transport reliability and connection

TCP or UDP for my application ?

Building networks from A to Z - Part 5 : Routing

Local and foreign networks

Address Resolution Protocol

Network Address Translation

Routing information through the Internet

Building networks from A to Z - Part 4 : Network !

Basical addressing

The problems begin

Variable-Length Subnet Mask

Building networks from A to Z - Part 3 : Data link and topology

Network terminology

Data link protocols and network topologies

1. The Ring networks

2. The Bus networks

3. The Mesh networks

4. The Star networks

Building networks from A to Z - Part 2 : let's get physical!

Cable Work time !

Building networks from A to Z - Part 1 : the basics

Welcome to my first series about networks !

Introductory speech about networks

Introducing the OSI model