How can people use the internet without IPs?

Highly recommend you read the previous article and spend time understanding IPs, sockets, and ports.

Why URL ?
Suppose an application is running on our computer and needs the IP and port of another application to communicate over the internet. But remembering IP addresses is hard, so we use human-readable names to name computers, which computers don’t understand, but we use a middleman called DNS to map the name to machine-readable numbers ( IP ). In addition to the computer's name, we need other kinds of information to communicate, like which PORT the server is running on and which application-level protocol we are using; we need to communicate what all the resources we need from the server, etc., so a URL is the standard way to represent the information needed to communicate with other computers, which is agreed globally.

The main Idea of protocols
I live in Chennai, India. If people get into a cab and tell the driver a location like “Chennai Central Railway Station”, the driver gets the exact location the passenger wants to go to. He doesn’t need to know the full address or geolocation. If the passenger and driver were both locals, they would both agree that the phrase “Chennai Central Railway Station” is the particular location through their experience.

The Master Format:
://:@:/;?#

Scheme (How are we connecting?)

From the previous article, we know that simply by knowing the IP ( host name )and PORT, we can able to create a socket connection. But the goal of a connection is to transfer resources/services from the server to the client. To achieve this, we need to transfer large amounts of data using a universally agreed standard to communicate. When we are building a server and client using TCP/UDP sockets, we have to write some extra logic for the standard that both party agrees.

Example
Let me build my own simple application protocol

The request from the client has to mention the type of the file, like jpg, txt, etc and on the next line I add txt, jpg, or binary file. I know the request format is going to come in the format and file from the server side; I can easily code to take the format, take the file from the next line, and save it as a file with its actual format, e.g., dummy.jpg

# socket is tool by OS to abstract network connection
import socket

# create a socket of IPV4 and TCP
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# giving the socket ip and port
server.bind(('127.0.0.1', 8080))
# telling the os to prepare connection , max 1
server.listen(1)
print("Server waiting for connection...")

# sever listen , when got request, create a dedicated connectionSocket for communication 
connectionSocket, clientAddress = server.accept()
# receves data
raw_data = connectionSocket.recv(4096)

# extract header of our protocol
header, first_chunk = raw_data.split(b'\n', 1)
extension = header.decode()

# create a file and save all the data in that file
filename = f"dummy.{extension}"
with open(filename, "wb") as f:
    f.write(first_chunk) # Write the body from the first packet

    # Keep receiving if the file is large
    while True:
        chunk = connectionSocket.recv(4096)
        if not chunk: break
        f.write(chunk)


# close the socket
print(f"Successfully saved {filename}")
server.close()
connectionSocket.close()

The above one is a one-time server

# socket is a tool provided by the OS to abstract network connections
import socket

# 1. Prepare data (Let's send a simple text file)
extension = "txt"
file_data = b"This is the content of my custom protocol file!"

# 2. Apply Your Protocol format: extension + \n + file_data
protocol_message = f"{extension}\n".encode() + file_data

# 3. Setup TCP Socket and Connect
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('127.0.0.1', 8080))

# 4. Send everything and close
client.sendall(protocol_message)
print("Message sent using custom protocol.")
client.close()

The above code is the client to connect to the server code

Like the scheme we created above, there are globally accepted schemes: HTTP, FTP, SMTP, and database schemes(e.g., postgres://, mongodb://, redis://).

**
Host (Where is the server?)**
We already discussed IP in depth, and the domain name is resolved to an IP. We already discussed this in our previous article in depth, so I’m skipping it with a link.

Path, query, and params in the URL are used to convey the client's requirements to the server in different styles.

Example

https://www.example-store.com/shop/shoes.html;color=red;size=10?sort=price&gender=mens#reviews

Here, the path is /shop/shoes.html, which is used to specify that, in this store, we are interested in the shoe HTML on the server.

Params are ;color=red;size=10, which tells the server that the client is interested in size 10 and red shoes.

The query was ?sort=price&gender=mens, which tells the server to sort the page by price, and we were interested in men's shoes

HTTPS is the scheme; from the scheme, we can derive that the default port is 443.

www.example-store.com is the domain name resolved by DNS into an IP address.

For some schemes like ftp, we need a username and password.

#reviews is the fragment here; this notation is used to indicate which part of the HTML document is viewed. Here, in the HTML page, the user views the review part. The fragment is useless for the server and is used as a bookmark.

How we use URLs practically

We know about URLs now; we never type the full URL in the browser in our daily life, and web developers themselves won’t use the full URL during development.

In the above example, this part https://www.example-store.com is the base URL; irrespective of what page/resource we see, this base URL has to be the same. The rest of the URLs, other than the base URL, are relative URLs

/shop/shoes.html;color=red;size=10?sort=price&gender=mens#reviews

The browser understands that the current page's base URL is the first part of the relative URL unless it is explicitly mentioned. Developers ( including myself ) build webpages using relative URLs; the base URL can be changed in one place if we are changing the domain name. I am personally part of the team building jewellery software, where we change theme colors, logo, and base URL to run. The remaining codes remain same; the company sold the same software to many jewellers.

We observe that the browser is intelligent enough to suggest the URL when we half type with the help of browser history. The browser, like Google, Safari, etc have their website details in its database; if we search a related word, we see it in the result. If we click, we will go to the website. The question that comes to my mind is how Google's DB knows all its websites. The answer is that the website owners themselves expose it to Google, Safari, etc nobody wants to hide their website in e-commerce competition. Even if they don’t, google have crawler that crawls the internet's pages; if it finds any webpage link that it does not know, it will go to the website and try to crawl all its pages and save the important keywords that help the google to show the website for relevant searches in the browser.

URL ENCODING AND DECODING
In this URL, we cannot use characters which was in other languages, or any special symbol which was not in ASCII, or any reserved words in a URL with a different meaning. So we have a workaround: the illegal characters mentioned here are converted to ASCII code, which is URL encoding. When the browser reads it, it knows it is some kind of special character, and on the receiving end of the request, it is decoded to normal form, called URL decoding. I will add the examples below.

1. Spaces (The most common) You type: apple pie Browser sends: apple%20pie
2. Reserved Characters (To prevent confusing the server) You type: Ben & Jerry (The & usually splits a query!) Browser sends: Ben%20%26%20Jerry (The & becomes %26) You type: 100% Browser sends: 100%25 (The % becomes %25)
3. Other Languages & Symbols You type: café Browser sends: caf%C3%A9 You type: hello 👋 Browser sends: hello%20%F0%9F%91%8B

We have a question: if we encode to make the URL safe, why can’t we do it for the entire URL text except the special characters?
The answer is no. The receiving side only expects special characters to be encoded and normal text to remain normal. Overdoing it here creates confusion.

From the above and the past two articles, we know about the OS role in communication ( sockets ) and socket addresses ( IP + PORT). Now we know, as common users, why we don't see IPs in our next articles; we explore More about the application layer and how frameworks and libraries were built to make API calls.