Doing Hard Things:An HTTP Server From Scratch

Petersburg Awinbe-Yisazuah — Thu, 13 Mar 2025 18:01:34 +0000

The craze of AI taking the jobs of developers really got me thinking about how i need to solidify my understanding of fundamental concepts so over the weekend i decided to build an HTTP server from scratch.

WHAT IS HTTP AND HOW DOES IT WORK

The HTTP(S) protocol is an application layer protocol which is stateless and is relies on TCP protocol for data transmission. When we say stateless we simply mean that a persistent connection between devices communicating is not maintained.The connection is broken after every data transfer.At the foundation of every HTTP server is a TCP server listening in for incoming connections and accepting. This implies that HTTP server is a wrapper around a TCP server. Based on this we can break down an HTTP server into its essential components and attempt building each of them and then finally putting them together

Main components of the HTTP server

Base TCP server
Request handler
Request processor
Response handler

Now lets pick each part and explore it.
Base TCP server:
The TCP server is a socket bound to a specific port and IP address, listening for incoming connections and deciding whether to accept or reject them.Most servers can handle concurrent connections and so I have to build the TCP server to handle connections concurrently.To handle this I took inspiration from the way databases handle connections concurrently with thread pools. For every accepted connection i spin up a worker thread to handle communications
Request Handler:
The request handler component i built extracts the necessary information from the request data and passes on to the processor component. For simplicity sake my focus was on serving static files. HTTP data is binary encoded and has three components; the request line,the request headers and the request body depending on the request method respectively as show below

example GET
GET / HTTP/1.1 Host: localhost:8080 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:135.0) Gecko/20100101 Firefox/135.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

example POST
POST /api/login HTTP/1.1\r\n\ Host: localhost:8080\r\n\ User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:135.0) Gecko/20100101 Firefox/135.0\r\n\ Content-Type: application/json\r\n\ Content-Length: 44\r\n\ Accept: */*\r\n\ Connection: keep-alive\r\n\ \r\n\ {"username": "test_user", "password": "12345"}\r\n

My request handler parses the data and retrieves the request line. The request line is also parsed to retrieve the request method and the path. I then pass both the request method and and path to my request processor.

Request processor
My request processor locates the HTML file requested and encodes it in binary format. The encoded HTML file is then passed to the request handler

Response handler
My response handler receives the encoded HTML data and constructs the response in the standard HTTP response format
This is how a sample response will look like
HTTP/1.1 200 OK\r\n Date: Sun, 09 Mar 2025 12:34:56 GMT\r\n Content-Type: text/html\r\n Content-Length: 125\r\n Connection: keep-alive\r\n Server: nginx/1.18.0\r\n \r\n <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Example Page</title> </head> <body> <h1>Hello, World!</h1> </body> </html>

Finally the base TCP server closes the connection and the communication is ended.
Access the code base for this project here

What Happens Behind the Scenes of an HTTP Request?

Petersburg Awinbe-Yisazuah — Mon, 10 Feb 2025 00:49:21 +0000

When you type a URL like google.com into your browser and hit Enter, a fascinating chain of events is set into motion. A request is sent to a server, and within seconds—thanks to a decent internet connection—you see the familiar white page with "Google" boldly written and a search bar in the center. But what exactly happens between hitting Enter and seeing the webpage? Let’s dive into the intricate processes that make this possible.

Step 1: DNS Resolution – Finding the Server’s Address

Before your browser can fetch the webpage, it needs to figure out where Google’s server is located. This process is known as Domain Name System (DNS) resolution.
What is DNS Resolution?
Computers don’t inherently understand human-readable domain names like google.com. Instead, they use IP addresses to locate servers on the internet. An IP address is a unique identifier for a computer or device on a network. Domain names are simply mnemonics to make it easier for humans to access websites without needing to remember complex numerical addresses. DNS acts like the internet’s phonebook, mapping domain names to their corresponding IP addresses so computers can locate web servers. For example, google.com might map to an IP address like 216.58.223.196.

How Does DNS Resolution Work?
When you enter google.com in your browser, the browser first tries to find its matching IP address. This is typically done in the following steps:
a.

Checking the Local Cache
The browser first checks its local DNS cache to see if it has a recent record of google.com's IP address. If found, the browser uses this cached IP and proceeds with the request, saving time.
b. Querying the Operating System’s Resolver
If the browser cache doesn’t have the IP, it makes a system call to the local DNS resolver, which is managed by the operating system. The OS resolver then checks its own cache. If it finds a matching record, it returns the IP to the browser.
Fun fact: To see the cached DNS records on a Linux system, you can use the following command: sudo resolvectl query google.com

c. Requesting from an External DNS Server
If no cached record exists, the local resolver queries an external DNS server, usually provided by your ISP or a public DNS provider like Google DNS (8.8.8.8) or Cloudflare DNS (1.1.1.1).

d. Navigating the DNS Hierarchy
DNS servers are structured in a hierarchy, and the request travels through multiple layers:
Root DNS Servers: These servers direct the request to the appropriate Top-Level Domain (TLD) server (e.g., .com for google.com, .org for other domains, etc.).
TLD Servers: The .com TLD server then directs the request to the Authoritative Name Server for google.com.
Authoritative DNS Server: This server holds the actual IP address for google.com and returns it to the resolver.

e. Returning the IP Address
Once the resolver obtains the IP address, it sends it back to the browser. The browser can now initiate the actual HTTP request to Google’s server using the GET method.

By completing this process, the browser now knows exactly where to send the request and can proceed to retrieve the webpage. But the journey doesn’t end here—next comes establishing a connection to the server.

Step 2: The Three-Way TCP Handshake – Establishing a Connection

At this point, you might be wondering why we can’t just receive the response from Google’s server immediately. It’s important to keep in mind that the data requested from servers isn’t always simple or small. Often, it includes additional elements like images, videos, and scripts. To ensure smooth communication and prevent data loss, both the server and your computer need to establish a reliable connection. This process is known as the three-way TCP handshake and can be thought of as a preparation stage to ensure both parties are ready for communication.
Here’s how it works:
SYN Packet
The client (your browser) sends a SYN (synchronize) packet to Google’s server, signaling that it’s ready to initiate communication.
SYN-ACK Packet
Google’s server responds with a SYN-ACK (synchronize-acknowledge) packet, acknowledging the receipt of the SYN packet and indicating its readiness to communicate.
ACK Packet
Finally, the client sends an ACK (acknowledge) packet to Google’s server, confirming that the SYN-ACK packet was received. At this point, the connection is fully established, and data transfer can begin.

Step 3: Fetching and Rendering the Webpage

Once the connection is established, Google’s server sends back the requested homepage. This typically includes HTML, CSS, JavaScript, and other resources like images or videos. Your browser then processes these files, renders the page, and displays it as you’re used to seeing.
And just like that, within seconds, the familiar Google homepage appears in your browser, ready for you to start searching.

Conclusion

What seems like a simple action—typing a URL and hitting Enter—involves a complex series of steps working seamlessly behind the scenes. From DNS resolution to the TCP three-way handshake, each step plays a crucial role in delivering the webpage to your screen. The next time you browse the web, you’ll have a deeper appreciation for the intricate processes that make it all possible

DEV Community: Petersburg Awinbe-Yisazuah