HTTP Protocol Overview

HTTP is so fundamental to web development that it is often overlooked. Let's take a minute to explore the protocol and gain a better understanding of how the HTTP request-response cycle works.

What is HTTP?

HTTP, or Hypertext Transfer Protocol, is an application level protocol for sharing hypertext.

protocol - A set of rules for how to transmit and receive data. If two programs use the same protocol then the receiver will know how to interpret the data from the sender.

application level - A protocol for programs that an end-user interacts with, as opposed to lower level programs that a user would not generally directly interact with. Most commonly a web browser like Google Chrome in the case of HTTP.

hypertext - Text documents displayed on a computer that allow for links to other documents. The most common way we lay out the structure of a hypertext document is with HyperText Markup Language, or HTML.

Summary - HTTP is a set of rules for how two high level programs communicate with each other. HTTP follows a client-server model, where a client makes a request and the server sends back a response. The client will usually be a web browser, and the server will usually be a web server, but this is not always the case.

GET Requests

The easiest way to see the raw data in a request is with netcat, a command line utility available on most systems as either netcat or nc. We will tell netcat to listen on a port, and then any data we send to that port will be dumped into the terminal for us to see.
nc -l -p 3000

• -l means "listen"
• -p 3000 means "use port 3000"

Now let's make a simple GET request to this port with curl in order to see what information is actually sent. With the previous netcat command running in another terminal, run curl localhost:3000. (Curl defaults to GET if no request method is specified.

The request sent by curl contains four lines of information, and netcat logs them to the terminal. The first line contains basic information about the request: GET / HTTP/1.1 The technical term is the "start line". More specifically, the start line of a request is the "request line", and the start line of a response is the "status line".

• GET the HTTP verb, specifies the type of request
• / the request target, specifies the resource we want - could be "/", "index.html", or "/api/users"
• HTTP/1.1 the protocol and the specific version - could be HTTP or HTTPS

After the start line we have three lines specifying different http header properties. They tell us more information about the request curl is making. Finally, it's easy to overlook the fourth line, which is simply an empty line signalling that we have reached the end of the request header.

This is a GET request, so there is no body, and we have reached the end of the request, but both the curl command and the netcat command are hanging rather than returning control to the user. Netcat is simply listening without responding, and will keep listening until the connection is closed, while curl is waiting until it either receives a response or the request it made times out. We can interrupt the curl command, which ends the connection, allowing netcat to exit as well.

Header: Host

The first header listed in the example above is Host: localhost:3000, which tells us the name (and optionally the port) of the server we are requesting. If we had multiple websites that we wanted to run on a server with only one IP address we could use the Host header to differentiate which requests went to which website. This can be configured relatively easily in Apache with virtual hosts or in Nginx with multiple server blocks.

Header: User-Agent

The User-Agent header provides information about the client making the request, and might specify the program name and the operating system. In this case we see the name of the program making the request, "curl", and the version, "7.68.0": User-Agent: curl/7.68.0. But this field can be confusing, especially when it comes to web browsers. Here is an interesting explanation of why Google Chrome's user agent string starts with "Mozilla".

Header: Accept

A media type or MIME type (Multipurpose Internet Mail Extensions) is a string declaring the format of an http body. It consists of two parts: the type and the subtype, separated by a forward slash. An asterisk is used as a wildcard.

The Accept header specifies what MIME types a client is capable of receiving for a specific request, like text/html or image/jpeg. Multiple values can be given as a comma separated list. In our example above Accept: */* means the client will accept any media type with any subtype.

We will see MIME types again when we look at the Content-Type header of a POST request.

The Connection

HTTP concerns itself with the formatting and handling of a request followed by a response, but it does not concern itself with the details of how a connection is made between the two programs, or how the data is actually transmitted over the wire. These details are handled by lower-level protocols, which for our purposes will almost always be TCP/IP.

In the below image we can see a Wireshark capture of the actual packets sent. The first three lines establish a connection in a process known as the TCP Three-Way Handshake, which consists of the client sending a packet with the SYN flag set (SYNchronize), the server responding with a SYN ACK packet (SYNchronize ACKnowledge), and finally the client sending a packet to ACKnowledge receiving the SYN ACK packet.

It is somewhat analogous to the way a phone call starts, establishing a connection but not sharing any actual information yet:

• Client sends SYN - Alice dials Bob's number
• Server sends SYN ACK - Bob picks up the phone and says "Hello?"
• Client sends ACK - Alice says "Hello" back

The fourth line, highlighted in green, is the packet that carries the actual HTTP request from curl to netcat, and the following line is netcat's acknowledgement to curl that it received that particular packet.

The transmission of those first five packets occurs practically instantly from the human perspective - all within the same millisecond. It's not until we abort the curl request five seconds later that we see the final three lines terminating the TCP connection.

POST Requests

Let's see what the HTTP request looks like when it is a POST with some data attached. We have three pieces of information: a name, "Joe Smith", an age, 64, and some letters, "abc". There are several ways we can format this data when sending it to the server (or netcat in this case), and the first one we will examine is application/x-www-form-urlencoded.

URL-encoded form data splits the information up into key value pairs, with the key and value separated by an equals sign =, and the pairs separated by an ampersand &. Any characters that are not letters or numbers are escaped with the percent sign %. So our information becomes: name=Joe%20Smith&age=64&letters=abc.

If we send a POST request with a body to netcat we see the headers, followed by a blank line, followed by the request body: curl -X POST -d "name=Joe%20Smith&age=64&letters=abc" localhost:3000

• -X POST this flag allows us to specify a method other than GET
• -d "..." include data in the body of the request

Headers: Content-Length & Content-Type

The POST request contains two new headers. Content-Length: 35 tells the recipient that the request contains the header, an empty line, and then 35 bytes of data. But first we need to tell the server how to interpret those 35 bytes of data. Curl adds the header Content-Type: application/x-www-form-urlencoded by default when you specify data with the -d flag, but if the data you are sending is of a different type you can override this default with another flag: -H "Content-Type: ...".

The data itself is just ones and zeros to the computer. Below is an image showing the tail end of a wireshark capture of the POST request. It shows both the raw ones and zeros (taking up most of the width), as well as those bits interpreted as ASCII text (narrow column to the right). The body of the POST is highlighted in blue:

We can make it more readable by viewing the bytes in hexadecimal form, where each set of two characters corresponds to eight bits, or one byte. (For example the binary 01101110 is 6e in hex, interpreted as n in ASCII). The body of the request is again highlighted in blue, and I've drawn a red line around the sequence of two new lines that signal the end of the header. Once the server recognizes that the header has ended, it knows that the next 35 bytes belong to the body of the request (the 35 pairs of hex letters highlighted in blue).

It's important to note that without a content-type header the server does not know (though it may be able to guess) that it should interpret the numbers as ASCII codes, much less that the ASCII characters are in the format of key-value pairs escaped with %.

We can send JSON data with curl and specify a content-type in a header with the -H flag. The server still interprets the individual bytes as ASCII codes, but those decoded characters are now treated as JSON rather than url-encoded key value pairs. curl -X POST -d '{"data": "ABC123"}' -H 'Content-Type: application/json' localhost:3000.

Conclusion

The program you are using to send data to the server may or may not automatically determine the right content-type header for your data, and knowing how to set and check headers is an essential skill. To learn more about the HTTP protocol check out the MDN guide or read the official standard, RFC 7230.