HTTP Deep Dive: What Every Backend Engineer Needs to Know

These are my own study notes, turned into a guide. I wrote it after learning how HTTP really works under the hood. If you are a backend engineer, or you want to become one, this covers the small part of HTTP that explains most of the bugs you will debug in production.

Table of Contents

1. The Two Main Ideas Behind HTTP
2. Network Layers and the History of HTTP
3. The Structure of an HTTP Message
4. HTTP Methods and Idempotency
5. CORS: The Browser's Security Guard
6. HTTP Status Codes
7. HTTP Caching
8. Content Negotiation and Compression
9. Persistent Connections and Large Payloads
10. TLS, SSL, and HTTPS

---

The Two Main Ideas Behind HTTP

HTTP is the language of the web. Every time a browser talks to a server, it uses HTTP. The whole protocol is built on two simple ideas: it is stateless, and it follows the client-server model.

Statelessness

HTTP has no memory of past requests. Every request is self-contained. It carries everything the server needs to handle it: headers, URL, method, and body. Once the server sends a response back, it forgets the request happened. If the same client sends another request one second later, the server treats it as something completely new, with no link to what came before.

This is why every request needs to carry proof of who is sending it, like an auth token or a session cookie. The server does not remember you from one request to the next, so you have to remind it every time.

Why is this actually good design?

• Simplicity: the server does not need to store session data in memory. This keeps the server code simpler.
• Scalability: you can spread requests across many servers with no problem, because no single server has to track a client's session. If one server crashes, the client is not affected, because there was no state to lose in the first place.

Since HTTP has no memory on its own, developers built techniques on top of it, like cookies, sessions, and tokens, to keep some kind of continuity for things like logins or shopping carts.

The Client-Server Model

In any HTTP flow, there is always a client and a server:

• Client: usually a browser or an app. It always starts the connection and sends the request.
• Server: hosts the resources, like websites or APIs, and waits for requests. When a request arrives, the server processes it and sends back a response, such as HTML, JSON, or a plain text file.

HTTP and HTTPS follow the same principles. HTTPS is just HTTP with TLS encryption on top for extra security.

Good questions for interviews or exams:

• What is statelessness in HTTP, and what are its benefits?
• How do we work around HTTP being stateless?
• Who always starts the connection in the client-server model?

---

Network Layers and the History of HTTP

Before a client and server can talk over HTTP, they need a path between them. HTTP needs a reliable transport, one that does not lose messages. That is why it runs on top of TCP, which is connection-based, instead of UDP.

In the OSI model, HTTP lives at Layer 7, the Application Layer. As backend engineers, this is mostly where our work happens. We do not need to go deep into TCP handshakes or the lower layers. That part is network engineering.

How HTTP Changed Over the Years

• HTTP/1.0: opened a new TCP connection for every request, then closed it. This was slow and wasted resources.
• HTTP/1.1: added persistent connections. One TCP connection could now handle many requests and responses. It also added caching and chunked transfer encoding.
• HTTP/2: added multiplexing, so many requests and responses can travel at the same time over one connection. It moved from plain text to binary framing, and added header compression, called HPACK, plus server push.
• HTTP/3: built on the QUIC protocol, which runs over UDP instead of TCP. This gives a faster connection start, lower latency, and fixes a problem called head of line blocking that still existed in HTTP/2.

Good questions for interviews or exams:

• What is the main difference between HTTP/1.0 and HTTP/1.1?
• What are the main additions in HTTP/2?
• Which OSI layer does HTTP work in?

---

The Structure of an HTTP Message

A request has

• Method (GET, POST, and so on)
• Resource URL (the path you are asking for)
• HTTP version (like HTTP/1.1)
• Host (the domain)
• Headers
• A blank line that separates the headers from the body
• Body (only if needed)

A response has

• HTTP version
• Status code (like 200)
• Status text (like OK)
• Headers
• A blank line
• Body

Headers: Extra Information About the Message

Headers are key-value pairs that carry extra information about the request or the response. Think about sending a parcel. You write the name and address of the receiver on the outside of the box, not inside it. This lets the shipping company read the information fast, without opening the box. Headers work the same way. They let the client and server share information without touching the actual data, which is the body.

1. Request Headers

These are sent by the client to tell the server who it is, what it wants, and what format it understands.

Header	Purpose	Example
User-Agent	Tells the server what client is making the request	Mozilla/5.0
Authorization	Carries login credentials	Bearer abc123token
Accept	Tells the server what format the client wants back	application/json
Accept-Language	The client's preferred language	en-US,ar-EG
Accept-Encoding	What compression the client supports	gzip, br

GET /users HTTP/1.1
Host: example.com
Accept: application/json
Authorization: Bearer token123

This request means: "send me JSON, and here is my token to prove I am logged in."

2. General Headers

These carry information about the message itself, and they are not specific to just the request or just the response.

Header	Purpose	Example
Date	When the message was sent	Date: Fri, 05 Jun 2026 01:00:00 GMT
Cache-Control	How the response should be cached	Cache-Control: max-age=3600
Connection	Whether the connection should stay open	Connection: keep-alive

3. Representation Headers

These describe the shape of the data inside the body.

Header	Purpose	Example
Content-Type	The format of the body	application/json, text/html, image/png
Content-Length	The size of the body in bytes	245
Content-Encoding	Whether the body is compressed	gzip
ETag	A fingerprint of the current version of the resource	"v1.7-abc"

ETag is technically a representation header, but in real use it works more like a cache validator than just a description of the content. You will see it a lot in the caching section below.

4. Security Headers

These help protect the connection from attacks.

Header	Purpose
Strict-Transport-Security (HSTS)	Forces the browser to only use HTTPS, so it cannot be tricked into using plain HTTP
Content-Security-Policy (CSP)	Limits where scripts, styles, and images can load from, which reduces XSS attacks
X-Frame-Options	Stops the page from being loaded inside an iframe, which prevents clickjacking
Set-Cookie with HttpOnly / Secure / SameSite	Stops JavaScript from reading the cookie, forces it to be sent only over HTTPS, and reduces CSRF

Set-Cookie: sessionId=abc123; HttpOnly; Secure; SameSite=Strict

Two more ideas worth remembering

• Extensibility: HTTP lets you add custom headers, often with an X- prefix, without breaking the protocol. The server can read them if it understands them, or just ignore them.
• Remote control: headers work like a remote control for the server. Content negotiation (Accept), caching (Cache-Control), and authentication (Authorization) all guide server behavior without changing the URL or the body.

Good questions for interviews or exams:

• Why do we put metadata in headers instead of the body?
• Give an example of a security header and explain what it does.
• What are representation headers? Give some examples.

---

HTTP Methods and Idempotency

Methods show the client's intent:

Method	Intent
GET	Get data, do not change anything
POST	Create new data, comes with a body
PATCH	Update part of a resource
PUT	Replace the whole resource
DELETE	Remove a resource

Best practice: many developers use PUT when they actually mean PATCH, and this is a mistake. A simple rule: always use PATCH for updates, unless you have a real reason to replace the whole resource. Only then use PUT.

Idempotent vs Non-Idempotent

Idempotent means: if you run the same request many times, the result stays the same and does not break the data.

• GET: fetching the data 10 times gives you the same data every time.
• PUT: replacing a resource with the same data 10 times still ends with the same final state.
• DELETE: deleting something once removes it. Deleting it again does nothing new.

Non-idempotent means the result changes with each call. POST is the main example. If you click "create note" twice, you get two different notes.

There is also OPTIONS. The client uses this not to get data, but to ask the server what it can do. This is the method behind CORS preflight requests, which we cover next.

Good questions for interviews or exams:

• What is the main difference between PUT and PATCH, and why is PATCH the best practice?
• Explain idempotency, with examples of idempotent and non-idempotent methods.

---

CORS: The Browser's Security Guard

Browsers follow a Same-Origin Policy. A frontend running on one domain is blocked from freely calling a backend on a different domain, for security reasons. CORS (Cross-Origin Resource Sharing) is the mechanism that lets a server allow specific origins to bypass this rule.

There are two request flows:

1. Simple Request

A request counts as "simple" if its method is GET, POST, or HEAD, and it does not carry complex headers.

The browser adds an Origin header on its own. If the server replies with Access-Control-Allow-Origin that matches this origin, or with *, the browser lets your JavaScript see the response. If not, it blocks it and shows a CORS error in the console.

2. Preflight Request

The browser sends a preflight request before the real one if any of these are true:

• The method is not "simple" (like PUT or DELETE).
• The request has non-simple headers, like Authorization or a custom header.
• The Content-Type is not simple. Most JSON APIs fall into this case, since application/json triggers a preflight.

The preflight OPTIONS request has no body. It is just a question: "can I send you a PUT? Can I include Authorization?" If the server agrees, it replies with 204 No Content, plus the allowed origins, methods, and headers, and Access-Control-Max-Age so the browser can save this approval instead of repeating the preflight on every single request.

Good questions for interviews or exams:

• When does the browser decide to send a preflight instead of a simple request?
• What is the OPTIONS method and what is its role in CORS?
• What does Access-Control-Max-Age do in the preflight response?

---

HTTP Status Codes

Status codes give the client a standard way to understand the result of a request, without needing to read the response body. They are grouped by their first digit:

1xx: Informational

• 100 Continue: the server got the headers and the client can now send the body (useful for large uploads).
• 101 Switching Protocols: used when moving from HTTP to WebSocket.

2xx: Success

• 200 OK: the request worked (usually with GET).
• 201 Created: the request worked and created a new resource (usually with POST).
• 204 No Content: the request worked but there is no body to return (common with OPTIONS or DELETE).

3xx: Redirection

• 301 Moved Permanently: the resource moved for good. For example, if you rename /user to /person, you return 301 so old clients still work and nothing breaks for them.
• 302 Found: a temporary redirect. The client should still use the original path in the future.
• 304 Not Modified: the resource did not change, so the client can use its cached copy (used together with ETag).

4xx: Client Errors

• 400 Bad Request: the client sent bad data.
• 401 Unauthorized: no auth token was sent, or the token expired.
• 403 Forbidden: the token is valid, but the client does not have permission for this action.
• 404 Not Found: the resource or path does not exist.
• 405 Method Not Allowed: using a method that this endpoint does not support.
• 409 Conflict: a conflict in the data, like trying to create something that already exists.
• 429 Too Many Requests: the client went over a rate limit.

5xx: Server Errors

• 500 Internal Server Error: something went wrong on the server and was not handled properly.
• 501 Not Implemented: this feature does not exist yet, but it might be added later.
• 502 Bad Gateway: seen with proxies like Nginx, when the main server sends back an invalid response.
• 503 Service Unavailable: the server is down for a short time, because of high traffic or maintenance.
• 504 Gateway Timeout: the proxy got no response from the main server in time.

401 vs 403, in one line: 401 means "I do not know who you are." 403 means "I know exactly who you are, but you are not allowed in."

Good questions for interviews or exams:

• What is the difference between 401 Unauthorized and 403 Forbidden?
• When do you use 409 Conflict?
• What is the difference between 301 and 302 redirects?

---

HTTP Caching

Caching means saving copies of responses, so we do not have to ask the server for them again. This lowers load time and saves bandwidth.

How ETag-based caching works

1. First GET: the server responds with 200, plus Cache-Control: max-age=10, an ETag (a fingerprint of the response), and Last-Modified.
2. Second GET for the same resource: the browser automatically adds If-None-Match (the old ETag) and If-Modified-Since (the old date). This basically asks the server: "is this ETag still current, or did the data change since this date?"
3. If nothing changed, the server responds with 304 Not Modified. This tells the browser it can safely use its own cached copy.
4. Once the resource gets updated, for example through POST or PATCH, the server generates a new ETag. On the next GET, the browser still sends the old ETag in If-None-Match, the server sees it is no longer valid, and responds with 200 OK, the fresh data, and the new ETag.

Best practice: relying only on plain HTTP caching with ETags in production can get complicated and puts extra load on the server to manage. A better option is often a client-side caching library, like React Query, which gives the client full control over when to fetch new data and when to reuse the cache. Still, it is worth knowing that the native HTTP caching option exists for simpler cases.

Good questions for interviews or exams:

• What is an ETag, and how does it work with If-None-Match to enable caching?
• When does the server respond with 304 Not Modified?

---

Content Negotiation and Compression

Content negotiation is how the client and server agree on the best format to exchange data. It has three main types:

1. Media type negotiation: through the Accept header (like application/json vs application/xml).
2. Language negotiation: through Accept-Language (like English vs Spanish).
3. Encoding negotiation: through Accept-Encoding (like gzip vs deflate).

The client can ask for JSON in Spanish, then switch to XML in English, and the server adapts, without the client needing to change anything else about the request.

Compression

Since we are already talking about Accept-Encoding, compression deserves its own note. On a dataset with 11,000 entries, a response compressed with gzip came out to about 3.8 MB, while the same response without compression was about 26 MB. That is the real impact of compression. It shrinks payload size by a large amount, and this matters a lot when many clients are downloading the same large file. The client, meaning the browser, receives the compressed data and decompresses it on its own.

Good questions for interviews or exams:

• List the different types of content negotiation and their headers.
• Why do we use HTTP compression, and which header controls it?

---

Persistent Connections and Large Payloads

Keep-Alive

In HTTP/1.0, every request opened and closed a new TCP connection. This was slow and wasted resources. HTTP/1.1 made persistent connections the default, using the Connection: keep-alive header. This lets the client and server reuse the same connection for many requests, until one side decides to close it, which reduces latency and saves resources.

Uploading large payloads: multipart/form-data

When you upload binary data, like an image or a video, the request uses multipart/form-data. The key detail here is the boundary parameter. Since binary data is sent in parts, a delimiter string, called the boundary, marks where each part starts and ends, so the server can split them apart inside the request body.

Downloading large payloads: streaming

To send a large text response without freezing, the server streams it in chunks instead of sending one huge block. Two things make this work:

1. Content-Type: text/event-stream: tells the client to expect the data as a series of events.
2. Connection: keep-alive: the connection needs to stay open until every chunk arrives.

The client keeps appending each chunk as it arrives, until the full file is complete.

Good questions for interviews or exams:

• What is the benefit of Connection: keep-alive?
• What is the boundary, and why do we use it in multipart/form-data requests?
• How does a server stream a large response to a client?

---

TLS, SSL, and HTTPS

You will not work with these directly most of the time as a backend engineer, but you still need to know what they mean.

• SSL (Secure Sockets Layer): the original protocol for encrypting data between client and server. It is now outdated and no longer used, because of known security weaknesses.
• TLS (Transport Layer Security): the modern replacement for SSL, and much more secure. It encrypts data while it travels, using certificates to confirm the server's identity and set up an encrypted connection. This stops eavesdropping and data leaks. The current recommended version is TLS 1.3.
• HTTPS: simply regular HTTP with TLS or SSL security added on top. It uses TLS as the encryption layer to protect login details and other sensitive data.

Good questions for interviews or exams:

• What is the difference between SSL and TLS, and why did we stop using SSL?
• What is HTTPS, and how does it work together with TLS?

---

Wrapping Up

Understanding this flow, from statelessness, to headers, methods, CORS, status codes, caching, negotiation, streaming, and TLS, is the key to debugging most of the problems you will meet as a backend engineer. You do not need to master TCP internals or the full TLS handshake to be effective. You need to picture this pipeline clearly enough to reason about where things break.

---

Bonus: A Few Code Examples

Some of these ideas are easier to understand with real code. Below are small examples in both C# (ASP.NET Core) and Go, covering idempotent updates, custom headers, CORS, and cache headers.

Example 1: PATCH (partial update) vs PUT (full replace)

C# (ASP.NET Core)

// PATCH: partial update, the recommended default
[HttpPatch("{id}")]
public async Task<IActionResult> PatchUser(int id, [FromBody] JsonPatchDocument<UserDto> patch)
{
    var user = await _userService.GetByIdAsync(id);
    if (user is null)
        return NotFound();

    patch.ApplyTo(user);
    await _userService.UpdateAsync(user);

    return NoContent(); // 204
}

// PUT: full replacement, only use it when you really mean it
[HttpPut("{id}")]
public async Task<IActionResult> ReplaceUser(int id, [FromBody] UserDto replacement)
{
    var exists = await _userService.ExistsAsync(id);
    if (!exists)
        return NotFound();

    await _userService.ReplaceAsync(id, replacement); // idempotent: same result every call
    return NoContent();
}

Go

// PATCH: partial update
func PatchUser(w http.ResponseWriter, r *http.Request) {
    id := mux.Vars(r)["id"]

    var patch UserPatch
    if err := json.NewDecoder(r.Body).Decode(&patch); err != nil {
        http.Error(w, "invalid body", http.StatusBadRequest)
        return
    }

    if err := userService.ApplyPatch(id, patch); err != nil {
        http.Error(w, "not found", http.StatusNotFound)
        return
    }

    w.WriteHeader(http.StatusNoContent) // 204
}

// PUT: full replacement, idempotent
func ReplaceUser(w http.ResponseWriter, r *http.Request) {
    id := mux.Vars(r)["id"]

    var replacement User
    if err := json.NewDecoder(r.Body).Decode(&replacement); err != nil {
        http.Error(w, "invalid body", http.StatusBadRequest)
        return
    }

    userService.Replace(id, replacement)
    w.WriteHeader(http.StatusNoContent)
}

Example 2: Enabling CORS for a specific origin

C# (ASP.NET Core)

// Program.cs
builder.Services.AddCors(options =>
{
    options.AddPolicy("AllowFrontend", policy =>
    {
        policy.WithOrigins("https://myfrontend.com")
              .AllowAnyMethod()
              .AllowAnyHeader()
              .AllowCredentials(); // needed if you send cookies or Authorization
    });
});

var app = builder.Build();
app.UseCors("AllowFrontend");

Go

func corsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        origin := r.Header.Get("Origin")
        if origin == "https://myfrontend.com" {
            w.Header().Set("Access-Control-Allow-Origin", origin)
            w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, PATCH, DELETE")
            w.Header().Set("Access-Control-Allow-Headers", "Content-Type, Authorization")
            w.Header().Set("Access-Control-Allow-Credentials", "true")
            w.Header().Set("Access-Control-Max-Age", "3600")
        }

        if r.Method == http.MethodOptions {
            w.WriteHeader(http.StatusNoContent) // 204 preflight response
            return
        }

        next.ServeHTTP(w, r)
    })
}

Example 3: ETag-based caching

C# (ASP.NET Core)

[HttpGet("{id}")]
public async Task<IActionResult> GetResource(int id)
{
    var resource = await _resourceService.GetByIdAsync(id);
    if (resource is null)
        return NotFound();

    var etag = $"\"{resource.Version}\"";

    if (Request.Headers.TryGetValue("If-None-Match", out var clientEtag) && clientEtag == etag)
        return StatusCode(StatusCodes.Status304NotModified);

    Response.Headers.ETag = etag;
    Response.Headers.CacheControl = "max-age=10";

    return Ok(resource);
}

Go

func GetResource(w http.ResponseWriter, r *http.Request) {
    id := mux.Vars(r)["id"]
    resource, ok := resourceStore.Get(id)
    if !ok {
        http.NotFound(w, r)
        return
    }

    etag := fmt.Sprintf(`"%s"`, resource.Version)

    if match := r.Header.Get("If-None-Match"); match == etag {
        w.WriteHeader(http.StatusNotModified) // 304
        return
    }

    w.Header().Set("ETag", etag)
    w.Header().Set("Cache-Control", "max-age=10")
    json.NewEncoder(w).Encode(resource)
}