This blog is for personal reference of CSAPP Ch 11 Reference • Network Programming
1. The Network Model - Three Layers That Matter
Ch 11 focuses on three layers of the network stack. Everything else (physical cables, ethernet, WiFi) is below this level and hidden by the OS.
| CORE IDEA | Your code only ever touches the Application and Transport layers. The OS handles everything below. When you call read() on a socket fd, the OS transparently handles IP routing, TCP reliability, and physical transmission. You just see bytes. |
|---|
2. Addressing - IP, Ports, and the Four-Tuple
2.1 IP Addresses - Finding the Machine
An IP address identifies a specific machine on the network. IPv4 addresses are 32-bit integers, written as four decimal numbers: 192.168.1.5. IPv6 addresses are 128-bit, written as hex groups: 2001:db8::1.
Important IP addresses to know:
- 127.0.0.1 (localhost) - loopback address. Packets sent here never leave the machine. Used for local inter-process communication.
- 0.0.0.0 - bind to ALL network interfaces. A server binding to this accepts connections on any network interface (WiFi, ethernet, loopback).
- 255.255.255.255 - broadcast address. Sends to all machines on local network.
2.2 Ports - Finding the Program
A port is a 16-bit integer (0-65535) that identifies a specific program on a machine. The OS uses it to route incoming packets to the correct process.
| Port Range | Name | Who uses it | Examples |
|---|---|---|---|
| 0-1023 | Well-known ports | Privileged - requires root to bind | 80 (HTTP), 443 (HTTPS), 22 (SSH), 25 (SMTP) |
| 1024-49151 | Registered ports | Any process | 5432 (PostgreSQL), 6379 (Redis), 8080 (dev HTTP) |
| 49152-65535 | Ephemeral ports | OS assigns automatically to clients | Randomly chosen per outgoing connection |
Ephemeral ports: when a client calls connect(), the OS automatically picks an unused port from the ephemeral range for that connection. The client doesn't choose it - the kernel does. This is what makes the four-tuple unique.
2.3 The Socket Address - IP + Port Combined
A socket address is the combination of an IP address and a port - uniquely identifying one endpoint of a connection.
2.4 The Four-Tuple - Uniquely Identifying Every Connection
Every TCP connection is uniquely identified by four values:
| KEY INSIGHT | A server on port 80 can handle thousands of simultaneous connections because each connection has a unique four-tuple. The server port (80) is the same for all of them - the client port differs for each. The kernel routes incoming packets to the right connection using the full four-tuple, not just the port. |
|---|
2.5 Byte Ordering - Network vs Host
Different CPU architectures store multi-byte integers differently (big-endian vs little-endian, from Ch 2). The network uses big-endian (network byte order) by convention. x86 CPUs use little-endian (host byte order). You must convert between them when putting addresses/ports in socket structs.
| Function | Converts | Use when |
|---|---|---|
| htons(x) | host → network (16-bit) | Setting sin_port in sockaddr_in |
| htonl(x) | host → network (32-bit) | Setting sin_addr in sockaddr_in |
| ntohs(x) | network → host (16-bit) | Reading port from received sockaddr_in |
| ntohl(x) | network → host (32-bit) | Reading IP from received sockaddr_in |
Helper functions for IP strings: inet_pton() converts "192.168.1.5" → binary. inet_ntop() converts binary → "192.168.1.5". Always use these instead of manual conversion.
3. TCP vs UDP - When to Use Each
| Property | TCP | UDP |
|---|---|---|
| Connection | Connection-oriented - 3-way handshake before data | Connectionless - just send packets |
| Reliability | Guaranteed delivery - lost packets retransmitted | No guarantee - packets can be lost silently |
| Ordering | Guaranteed in-order delivery | No ordering guarantee - packets can arrive out of order |
| Message boundaries | NONE - byte stream, any chunking possible | Preserved - one send = one receive (or nothing) |
| Flow control | Yes - slows sender if receiver buffer full | None - sender can overwhelm receiver |
| Congestion control | Yes - slows down when network is congested | None - can make congestion worse |
| Speed/latency | Slower - handshake overhead, retransmit delays | Faster - no setup, no waiting for retransmits |
| Use cases | HTTP, databases, SSH, email, file transfer | DNS, video streaming, gaming, QUIC |
| TCP BYTE STREAM WARNING | TCP has NO concept of message boundaries. Two write() calls on the sender can arrive as one read() on the receiver, or one write() can arrive as multiple reads. Every TCP protocol must explicitly handle message framing. This is the most common networking bug for beginners. |
|---|
| REAL WORLD | QUIC (used by HTTP/3) is built on UDP because it needs to implement its own multiplexed streams with independent reliability per stream. TCP's head-of-line blocking (one lost packet stalls all data) is unacceptable for HTTP/3's multiple parallel streams. UDP gives QUIC full control to implement exactly the reliability semantics it needs. |
|---|
4. The TCP 3-Way Handshake
4.1 The Handshake Sequence
4.2 What Each Step Establishes
- SYN: 'I want to connect, my sequence numbers start at x.' Sequence numbers are used to detect lost packets and reorder out-of-order delivery.
- SYN-ACK: 'I got your SYN (ack=x+1 means I expect x+1 next), and my sequence numbers start at y.' Server also allocates kernel buffers for this connection here.
- ACK: 'I got your SYN (ack=y+1 means I expect y+1 next).' Connection is fully open in both directions.
The kernel handles the handshake - not your application. Your server process is asleep in accept() during the entire handshake. The kernel completes it, then wakes your process when the connection is ready.
4.3 The Connection Queues
The kernel maintains two queues per listening socket:
| SYN FLOOD ATTACK | An attacker sends thousands of SYNs without completing the handshake. The incomplete queue fills up, server allocates buffers for each half-open connection, memory exhausted, legitimate connections refused. Defense: SYN cookies - server encodes connection state in the SYN-ACK's sequence number instead of allocating buffers, so no memory is used until the final ACK arrives. |
|---|
4.4 TCP Connection Teardown - 4-Way Handshake
Half-closed connections: after client sends FIN, client can no longer send data - but can still receive. The server→client direction is still open until the server sends its own FIN. This asymmetry is used by HTTP keep-alive and streaming responses.
TIME_WAIT state: after sending the final ACK, the client enters TIME_WAIT for 2×MSL (Maximum Segment Lifetime, ~60s). This prevents delayed packets from a dead connection being misinterpreted by a new connection using the same four-tuple. Servers that rapidly recycle connections (like load balancers) can run out of ephemeral ports due to TIME_WAIT accumulation - tunable via SO_REUSEADDR.
5. The Socket Interface - Syscalls in Detail
5.1 Server Side - 4 Syscalls in Order
SO_REUSEADDR: always set this option before bind() on server sockets. Without it, if your server crashes and restarts, bind() fails with 'Address already in use' because the old socket is in TIME_WAIT. setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &optval, sizeof(optval)) before bind() fixes this.
5.2 Client Side - 2 Syscalls
socket() → connect()
connect() - Initiate connection to server
5.3 Server vs Client Syscall Comparison
| Syscall | Server | Client | What it does |
|---|---|---|---|
| socket() | Yes - first call | Yes - first call | Create socket fd |
| bind() | Yes - assign port | No - OS assigns ephemeral port automatically | Attach address to socket |
| listen() | Yes - mark as passive | No | Enable incoming connections, set queue size |
| accept() | Yes - get connections | No | Block until client connects, return connfd |
| connect() | No | Yes - initiate connection | Trigger 3-way handshake, block until done |
5.4 getaddrinfo() - Modern Address Lookup
Instead of manually filling sockaddr_in structs, modern code uses getaddrinfo() - it resolves hostnames ("www.google.com") to IP addresses, handles IPv4/IPv6 transparently, and returns a linked list of possible addresses to try.
| REAL WORLD | getaddrinfo() is what Go's net.Dial() and Rust's TcpStream::connect() use internally when you pass a hostname. It handles DNS resolution transparently. The linked list of results matters: a hostname may resolve to multiple IPs (for redundancy/load balancing) - trying each one is called happy eyeballs. |
|---|
6. The accept() Lifecycle - Full Kernel Picture
| KEY INSIGHT | The client's connect() returns BEFORE the server's accept() returns. The client can start sending data immediately after connect() returns - data sits in the kernel's receive buffer until the server calls read(). The server's application code is not involved in the handshake at all. |
|---|
7. Message Framing - Solving the Byte Stream Problem
TCP has NO message boundaries. Two writes may arrive as one read, or one write may arrive as multiple reads. Every TCP protocol must explicitly solve the framing problem.
7.1 Solution 1 - Fixed-Length Messages
7.2 Solution 2 - Delimiter-Based Framing
7.3 Solution 3 - Length-Prefixed Framing
7.4 How Real Protocols Frame Messages
| Protocol | Framing Approach | Details |
|---|---|---|
| HTTP/1.1 | Delimiter + Content-Length | Headers delimited by \r\n, blank line ends headers, Content-Length header specifies body size |
| HTTP/2 | Length-prefixed binary frames | 9-byte frame header (3 bytes length + 1 type + 1 flags + 4 stream ID) |
| gRPC | 5-byte header + body | 1 byte compression flag + 4 bytes length, then body |
| Redis (RESP) | Delimiter-based | *2\r\n\$5\r\nHELLO\r\n - type prefix + length + data + \r\n |
| WebSocket | Length-prefixed binary frames | 2-byte header minimum, extended for large payloads |
| PostgreSQL | Length-prefixed | 1-byte message type + 4-byte length + body |
| PRODUCTION BUG | Treating TCP like it preserves message boundaries (one write = one read) is one of the most common and hardest-to-debug networking bugs. It works perfectly on localhost (data almost always arrives in one chunk on loopback) and breaks randomly in production when network conditions split messages. Always implement explicit framing. |
|---|
8. Concurrent Servers - Handling Multiple Clients
8.1 The Iterative Server Problem
8.2 Fork-Per-Connection - Process-Based Concurrency
Why each close is mandatory:
- Child closes listenfd: child will never call accept(). Holding listenfd open wastes an fd slot and inflates the open file table's ref count. If enough children accumulate, the listening socket's resources are never freed.
- Parent closes connfd: parent won't talk to this client. If parent doesn't close connfd, the connection's ref count stays at 2. Even after child finishes (ref count→1), TCP connection CANNOT close - no FIN is sent to client - because parent still holds a reference. Connection hangs open indefinitely, leaking kernel resources.
| FD LEAK PATTERN | A server that forks but doesn't close unused inherited fds accumulates one leaked fd per connection. After thousands of connections, the server hits the fd limit (EMFILE) and can no longer accept new connections. lsof -p <pid> reveals the accumulated fds. Always close every fd you inherit but don't need. |
|---|
8.3 Limitations of Fork-Per-Connection
| Limitation | Why it matters | At what scale it breaks |
|---|---|---|
| fork() overhead | Creating a process takes time - PCB allocation, page table copy, kernel setup | ~1000+ connections/second |
| Memory per process | Each process has its own address space - even with COW, metadata is duplicated | ~10,000+ concurrent clients |
| IPC complexity | Processes can't share variables - need pipes/shared memory for shared state | Any shared state (cache, rate limiting, connection counts) |
| Zombie accumulation | Every dead child needs reaping - requires SIGCHLD handler calling waitpid() | Busy server with no reaping |
| C10K problem | Can't sustain 10,000+ simultaneous connections with one process per connection | 10,000+ concurrent connections |
8.4 The Evolution of Server Concurrency Models
| REAL WORLD | The C10K problem (handling 10,000 simultaneous connections) drove the invention of epoll (Linux) and kqueue (BSD/macOS) in the early 2000s. nginx was specifically designed around epoll to solve C10K. Today Go's goroutines and Rust's Tokio are the standard answers - both abstract over epoll under the hood. |
|---|
9. HTTP - Application Layer Protocol
9.1 HTTP Request Format
9.2 HTTP Response Format
9.3 HTTP Status Codes
| Code | Meaning | Common cause |
|---|---|---|
| 200 OK | Success | Request processed successfully |
| 301 Moved Permanently | Permanent redirect | URL has moved, update bookmarks |
| 302 Found | Temporary redirect | URL temporarily at different location |
| 400 Bad Request | Malformed request | Client sent invalid HTTP |
| 401 Unauthorized | Authentication required | No/invalid credentials |
| 403 Forbidden | Permission denied | Valid credentials but no access |
| 404 Not Found | Resource doesn't exist | Wrong URL or deleted resource |
| 500 Internal Server Error | Server-side bug | Unhandled exception in server code |
| 502 Bad Gateway | Upstream error | Reverse proxy couldn't reach backend |
| 503 Service Unavailable | Server overloaded/down | Too many requests or maintenance |
9.4 HTTP Framing - How It Solves the Byte Stream Problem
HTTP/1.1 uses a combination of delimiter and length-prefix framing:
- Headers: delimited by \r\n. Each header line ends with \r\n. The header block ends with \r\n\r\n (blank line). Read line by line until blank line.
- Body: length specified by Content-Length header. Read exactly that many bytes after the blank line. For streaming: Transfer-Encoding: chunked uses its own framing within the body.
9.5 Static vs Dynamic Content
| Type | What the server does | Example URLs | Modern equivalent |
|---|---|---|---|
| Static content | Finds file on disk, sends it directly. No computation. stat() → open() → send bytes. | /index.html, /style.css, /logo.png | CDN serving files, nginx serving static assets |
| Dynamic content | Forks a child, execs a CGI program, connects its stdout to the socket. Program generates response. | /cgi-bin/search?q=foo, /cgi-bin/login | Web frameworks (Django, Express, Rails), API servers |
11. Quick Reference - Things to Remember Cold
Addressing
- IP address: identifies the machine
- Port: identifies the program (0-1023 privileged, 49152-65535 ephemeral)
- Four-tuple: (client IP:port, server IP:port) uniquely identifies every TCP connection
- Socket address: IP + port combined. Store in sockaddr_in struct.
- Byte order: always htons()/htonl() when putting values into sockaddr structs, ntohs()/ntohl() when reading them out
TCP guarantees (and non-guarantees)
- Guarantees: reliable delivery, ordered bytes, flow control, congestion control
- Does NOT guarantee: message boundaries. TCP is a byte stream. Always frame messages explicitly.
- UDP: no reliability, no ordering, but preserves message boundaries and is faster
Server syscall order
- socket() → create fd
- bind() → assign IP:port to socket
- listen() → mark passive, set backlog queue size
- accept() → block until client connects, return connfd
- read(connfd) / write(connfd) → communicate with client
- close(connfd) → close connection
Client syscall order
- socket() → create fd
- connect() → trigger 3-way handshake, block until done
- read/write → communicate
Three-way handshake one-liner
- SYN: client starts, sends sequence number x
- SYN-ACK: server acknowledges x, sends own sequence number y
- ACK: client acknowledges y. Connection open. Kernel handles all of this - your process is asleep in accept().
Message framing - three approaches
- Fixed length: always read/write exactly N bytes. Simple, inflexible.
- Delimiter: read until \n or \r\n. Used by HTTP headers, Redis, SMTP.
- Length-prefix: send 4-byte length then body. Used by gRPC, PostgreSQL, most binary protocols.
Fork-per-connection rules
- Child: always close(listenfd) immediately after fork
- Parent: always close(connfd) immediately after fork
- Why: open file table ref counts - connection won't fully close until ALL references are closed
HTTP status codes cold
- 200 = OK, 301 = permanent redirect, 302 = temp redirect
- 400 = bad request, 401 = unauthorized, 403 = forbidden, 404 = not found
- 500 = server error, 502 = bad gateway, 503 = unavailable
CSAPP Ch 11 Reference • Network Programming
Top comments (0)