HTTP Chunk Encoding and Transfer-Encoding: chunked in Practice

Originally published on Code Beneath.

When you need to send an HTTP response but do not know the total size in advance, the chunked transfer encoding is your tool. Instead of buffering everything or setting Content-Length, you declare Transfer-Encoding: chunked and send data in discrete chunks. This is how live streams, real-time logs, and server-sent events work beneath the framework abstractions. An HTTP transfer encoding chunked example implementation shows the real wire format: chunk size in hex, the data, a CRLF pair, repeat, then a final zero-sized chunk with optional trailers. This article walks through working code that does exactly this, why size selection matters, and the gotchas that trap most engineers the first time.

Why Chunked Transfer Encoding

In HTTP/1.1, a response must tell the client how many bytes are coming. You either set Content-Length (which means you must know the size before you start sending headers) or you close the connection (HTTP/1.0 style, wasteful). Chunked encoding lets you send headers immediately, start streaming data, and signal the end later. The server never waits to buffer the entire body.

This is critical for:

• Long-running computations that produce output incrementally
• Large files served from disk without precomputing size
• Real-time data feeds (stock tickers, log tail, metrics)
• Server-sent events (SSE), which relies on chunked encoding by default
• Proxies that do not know downstream content length

The HTTP/1.1 spec (RFC 7230, Section 4.1) defines the wire format precisely. You send each chunk as: a line with the chunk size in hexadecimal, a CRLF pair (carriage return, line feed: "\r\n"), the chunk data itself, another CRLF pair, and repeat. The stream ends with a chunk of size 0 followed by optional trailer headers and a final CRLF.

The Wire Format in Detail

Let me show you exactly what travels over the socket:

5\r\n
Hello\r\n
6\r\n
 World\r\n
0\r\n
\r\n

The "5" is hex for 5 bytes. Then 5 bytes of data ("Hello"). Then "6" for 6 bytes ("space World"). Then "0" signals the end, followed by a blank line (the final CRLF). The client reads the hex size, reads exactly that many bytes, discards the trailing CRLF, and repeats until it sees a zero chunk. Simple, but easy to get wrong when writing raw sockets.

Go Implementation: Manual Chunk Writing

Go's http.ResponseWriter detects when you do not set Content-Length and automatically uses chunked encoding. But let us do it manually first to show what really happens:

package main

import (
    "bufio"
    "fmt"
    "net"
    "time"
)

func main() {
    listener, err := net.Listen("tcp", ":8080")
    if err != nil {
        panic(err)
    }
    defer listener.Close()

    fmt.Println("Listening on :8080")

    for {
        conn, err := listener.Accept()
        if err != nil {
            panic(err)
        }
        go handleConnection(conn)
    }
}

func handleConnection(conn net.Conn) {
    defer conn.Close()

    reader := bufio.NewReader(conn)
    // Read the HTTP request line
    line, err := reader.ReadString('\n')
    if err != nil {
        return
    }

    // Consume the rest of the headers
    for {
        hline, err := reader.ReadString('\n')
        if err != nil {
            return
        }
        if hline == "\r\n" {
            break
        }
    }

    // Send HTTP response headers with chunked encoding
    response := "HTTP/1.1 200 OK\r\n"
    response += "Transfer-Encoding: chunked\r\n"
    response += "Content-Type: text/plain\r\n"
    response += "\r\n"

    conn.Write([]byte(response))

    writer := bufio.NewWriter(conn)

    // Send chunks manually
    chunks := []string{"Hello", " ", "World", "!", "\n", "Streaming ", "data."}

    for _, data := range chunks {
        // Write chunk size in hex
        chunkSize := fmt.Sprintf("%x\r\n", len(data))
        writer.WriteString(chunkSize)

        // Write chunk data
        writer.WriteString(data)

        // Write CRLF after data
        writer.WriteString("\r\n")

        writer.Flush()

        time.Sleep(500 * time.Millisecond)
    }

    // Send final zero-sized chunk (terminator)
    writer.WriteString("0\r\n")
    writer.WriteString("\r\n")
    writer.Flush()
}

Run this with `go run main.go` and connect with `curl -v http://localhost:8080`. You will see each chunk arrive as a separate write over the socket. curl will buffer and reassemble them transparently. The key lines:

• `fmt.Sprintf("%x\r\n", len(data))` converts the byte count to hex and adds a CRLF
• Every chunk has data and a trailing CRLF
• The terminator "0\r\n\r\n" signals the end (chunk size 0, then blank line for optional trailers)
• `writer.Flush()` forces the data onto the socket immediately, simulating streaming

Python Implementation with Real-Time Output

Python's http.server.BaseHTTPRequestHandler hides chunking by default. Here is a low-level socket version:

#!/usr/bin/env python3

import socket
import time

def handle_client(client_socket, addr):
    # Read request
    request = client_socket.recv(4096).decode('utf-8')
    if not request:
        client_socket.close()
        return

    # Send HTTP headers
    response_headers = (
        "HTTP/1.1 200 OK\r\n"
        "Transfer-Encoding: chunked\r\n"
        "Content-Type: text/plain\r\n"
        "\r\n"
    )
    client_socket.sendall(response_headers.encode('utf-8'))

    # Generator that yields chunks
    def stream_chunks():
        yield "Line 1: Processing started\n"
        time.sleep(0.5)
        yield "Line 2: Still working\n"
        time.sleep(0.5)
        yield "Line 3: Almost done\n"
        time.sleep(0.5)
        yield "Line 4: Complete\n"

    # Send each chunk
    for data in stream_chunks():
        # Chunk size in hex
        chunk_size = hex(len(data))[2:]  # Remove '0x' prefix

        # Format: size\r\ndata\r\n
        chunk_line = f"{chunk_size}\r\n"
        client_socket.sendall(chunk_line.encode('utf-8'))
        client_socket.sendall(data.encode('utf-8'))
        client_socket.sendall(b"\r\n")

    # Send terminating chunk
    client_socket.sendall(b"0\r\n\r\n")
    client_socket.close()

def main():
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_socket.bind(('127.0.0.1', 8081))
    server_socket.listen(5)

    print("Server listening on 127.0.0.1:8081")

    try:
        while True:
            client_socket, addr = server_socket.accept()
            print(f"Connection from {addr}")
            handle_client(client_socket, addr)
    except KeyboardInterrupt:
        print("Shutting down")
    finally:
        server_socket.close()

if __name__ == "__main__":
    main()

Run this with `python3 main.py` and curl it:

curl -v http://127.0.0.1:8081

You will see the response arrive in real time, 500ms apart, with each chunk size printed by curl. The hex conversion (`hex(len(data))[2:]`) is the Python equivalent of Go's `fmt.Sprintf("%x", ...)`.

Chunk Size Selection

How big should each chunk be? There is no magic number. Common strategies:

Strategy	Size	Use Case
Line-based	Variable (each line)	Logs, line-delimited JSON, text streams
Fixed size	4KB, 8KB, 64KB	File streaming, general-purpose
Semantic unit	1 JSON object, 1 frame	SSE, protocol buffers, video frames
Flush on timeout	Buffered + 100ms	Interactive streams (avoid latency)

Smaller chunks mean lower latency (client sees data sooner) but more overhead (more CRLF pairs, more writes). Larger chunks mean better throughput but higher latency. For interactive streams, prefer smaller chunks or flush on timeout. For bulk file transfer, 64KB chunks are standard.

Common Mistakes

Forgetting the trailing CRLF after data

Every chunk must have a CRLF after the data itself, not just after the size line. Missing this breaks the format and hangs the client.

Not flushing to the socket

Buffering all your chunks and sending them at once defeats the purpose of chunked encoding. Call `flush()` (Python) or `Flush()` (Go bufio) after each chunk if you want streaming behavior.

Mixing chunked with Content-Length

Set Transfer-Encoding: chunked OR Content-Length, never both (HTTP/1.1 forbids it). If both are present, Transfer-Encoding takes precedence and Content-Length is ignored, leading to confusion.

Forgetting the terminating zero chunk

The client waits for "0\r\n\r\n" to know the response is complete. Without it, the client hangs. Always send it, even if you send nothing else.

Sending non-hex chunk sizes

The size must be in hexadecimal, lowercase or uppercase (RFC says lowercase is preferred). Sending decimal sizes (e.g., "10" for 10 bytes instead of "a") creates a protocol violation.

Trailers: Sending Headers After Data

Chunked encoding allows optional trailer headers after the final chunk. This is useful for checksums, signatures, or metadata computed during streaming:

// In Go, after the last chunk:
writer.WriteString("0\r\n")
writer.WriteString("X-Checksum: abc123\r\n")
writer.WriteString("X-Rows-Processed: 1000\r\n")
writer.WriteString("\r\n")
writer.Flush()

The format is: zero chunk size, then trailer headers in the same format as response headers, then a final CRLF. Not all clients support trailers (trailers were poorly supported in HTTP/1.1 and are better in HTTP/2), so test thoroughly. For dynamic HTTP response types, trailers can signal computed metadata.

Browser and Client Behavior

Browsers handle chunked responses transparently. curl, wget, and httpx all reassemble chunks into the full body before the application sees it. This is why developers often forget chunking exists: the abstraction hides it. The real test is watching with tcpdump or Wireshark to confirm chunks arrive separately on the wire.

Some differences:

• Browsers do not display partial content incrementally by default (though they do for SSE with text/event-stream)
• CLI tools like curl buffer everything before printing
• JavaScript fetch() receives events as chunks arrive if you use a ReadableStream
• HTTP/2 and HTTP/3 do not use chunked encoding at all (they have native streaming), so Transfer-Encoding: chunked is HTTP/1.1 specific

See HTTP basics for more on protocol versions.

Testing and Debugging

To see the raw bytes on the wire:

curl --raw http://localhost:8080 | od -c

Or with netcat:

echo -e "GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" | nc localhost 8080 | od -c

You will see the hex chunk sizes, the data, and the CRLF sequences clearly. Wireshark is also invaluable: capture port 8080, filter by "http.transfer_encoding == chunked", and inspect the frames.

FAQ

Do I have to use chunked encoding if I don't know Content-Length?

No. You can close the connection after the body (Connection: close), and HTTP/1.0 clients will work. But chunked encoding is cleaner for HTTP/1.1 because it allows Connection: keep-alive and pipelining.

What happens if I send a chunk size larger than the actual data?

The client reads that many bytes and hangs waiting for more. You have created a protocol violation. The client times out, confused. Always send the exact number of bytes you declare in the chunk size.

Can I send chunks in different orders or out of sequence?

No. The HTTP/1.1 specification requires chunks to arrive in order and to be processed sequentially. Reordering them breaks the protocol.

Does HTTP/2 support chunked encoding?

No. HTTP/2 uses frames instead, with native streaming support. The Transfer-Encoding header is not sent over HTTP/2 connections. The proxy or server translates chunked to frames transparently.

What is the maximum chunk size?

Theoretically, the chunk size hex can be as large as your system's integer can represent. Practically, most servers and clients limit it to 2GB or less. Stick to reasonable sizes (4KB to 1MB) unless you have a specific reason not to.

Should I compress chunked responses?

Yes. You can send Transfer-Encoding: chunked and Content-Encoding: gzip together. Many servers (nginx, Apache) do this automatically. Write your data to a gzip compressor, then chunk the compressed output.