TOKUJI

Posted on Jun 21

How a from-scratch HTTP/2 server actually works (part 2) — flow control, DATA, and security

#webdev #python #http #learning

Recap from part 1: We traced a connection from the 24-byte preface through SETTINGS exchange, the 9-byte frame header, the seven-loop guard tower, and stream birth — how a HEADERS frame passes admission control and creates a new stream. The stream is now OPEN. This part follows what happens next: body delivery, stream death, connection shutdown, and the security guards every from-scratch implementer needs.

1. Body delivery — DATA, flow control, content-length (§6.1, §6.9.1)

Once the stream is OPEN, the peer sends DATA frames. Each DATA frame consumes flow-control credit — and HTTP/2 has two independent windows.

Why two windows?

The stream window stops any one stream from hogging the connection — fairness between streams. The connection window caps the total data in flight across all streams — a bound on how much the receiver has to buffer. Both must have positive credit before a DATA frame may be sent.

The connection window is the easy half to forget, and forgetting it fails late: credit only the stream window and everything works until ~65,535 cumulative bytes have flowed across all streams, at which point the shared connection window reaches zero and every stream stalls — even ones with plenty of their own credit.

After delivering a DATA frame to the application, HTTP2Actor._on_data_frame() sends two WINDOW_UPDATEs:

# HTTP2Actor._on_data_frame() — after delivering DATA to the application
await self.send_frame(factory.window_update(stream.stream_id, frame.length))  # stream
await self.send_frame(factory.window_update(0, frame.length))                  # connection

Back-pressure

The WINDOW_UPDATE is sent only after put_DATAFrame returns True — the recipient's queue accepted the frame. A full application queue withholds the credit, naturally stalling the peer. This is the RFC's intended back-pressure mechanism: flow control doubles as a signalling channel between the TCP receive buffer and the application handler.

Content-length enforcement (§8.1.2.6)

stream.received_data_bytes accumulates across DATA frames (padding excluded). If the declared content-length is exceeded on any frame → RST_STREAM PROTOCOL_ERROR immediately. If END_STREAM arrives and the total is short → same error. §8.1.1 makes the server, not the application, responsible for this check — a malformed request must never reach handler code.

HALF_CLOSED_REMOTE

When a DATA frame carries END_STREAM, the stream transitions from OPEN to HALF_CLOSED_REMOTE. The peer has finished sending. DATA on a stream already in HALF_CLOSED_REMOTE or CLOSED → RST_STREAM STREAM_CLOSED.

2. Stream death — cleanup and late frames (§5.1)

A stream reaches CLOSED when either the response END_STREAM is sent or an RST_STREAM is received. At that point HTTP2Actor._make_done_cb() — the task done-callback — fires:

# HTTP2Actor._make_done_cb()
self._closed_streams[stream_id] = False  # closed via END_STREAM (closed_via_rst=False)

The Stream node is pruned from the priority tree, but the stream ID lives on in _closed_streams: dict[int, bool]. This is important: keeping a full Stream object per completed request would cost hundreds of bytes each; a dict entry costs ~72 bytes. The boolean tracks how the stream closed — False means clean END_STREAM, True means RST_STREAM — which affects which error code a late frame triggers.

Late frames on a closed stream still hit the CLOSED branch of §5.1 validation using just an integer lookup. PRIORITY is always allowed through; HEADERS and CONTINUATION on a closed stream are a connection STREAM_CLOSED error; everything else gets a stream-level RST_STREAM.

3. Connection death — GOAWAY (§6.8)

GOAWAY is asymmetric.

Incoming (HTTP2Actor._on_goaway_frame()): the peer is closing the connection. We echo a GOAWAY back so the peer knows which stream IDs we processed, then inject http.disconnect into every active recipient and return from _frame_loop:

async def _on_goaway_frame(self, last_stream_id: int) -> None:
    await self.send_frame(self.factory.goaway(last_stream_id))
    _signal_recipients(self._recipients)

Outgoing (HTTP2Actor._connection_error()): we detected a protocol error and must close. Build a GOAWAY with _last_peer_stream_id (the highest stream we've ever seen from the peer), flush it, then call writer.close() so the peer sees FIN after the GOAWAY. The _goaway_sent flag makes this idempotent — a second connection error doesn't send a second GOAWAY.

Sidebar: WebSocket over HTTP/2 (RFC 8441) — the actor reuse trick

RFC 8441 bootstraps WebSocket over HTTP/2 via an extended CONNECT request with :protocol=websocket. There's no 101 Switching Protocols — the response is a 200 HEADERS frame and data flows as DATA frames on the same stream.

BlackBull reuses WebSocketActor without modification. WebSocketActor._send() reads a callback from scope['_ws_send_101']. For HTTP/2 WebSocket, that callback is replaced with _ws_send_200, which sends a 200 HEADERS frame instead of 101. The actor never knows it's running over HTTP/2.

HTTP2WSReader and HTTP2WSWriter bridge the HTTP/2 frame-level I/O to the WebSocket actor's readexactly / write interface, and they implement backpressures_via_credit = True — flow control credit is withheld until the reader buffer drains, using HTTP/2's own flow-control mechanism as the back-pressure signal.

Sidebar: Three security guards

Rapid Reset (CVE-2023-44487, RFC 9113 §10.5). Disclosed in October 2023, this attack sends HEADERS + RST_STREAM in a tight loop — the server spawns a task for each stream, but the stream is already reset by the time the handler starts. SETTINGS_MAX_CONCURRENT_STREAMS doesn't help because the streams close too fast to accumulate. BlackBull uses a rolling 1-second RST_STREAM rate limit (default 20/sec) in _frame_loop(), checked before stream-state validation so RSTs on idle/unknown streams also count. Exceeded → GOAWAY ENHANCE_YOUR_CALM.

CONTINUATION flood (CVE-2024-27983, RFC 9113 §10.5.1). A HEADERS frame with END_HEADERS=0 followed by unlimited CONTINUATION frames can grow the header accumulator until the process OOMs. BlackBull checks len(header_frame.raw_block) > self._header_max_total (default 64 KiB — the same budget as HTTP/1.1) and resets with ENHANCE_YOUR_CALM before calling parse_payload(). nginx and Envoy use the same approach and the same error code.

stream_id==0 for stream-only frames (RFC 9113 §6.1, §6.2, §6.4, §6.6, §6.10). Less flashy but load-bearing: DATA, HEADERS, RST_STREAM, PUSH_PROMISE, PRIORITY, and CONTINUATION MUST NOT appear on stream 0. _STREAM_ONLY_FRAME_TYPES is a class-level frozenset that consolidates six independent checks into one lookup. Violation → connection PROTOCOL_ERROR. BlackBull fixed the PUSH_PROMISE and RST_STREAM cases (previously missing from the individual checks) in v0.42.1 and v0.42.2 respectively.

Where to go from here

The full RFC→code table, with section numbers, method names, and file:line references, lives at
docs/about/rfc9113-implementation.md in the BlackBull docs.

Part 1 covered the connection preface, frame reading, the seven-loop guard tower, and stream birth — read it if you want the full lifecycle from the beginning.

The source is blackbull/server/http2_actor.py — 1200+ lines, nearly every block annotated with its RFC cite. gRPC over HTTP/2 is the natural next step: the frame layer already handles multiplexed streams; what's missing is the Protobuf framing and trailers.

BlackBull is a personal learning project — pure-Python HTTP/1.1, HTTP/2, and WebSocket, no C extensions in the protocol stack. Issues and PRs welcome.

DEV Community