Take a 1 MiB receive window. A peer sends 256 KiB, leaving 768 KiB of credit. The receiver takes that buffer out of its queue and starts writing it to a socket. The socket stalls.
The queue is empty, but the buffer still occupies 256 KiB. Returning credit at that point would let replacement data arrive while the write was still pending.
BibaVPN keeps those bytes accounted for until write_all() completes. This is part of the TCP multiplexer in my experimental Rust tunnel, which carries several logical streams through a shared WebSocket connection.
Conceptual receive path. The independent pumps separate socket writes from shared record dispatch; all streams still share the outer TCP transport.
A mux record starts with a nine-byte header:
Offset 0 4 5 9
+-----------------+-------+-----------------+---------+
| stream_id | flags | payload_length | payload |
| u32, big-endian | u8 | u32, big-endian | |
+-----------------+-------+-----------------+---------+
The decoder checks the declared payload length and rejects trailing bytes. Once decoded, the stream ID tells the receiver where to dispatch the record. The flags identify operations such as OPEN, DATA, CLOSE, RST, and WIN.
Writing directly to a destination socket from the shared reader would make every stream depend on that destination's speed. Instead, the implementation keeps per-stream receive state and separate pumps for the two directions. Its optional, negotiated byte-credit extension controls how much data can remain outstanding.
Here is the 256 KiB example through one complete receive cycle:
Worked example in negotiated credit mode. No other data arrives during this sequence; the empty queue triggers publication of the accumulated credit after the write.
The relevant operations in tcp_mux_flow.rs are ordered like this:
write.write_all(&data).await?;
let len = data.len();
drop(data);
flow.consumed(len);
The window-update logic follows those operations. It batches consumed bytes and publishes credit when the accumulated amount reaches one eighth of the receive limit, or when the queue is empty. While the socket write is pending, the buffer remains charged to the stream.
There is a second limit at session level. Each admitted stream reserves its local receive allowance against a 64 MiB logical receive budget:
| Local receive window | Reservations that fit in 64 MiB |
|---|---|
| 1 MiB | 64 |
| 2 MiB | 32 |
| 3 MiB | 21 |
| 4 MiB | 16 |
These are admission-budget calculations. Process memory also includes retained allocations, output queues, TLS state, tasks, and other overhead.
bytes::Bytes makes the allocation detail worth checking. A short slice can retain the allocation from which it was taken:
Backing allocation: 8 KiB
+-------------------------------------------------------------+
| header / padding | payload slice | unused capacity |
+-------------------------------------------------------------+
^^^^^^^^^^^^^
visible bytes
Keeping the slice can keep the entire allocation alive.
Counting only slice.len() can therefore understate memory retained by queued data. The receive implementation includes compaction and backing-allocation accounting alongside its logical byte limits.
Closing has a related lifetime problem. A client may finish sending its request while it still expects a response. In negotiated mode, CLOSE is directional: the other pump can continue. Legacy mode keeps its older behavior of closing both directions.
None of the per-stream accounting changes TCP's ordered delivery. Packet loss on one outer connection can still delay data for every logical stream carried by it. Separate receive queues address slow destination sockets inside the application; outer transport loss remains shared.
For code review, the useful places to start are Flow::enqueue, Flow::receive, Flow::consumed, and the downlink write loop. Together they show when a buffer enters the accounting, who owns it while a write is blocked, and when the peer gets permission to send more.
Source code, protocol documentation, and tests: BibaVPN on GitHub.
Top comments (0)