A one-to-one video call is easy: two peers send packets directly to each other. A forty-person meeting is where the interesting engineering starts, because the naive approach falls apart fast, and the fix explains how every serious video platform is built.
Why not just connect everyone to everyone
In a mesh, each participant sends their video to every other participant. With N people that is N minus 1 uploads from each device. At four people it is fine. At twenty, every laptop is trying to encode and upload nineteen copies of its own video, which no consumer uplink can handle. Mesh does not scale past a handful of people. You need a server in the middle.
SFU vs MCU
There are two kinds of server in the middle, and the difference is the central design decision.
An MCU (multipoint control unit) receives every participant's stream, decodes them, composites them into a single combined video, re-encodes it, and sends one stream back to each person. The client's job is trivial: it uploads one stream and downloads one stream. The cost is brutal on the server. Decoding and re-encoding video for every meeting is enormously CPU-heavy, and the re-encode adds latency and degrades quality.
An SFU (selective forwarding unit) does far less. It receives each participant's stream and simply forwards the packets to the others, without decoding or mixing. Each client uploads one stream but downloads one stream per other participant, then arranges them locally. The server is now cheap, just routing packets, which is why SFU is what almost everyone uses for large calls. The trade-off is more download bandwidth and more decoding work on the client, but modern devices handle that better than servers handle mass transcoding.
WebRTC and why UDP
The transport underneath is WebRTC, and the key choice it makes is UDP over TCP. TCP guarantees every byte arrives in order, retransmitting anything lost. For real-time video that guarantee is a liability. A packet that arrives 300 milliseconds late because it was retransmitted is useless, the moment it described has already passed. Better to drop it and move on. UDP does exactly that: it does not wait, does not retransmit by default, and lets the video codec conceal the occasional missing packet. Late video is worse than lost video.
Simulcast: the trick that makes SFU work
Here is the problem an SFU creates. It is just forwarding one stream from each sender, but the receivers are wildly different. One is on a laptop on fiber that wants full resolution; another is on a phone on spotty cellular that can barely handle a thumbnail. If the sender uploads one high-quality stream, the phone drowns. If it uploads one low-quality stream, the laptop gets a blurry picture.
Simulcast solves it by having each sender upload several versions of its video at once, say high, medium, and low resolution, as separate streams. The SFU then forwards the appropriate version to each receiver based on that receiver's measured bandwidth and the size it is displaying the video at. The person shown in a small tile gets the low stream; the active speaker shown large gets the high stream. The sender does a bit more encoding work, but the SFU can match every receiver's capacity without any transcoding. This is how one call serves a phone and a workstation gracefully.
The trade-offs
SFU plus simulcast is the modern default because it keeps the server cheap and adapts to heterogeneous clients. The costs are real: clients decode many streams, and uploading three resolutions raises the sender's outbound bandwidth. For very large webinars with thousands of passive viewers, platforms shift those viewers to a one-way streaming path (HLS or similar) since they only watch and never send, and pure real-time latency matters less for an audience than for participants.
How the real systems do it
Zoom, Google Meet, and Microsoft Teams all run SFU-based architectures with simulcast (or its cousin, scalable video coding) so one meeting adapts across devices and networks. WebRTC over UDP is the common transport, with the codec doing packet-loss concealment rather than the transport retransmitting. The through-line is the same everywhere: forward, do not mix; send multiple qualities; and never wait for a late packet.
I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-zoom
Top comments (0)