I recently interviewed with a popular streaming platform for a Senior Software Engineer role focused on low-level media streaming. As someone with prior experience in this space, I used the prep process to go deeper into the internals of how streaming actually works. The interview went well, but the role wasn't quite the right fit at the time. In an effort not let that knowledge go to waste, I thought I'd write an article to share what I learned during this period.
This article is the first in a series introducing how media streaming works on the client side: the steps involved in getting a piece of media from a server onto a client device over the internet, so off we go!
The Problem with Downloading
I believe at some point you've consume some sort of media on the internet today with either your phone, your laptop, maybe even a pager if you're reading this from 1998. Whether it's a short clip on Instagram or X(formerly Twitter), or a two-hour film on Netflix or Amazon Prime, or maybe even listening to your favourite music on Spotify that content had to get to your device somehow.
These media content can be delivered in various ways but we'd be looking at two of them. In the early days of the internet, there was really just one go-to option, downloading the entire file before playback actually starts. Yeah that's works fine for a few second clip, but where the problem lies is when a long form media content is being served to a user. Downloading an hour of 720p video over a 4G connection can take several minutes depending on your connection quality, now imagine how that scales when the media resolution is larger with a longer duration. Nobody wants to stare at a loading bar for ten minutes before a movie starts.
Enters Streaming
The solution to this was streaming. In context media, streaming is the process of continually delivering multimedia content, like video, audio, or subtitles over the internet in near realtime.
Instead of waiting for the entire file to download, streaming breaks the media into small chunks (typically a few seconds each) that are delivered and buffered continuously. Playback can start as soon as the first chunk of the media arrives, rather than waiting for the whole file. See how that solves the download problem?...now let's dive a bit deeper into how these streaming technologies work.
Before playback can start with streaming, the player(client) needs to know what to fetch and in what order. That's the job of a manifest file, a kind of playlist that lists the available chunks and quality levels. depending on the streaming protocol, this manifest file can be formatted differently. HLS uses a .m3u8 manifests, while MPEG-DASH uses .mpd manifests. We won't go deep on these yet, but it's worth knowing they exist: they're the reason a player can jump between quality levels mid-playback without you noticing something we'll cover properly in Part 2 when we get into adaptive bitrate streaming.
The Two Big Streaming Technologies
Since streaming's early days, several technologies have emerged, but two dominate today RIP Adobe Flash:
- HTTP Live Streaming (HLS) β developed by Apple
- Dynamic Adaptive Streaming over HTTP (DASH), also known as MPEG-DASH
Both work on the same core idea described above: chunked delivery guided by a manifest. Before any video can be delivered this way, though, it needs to be prepared, encoded, packaged, and often encrypted. Tools like FFmpeg and Shaka Packager handle this: taking a raw, hour long video file and turning it into the chunks and manifests a streaming player can actually use.
Containers and Codecs
To support an the vast array of browsers, OS, and devices available today you can't rely on just a single media format. This is where containers and codecs come in, two terms that get used interchangeably but mean different things.
think of container as just that, a shell for housing multiple data streams maybe an audio, video and or subtitle stream, bundled into one file. An MP4 file, for example, is a container that might hold an H.264 video stream and an AAC audio stream inside it.
Each of those individual streams is compressed using a codec(encoder/decoder), an algorithm designed to shrink raw, usually oversized media data into something efficient to store and transmit over the internet. some of the common video codecs include:
- H.264 / AVC (Advanced Video Coding)
- H.265 / HEVC (High Efficiency Video Coding), the successor to H.264
- AV1, a newer, royalty-free codec
And common audio codecs include:
- AAC (Advanced Audio Coding)
- Opus
- FLAC
The codec is what does the heavy lifting of compression; the container is just the box everything ships in ;)
What's Next
To keep this article from running longer than the Dead Sea Scrolls, I'll stop here. Quick recap of what we covered:
- Why downloading doesn't scale for long-form video, and how streaming solves it
- How chunked delivery and manifest files work together
- HLS and DASH as the two dominant streaming protocols
- The difference between containers (the shell) and codecs (the compression)
In Part 2, we'll dig into bitrate vs. resolution and how they affect playback quality, how adaptive bitrate streaming actually decides what to switch to and when, and how premium content gets protected with DRM and how the client platform handle securing these files.
Stay tuned!
Top comments (0)