Yet Another Article on Media Streaming: Manifest parsing, Buffer management, Demuxing, Decoding.

Hey there! In Part 1 and Part 2, we covered the network side of streaming: why chunked delivery beats downloading, how manifests and ABR work, and how DRM protects content in transit. That's the "getting the bytes to the device" story.

This part picks up right where the bytes land. A segment has arrived on the device, encrypted or not, and now something has to turn it into pixels and sound coming out of your speakers within milliseconds. That's the playback pipeline, and it's where a lot of my day to day work as an Android engineer actually lives. I'll be leaning on Media3/ExoPlayer for the code examples since that's the stack I know best, but the concepts apply broadly to any modern player(AVFoundation/ShakaPlayer).

Here's the rough shape of what we're covering, in the order it actually happens:

Manifest → Segment Fetch → Buffer → Demux → Decode → Sync → Render

Let's Dive into them.

Manifest Parsing

Before anything can play, the player needs to know what it's dealing with, how many renditions exist, where the segments live, what codecs are in use, whether there's DRM involved. That's what the manifest (.m3u8 for HLS, .mpd for DASH) tells it.

Parsing this isn't just abount reading a file, the player builds an internal representation of the whole stream structure: available tracks (video, audio, subtitles), the rendition ladder for each, segment URLs or byte ranges, and timing information. In Media3, this is handled by a MediaSource for adaptive content specifically, HlsMediaSource or DashMediaSource.

val mediaItem = MediaItem.Builder()
    .setUri("https://example.com/stream.mpd")
    .setMimeType(MimeTypes.APPLICATION_MPD)
    .build()

val mediaSource = DashMediaSource.Factory(dataSourceFactory)
    .createMediaSource(mediaItem)

exoPlayer.setMediaSource(mediaSource)
exoPlayer.prepare()

That prepare() call is doing more than it looks like, under the hood, ExoPlayer fetches the manifest, parses it, and builds out the track structure before it can fetch a single video segment.

Segment Fetching

Once the player knows what's available, it needs to actually pull segments over the network and this is where the ABR decision from Part 2 gets executed in practice. For every segment, the player's track selector picks a rendition, and a DataSource fetches the corresponding chunk.

The Player prefetches chunks ahead of the playback position to keep the buffer healthy, and the LoadControl in ExoPlayer is what governs how aggressively that happens.

val loadControl = DefaultLoadControl.Builder()
    .setBufferDurationsMs(
        /* minBufferMs = */ 15_000,
        /* maxBufferMs = */ 50_000,
        /* bufferForPlaybackMs = */ 2_500,
        /* bufferForPlaybackAfterRebufferMs = */ 5_000
    )
    .build()

val exoPlayer = ExoPlayer.Builder(context)
    .setLoadControl(loadControl)
    .build()

bufferForPlaybackMs is worth calling out specifically as it's the minimum amount of buffered media required before playback starts at all, which is a direct trade off between "start playing fast" and "don't stall two seconds in."

Buffer Management

Fetched segments don't go straight to the decoder. They land in a buffer first, which exists for one reason: to absorb network variability so playback doesn't stall every time throughput dips for a moment.

The buffer isn't one undifferentiated pool, either. There's a distinction worth knowing:

• Media buffer compressed segment data that's been downloaded but not yet decoded
• Decoded/output buffer frames that have already been decoded and are waiting to be rendered

The minBufferMs/maxBufferMs values above are what keep the media buffer in a healthy range too little, and any network hiccup causes a stall; too much, and you're wasting memory and bandwidth on segments that might get abandoned anyway if the ABR algorithm decides to switch renditions.

Demuxing

Here's a term from Part 1 worth revisiting: a container (like an .mp4 or .ts segment) holds multiple streams bundled together — video, audio, sometimes subtitles. Before any of those can be decoded, they need to be pulled apart. That's demuxing (short for demultiplexing).

A demuxer reads the container format, identifies each embedded stream, and routes the compressed video frames to a video decoder and the compressed audio frames to an audio decoder, in parallel. This matters because video and audio are decoded completely independently at this stage they only get reunited later, at the sync stage.

In ExoPlayer, this is handled by an Extractor (there's one per container format — Mp4Extractor, TsExtractor, FragmentedMp4Extractor, and so on), and it's mostly invisible to us as developers unless you're working with a custom or unusual container format(we'll talk about this another article).

Decoding

This is where compressed data actually becomes raw, playable frames, and it's also the most hardware-dependent stage in the whole pipeline.

On Android, this is MediaCodec's job. Every device ships with a set of hardware decoders (fast, power-efficient, but fixed at the chip level) for common codecs like H.264/AVC and often H.265/HEVC. When a hardware decoder is available for the stream's codec, the player uses it.

val trackSelector = DefaultTrackSelector(context)
trackSelector.parameters = trackSelector.buildUponParameters()
    .setTunnelingEnabled(true)
    .build()

(Tunneling here lets the hardware decoder pipe decoded frames directly toward the display in supported cases, cutting out extra copies, a small optimization, but a good example of how much is happening below the API surface you normally touch.)

What happens when the device can't decode a codec?

This is worth its own section, because it's a real problem, not a theoretical edge case. Say your stream was encoded in AV1, but you're playing it on a mid-range device from a few years back with no AV1 hardware decoder. What now?

we've got a few options, roughly in order of preference:

1. Serve a different rendition. If the manifest includes a fallback codec (say, H.264 alongside AV1), the player can just pick a rendition it can actually decode. This is the cleanest fix and why most large platforms encode in multiple codecs, not just the newest/most efficient one.
2. Fall back to software decoding. Every codec also has a software decoder implementation that runs on the CPU instead of dedicated hardware. It works everywhere, but it's slower and burns significantly more battery fine for short clips, rough for a two-hour movie on a phone at 30% battery(faced a similar issue earlier in my career with FFmpeg).
3. Register a custom decoder extension. Media3 supports this directly, you can plug in a software codec implementation (Google publishes extensions for AV1 and others) that ExoPlayer will fall back to when no hardware decoder matches.

val renderersFactory = DefaultRenderersFactory(context)
    .setExtensionRendererMode(
        DefaultRenderersFactory.EXTENSION_RENDERER_MODE_ON
    )

val exoPlayer = ExoPlayer.Builder(context, renderersFactory).build()

Setting EXTENSION_RENDERER_MODE_ON tells ExoPlayer to consider extension decoders (like the AV1 software extension) as a fallback if the platform's built-in MediaCodec decoders can't handle the stream — it tries hardware/platform decoders first, and only falls back to the extension if nothing else works. There's also EXTENSION_RENDERER_MODE_PREFER, which flips that priority, though that's a less common choice since hardware decoding is almost always the better default.

This is a genuinely practical thing to get right: pick the wrong strategy and you either get playback failures on real devices in the field, or you get playback that "works" but tanks battery life and generates support tickets you can't easily explain.

Audio/Video Synchronization

Remember how demuxing split audio and video into separate decode paths? They now need to come back together — and this is trickier than it sounds, because audio and video decode at different rates and have different latency characteristics.

The mechanism that ties them back together is the presentation timestamp (PTS) every decoded frame, audio or video, carries a timestamp saying exactly when it should be presented to the viewer. The player picks one clock as the reference (almost always the audio clock, since audio glitches are far more noticeable to human ears than a dropped video frame) and continuously adjusts video rendering to match it — dropping a video frame if it's fallen behind, or holding one briefly if it's ahead.

This is also why you'll occasionally see a video frame get skipped entirely under heavy load rather than the whole playback stalling — the player is prioritizing staying in sync over rendering every single frame.

Rendering to the Screen

The final stage: decoded frames need to actually reach the display, and decoded audio needs to reach the speakers.

For video, decoded frames are typically handed off to a Surface — on Android, this is usually backed by a SurfaceView or TextureView, which composites with the rest of the UI and gets pushed to the display at the device's refresh rate. For audio, decoded PCM data goes to an AudioTrack, which queues it for playback through the device's audio pipeline.

playerView.player = exoPlayer

That one line is doing a lot of quiet work — PlayerView wires up the Surface the video renderer writes to, alongside playback controls, subtitle rendering, and the various render-state callbacks.

How ExoPlayer Orchestrates All of This

Pulling it together: ExoPlayer itself doesn't do the demuxing, decoding, or rendering directly. It's an orchestrator — it owns the playback timeline and state machine, and delegates the actual work to a set of collaborating components:

• MediaSource manifest parsing and segment/track structure
• LoadControl buffer thresholds and prefetch behavior
• TrackSelector which rendition and codec to use, including ABR decisions
• Renderer (one per track type, video, audio, text) demuxing, decoding, and rendering for that track
• MediaClock the timing authority that keeps renderers in sync

Each Renderer runs its own decode loop, pulling samples from the MediaSource, feeding them to a decoder (MediaCodec or an extension), and pushing output toward the display or audio track all while checking in against the shared clock to stay synchronized.

If you take one thing away from this article, it's this: everything from "the video looks blurry for a second" to "the app crashed trying to play this file" traces back to one of these seven stages. Knowing which stage you're debugging is most of the battle when something goes wrong in a media player.

Wrapping Up

Quick recap:

• Manifest parsing builds the player's internal picture of the stream — tracks, renditions, timing
• Segment fetching executes ABR decisions and prefetches ahead to keep the buffer healthy
• Buffering absorbs network variability so playback doesn't stall on every dip
• Demuxing splits a container into independent audio and video streams
• Decoding turns compressed data into raw frames — hardware-first, with software fallback or custom extensions when the device can't handle the codec natively
• A/V sync uses presentation timestamps, anchored to the audio clock, to keep everything aligned
• Rendering pushes decoded frames and audio to the screen and speakers
• ExoPlayer orchestrates all of it through MediaSource, LoadControl, TrackSelector, and per-track Renderers, all checking in against a shared clock

That's the full pipeline from segment arrival to pixels on screen. In Part 4, I want to go deeper into a couple of these stages specifically — likely MediaCodec internals and buffer/memory management under real-world constraints, since those are the areas with the least beginner-friendly documentation out there.

This is Part 3 of a multi-part series on media streaming internals. Catch Part 1 and Part 2 here.