DEV Community

Mason K
Mason K

Posted on

Audit your HLS audio ladder: what you're actually shipping, and whether xHE-AAC is reachable

#ffmpeg #video #tutorial #webdev

TL;DR

We'll probe a real HLS stream to find out what audio it actually delivers, build a two-rung audio rendition set with correct EXT-X-MEDIA and CODECS declarations, and work out whether xHE-AAC is reachable for your stack. Spoiler on that last one: decoding is universal, encoding needs a license.

Your encoding profile has been re-tuned four times on the video side. The audio side is one 128 kbps AAC-LC track that nobody has touched since setup. Let's find out what that's costing and what the options are. You'll need ffmpeg 7.x or 8.x and, optionally, bento4.

1. 🔎 Find out what you're actually shipping

Start with the master playlist, not your encoding config. These disagree more often than you'd think.

curl -s https://example.com/stream/master.m3u8
Enter fullscreen mode Exit fullscreen mode 
#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=628000,CODECS="avc1.4d401e,mp4a.40.2",RESOLUTION=640x360
360p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1628000,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3128000,CODECS="avc1.640028,mp4a.40.2",RESOLUTION=1920x1080
1080p.m3u8
Enter fullscreen mode Exit fullscreen mode

Two things to notice. First, mp4a.40.2 on every line: that's AAC-LC, the same audio in all three variants. Second, there's no EXT-X-MEDIA block, so audio is muxed into each video rendition and there is no separate audio rendition to switch.

Now confirm what the bytes say, because manifests lie:

ffprobe -v error -select_streams a:0 \
  -show_entries stream=codec_name,profile,bit_rate,channels,sample_rate \
  -of json https://example.com/stream/360p.m3u8
Enter fullscreen mode Exit fullscreen mode 
{
  "streams": [
    {
      "codec_name": "aac",
      "profile": "LC",
      "sample_rate": "48000",
      "channels": 2,
      "bit_rate": "128000"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Here's the number that should bother you. On the 360p rendition, total bandwidth is 628 kbps and audio is 128 of it. Roughly a fifth of the bytes going to your worst-connected viewers are audio, encoded at exactly the same quality as the audio going to the viewer on fibre. That is the whole problem in one line.

💡 Tip: run this against a live playback URL rather than a mezzanine file. What your pipeline intends to produce and what your packager emits are separate facts.

2. Split audio out so it can vary

Muxed audio means audio is welded to the video rendition. Demuxed audio, declared with EXT-X-MEDIA, means the player picks an audio rendition independently. That's the prerequisite for everything else.

# encode video renditions with no audio
ffmpeg -i source.mp4 -an -c:v libx264 -preset veryfast \
  -g 48 -keyint_min 48 -sc_threshold 0 \
  -b:v 3000k -s 1920x1080 v_1080.mp4

# encode audio renditions separately
ffmpeg -i source.mp4 -vn -c:a aac -b:a 128k -ac 2 a_hi.mp4
ffmpeg -i source.mp4 -vn -c:a aac -b:a 64k  -ac 2 a_lo.mp4
Enter fullscreen mode Exit fullscreen mode

Then package to fMP4 HLS. Bento4's mp4hls handles the manifest authoring:

mp4hls --output-dir=out --hls-version=7 \
  v_1080.mp4 v_720.mp4 v_360.mp4 a_hi.mp4 a_lo.mp4
Enter fullscreen mode Exit fullscreen mode

The resulting master playlist declares audio as a separate group:

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-hi",NAME="English",DEFAULT=YES,\
AUTOSELECT=YES,LANGUAGE="en",URI="audio/hi/stream.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-lo",NAME="English",DEFAULT=YES,\
AUTOSELECT=YES,LANGUAGE="en",URI="audio/lo/stream.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=3128000,CODECS="avc1.640028,mp4a.40.2",\
RESOLUTION=1920x1080,AUDIO="audio-hi"
video/1080/stream.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=464000,CODECS="avc1.4d401e,mp4a.40.2",\
RESOLUTION=640x360,AUDIO="audio-lo"
video/360/stream.m3u8
Enter fullscreen mode Exit fullscreen mode

Now the 360p variant pairs with 64 kbps audio, and its total bandwidth drops from 628 to 464 kbps. That's a real saving on the rung where bandwidth is scarcest, and it required no new codec.

⚠️ Common mistake: pointing every EXT-X-STREAM-INF at the same AUDIO group and then wondering why audio never changes. The group ID is the switch. One group per audio quality tier.

3. Verify the declaration matches the bytes

The CODECS attribute is a promise to the player. If it's wrong, some clients refuse to play before they ever fetch a segment.

Codec CODECS string Notes
AAC-LC mp4a.40.2 The default everywhere
HE-AAC v1 mp4a.40.5 SBR
HE-AAC v2 mp4a.40.29 SBR + PS
xHE-AAC mp4a.40.42 fMP4 only, common encryption only

Check yours against the actual streams:

for f in out/audio/*/stream.m3u8; do
  echo "== $f"
  ffprobe -v error -select_streams a:0 \
    -show_entries stream=codec_name,profile,bit_rate -of csv=p=0 "$f"
done
Enter fullscreen mode Exit fullscreen mode 
== out/audio/hi/stream.m3u8
aac,LC,128000
== out/audio/lo/stream.m3u8
aac,LC,64000
Enter fullscreen mode Exit fullscreen mode

Apple's HLS tools (mediastreamvalidator) will catch mismatches too, and as of the 2026 release they run on macOS, RHEL 9.5, Ubuntu 24.04.3 LTS and Debian 13.2 with full feature parity, so this can live in CI on Linux now.

4. 🎧 So where does xHE-AAC come in

xHE-AAC is MPEG-D USAC (the Extended HE-AAC profile) plus MPEG-D loudness and dynamic range control. Two properties make it interesting for the low rung of your ladder:

  1. Range. Fraunhofer specifies stereo operation from 12 kbps to over 320 kbps in one codec. Apple's HLS authoring guidance puts the practical stereo band at 24 kbps up to a recommended 160.
  2. Mandatory loudness and DRC metadata in every bitstream. The playback device adapts level to its own output. That's a normalisation stage you stop owning.

Playback support is not the blocker. Native decode has shipped since Android 9 Pie (2018) and iOS 13 / macOS 10.15 / tvOS 13, Safari decodes it through AVFoundation, and Fraunhofer puts the Android installed base above seven billion devices. Android 17 makes xHE-AAC encoding a standard feature this year, which is new.

Constraints to know before you plan around it:

  • fMP4 only. No MPEG-2 TS carriage.
  • Common encryption only. If you're on full-segment AES-128, that's a packaging change.
  • AVPlayer does mono and stereo only. No multichannel, so a 5.1 ladder stays as-is.

5. The encoding catch

Here's the part that decides this for most teams. Stock FFmpeg cannot encode xHE-AAC.

$ ffmpeg -h encoder=libfdk_aac 2>&1 | grep -A6 'profile'
     -profile           <int>  E...A...... (from 0 to 5)
       aac_low          2      LC
       aac_he           5      HE-AAC
       aac_he_v2        29     HE-AACv2
       aac_ld           23     Low Delay
       aac_eld          39     Enhanced Low Delay
Enter fullscreen mode Exit fullscreen mode

No USAC, no Extended HE-AAC. libfdk_aac is the open-source Fraunhofer decoder/encoder kit and it stops at HE-AAC v2. Encoding xHE-AAC requires a licensed implementation: Fraunhofer's own, or the MainConcept xHE-AAC encoder plugin for FFmpeg. Packaging is fine either way, since Bento4 handles xHE-AAC in DASH and HLS without special options.

So the honest summary:

xHE-AAC Cost
Decode Universal, already deployed Free
Encode Licensed implementations only Purchase order
Package Bento4, no special flags Free

If you're at a scale where a codec licensing conversation is routine, this is a straightforward win on your low rung. If you're a small team with an FFmpeg box, it isn't reachable this quarter, and the right move is section 6.

6. What to do instead, today

Add HE-AAC v1 to the bottom rung. If your FFmpeg build has libfdk_aac (most distro builds don't, because of the license, so check), you get a codec that holds together at 48 kbps:

ffmpeg -i source.mp4 -vn -c:a libfdk_aac -profile:a aac_he \
  -b:a 48k -ac 2 a_lo.mp4
Enter fullscreen mode Exit fullscreen mode

Declare it as mp4a.40.5. Test it on your actual device mix before shipping, because HE-AAC decode is very widely supported but not quite as universally as AAC-LC.

Do the loudness pass yourself, correctly. Since you're not getting DRC metadata for free, two-pass loudnorm is the substitute:

# pass 1: measure
ffmpeg -i source.mp4 -af loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json \
  -f null - 2>&1 | tail -12
Enter fullscreen mode Exit fullscreen mode 
{
  "input_i" : "-23.41",
  "input_tp" : "-4.20",
  "input_lra" : "8.10",
  "input_thresh" : "-33.87",
  "target_offset" : "0.13"
}
Enter fullscreen mode Exit fullscreen mode 
# pass 2: apply the measured values
ffmpeg -i source.mp4 -vn -af \
  loudnorm=I=-16:TP=-1.5:LRA=11:measured_I=-23.41:measured_TP=-4.20:\
measured_LRA=8.10:measured_thresh=-33.87:offset=0.13:linear=true \
  -c:a aac -b:a 128k a_hi.mp4
Enter fullscreen mode Exit fullscreen mode

Single-pass loudnorm is a dynamic normaliser and will pump on content with wide dynamic range. Two-pass measures first, then applies a fixed gain. Use two-pass.

7. "Why not just use Opus?"

This comes up every time, so let's settle it. Opus is free, it is excellent at low bitrates, and ffmpeg -c:a libopus works in every build you have. On paper it solves the same problem.

The blocker is carriage and client support in the HLS world specifically. Opus is at home in WebM and in WebRTC, where it is effectively the default. In HLS the story is much thinner: Apple's client stack does not decode Opus in HLS, so shipping an Opus-only audio rendition means shipping a stream that does not play on iPhones, iPads, Apple TV or Safari. For most consumer catalogues that ends the conversation before it starts.

Where it does make sense:

Context Opus xHE-AAC AAC-LC
WebRTC / real-time Yes, the default No Rare
DASH to browsers (no Safari requirement) Works well Works Works
HLS to Apple devices Not supported Supported Supported
Cheap and universal Free encoder Licensed encoder Free encoder

So if you ship DASH to a controlled device set, or you are doing real-time, Opus is the pragmatic answer and you can stop reading. If HLS to consumer devices is your delivery path, the choice is between AAC-LC, HE-AAC and xHE-AAC, and the licensing question in section 5 is unavoidable.

# for the DASH-only case, this is all it takes
ffmpeg -i source.mp4 -vn -c:a libopus -b:a 48k -vbr on -application audio a_lo_opus.mp4
Enter fullscreen mode Exit fullscreen mode

⚠️ Test playback on the real device matrix before committing to any of this, not just on your laptop. Audio codec support is the area where "it works in Chrome" is the least informative test result available.

What's next

  • Run the section 1 probe against production right now. If audio is muxed and identical across every rung, you have a bandwidth win available that costs you nothing but a repackage.
  • Decide the bottom rung deliberately: 64k AAC-LC, 48k HE-AAC, or a licensing conversation about xHE-AAC. Any of the three beats "128k everywhere by default".
  • If you're already on fMP4 and common encryption, you've met the xHE-AAC packaging prerequisites, so the only open question is the encoder.

Worth reading next: Apple's HLS authoring specification for the audio rendition rules, and Bento4's multi-bitrate audio guide for the packaging side.

Top comments (0)