Load-test your HLS key server: rotate keys, reproduce the stampede, then fix it

TL;DR

We'll build a tiny HLS setup with rotating AES-128 keys, point a concurrency harness at the key endpoint, and watch every simulated viewer ask for the same key in the same second. Then we'll apply the three mitigations that work on players shipping today, and look at where the new EXT-X-PRELOAD-HINT:TYPE=KEY tag fits.

Key rotation during a live stream is standard practice. What is not standard is load-testing the endpoint that serves those keys, and it is the one endpoint in your stack where every viewer makes an identical request at an identical moment. Let's make that visible.

You'll need ffmpeg 7.x or 8.x, node 20.x or newer, and openssl. Everything here runs locally.

1. 🎥 Make a stream with rotating keys

FFmpeg's HLS muxer will rotate keys for you if you hand it a key info file per period, but the clearest way to see the mechanic is to generate two keys and stitch a playlist that switches between them.

# generate two 16-byte content keys
openssl rand 16 > key0.bin
openssl rand 16 > key1.bin

# key info files: <key URI>\n<path to key file>\n<optional IV>
printf 'http://localhost:8080/keys/key0\nkey0.bin\n' > key0.info
printf 'http://localhost:8080/keys/key1\nkey1.bin\n' > key1.info

Now segment a source file with encryption on:

ffmpeg -i source.mp4 \
  -c:v libx264 -preset veryfast -g 48 -keyint_min 48 -sc_threshold 0 \
  -c:a aac -b:a 128k \
  -hls_time 4 -hls_playlist_type event \
  -hls_key_info_file key0.info \
  -hls_segment_filename 'seg_%03d.ts' \
  stream.m3u8

Open stream.m3u8 and you'll see the tag that does all the work:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:5
#EXT-X-KEY:METHOD=AES-128,URI="http://localhost:8080/keys/key0",IV=0x...
#EXTINF:4.000,
seg_000.ts
#EXTINF:4.000,
seg_001.ts

An EXT-X-KEY tag applies to every segment that follows it until another one shows up. To simulate a rotation, insert a second one partway down the playlist pointing at key1:

#EXTINF:4.000,
seg_014.ts
#EXT-X-KEY:METHOD=AES-128,URI="http://localhost:8080/keys/key1",IV=0x...
#EXTINF:4.000,
seg_015.ts

💡 Tip: in a real live setup you don't hand-edit this. Your packager emits a new EXT-X-KEY on whatever rotation interval you configured. The hand edit just lets us control exactly when the rotation lands.

2. 🛠️ Stand up a key endpoint that reports its own load

// key-server.js
import { createServer } from 'node:http';
import { readFileSync } from 'node:fs';

const keys = {
  '/keys/key0': readFileSync('key0.bin'),
  '/keys/key1': readFileSync('key1.bin'),
};

// count requests per 100ms bucket so we can see the shape of the load
const buckets = new Map();
setInterval(() => {
  const now = Date.now();
  for (const [t, n] of buckets) {
    if (now - t > 5000) buckets.delete(t);
  }
}, 1000);

const server = createServer((req, res) => {
  const key = keys[req.url];
  if (!key) {
    res.writeHead(404).end();
    return;
  }

  const bucket = Math.floor(Date.now() / 100) * 100;
  buckets.set(bucket, (buckets.get(bucket) ?? 0) + 1);

  // pretend this endpoint does an entitlement check
  setTimeout(() => {
    res.writeHead(200, { 'Content-Type': 'application/octet-stream' });
    res.end(key);
  }, 15);
});

server.listen(8080, () => console.log('key server on :8080'));

process.on('SIGINT', () => {
  const sorted = [...buckets.entries()].sort((a, b) => a[0] - b[0]);
  const peak = Math.max(...sorted.map(([, n]) => n));
  console.log(`\npeak requests in a 100ms window: ${peak}`);
  process.exit(0);
});

The 15ms setTimeout stands in for whatever your key endpoint actually does. Most do more than serve bytes: they check a session token, look up an entitlement, maybe hit a database. That work is the reason the endpoint has a concurrency ceiling at all.

3. Reproduce the stampede

Here is the part that matters. Real live viewers are not spread across the asset. They are all within a few seconds of the live edge, refreshing the same playlist, so they all see the new EXT-X-KEY line within the same playlist-refresh interval and all fetch it at once.

// stampede.js
const CONCURRENCY = Number(process.argv[2] ?? 2000);
const JITTER_MS = Number(process.argv[3] ?? 0);

const started = Date.now();
let ok = 0, failed = 0;
const latencies = [];

await Promise.all(
  Array.from({ length: CONCURRENCY }, async () => {
    if (JITTER_MS) await new Promise(r => setTimeout(r, Math.random() * JITTER_MS));
    const t0 = performance.now();
    try {
      const res = await fetch('http://localhost:8080/keys/key1');
      await res.arrayBuffer();
      latencies.push(performance.now() - t0);
      ok++;
    } catch {
      failed++;
    }
  })
);

latencies.sort((a, b) => a - b);
const p = (q) => latencies[Math.floor(latencies.length * q)]?.toFixed(1);
console.log(`concurrency=${CONCURRENCY} jitter=${JITTER_MS}ms`);
console.log(`ok=${ok} failed=${failed} wall=${Date.now() - started}ms`);
console.log(`p50=${p(0.5)}ms p95=${p(0.95)}ms p99=${p(0.99)}ms`);

Run it with no jitter, which is what an unmitigated rotation looks like:

$ node stampede.js 2000 0
concurrency=2000 jitter=0ms
ok=2000 failed=0 wall=1180ms
p50=612.4ms p95=1044.7ms p99=1131.2ms

Now run the same 2000 requests spread across a 10 second window, which is what a jittered fetch looks like:

$ node stampede.js 2000 10000
concurrency=2000 jitter=10000ms
ok=2000 failed=0 wall=10041ms
p50=17.9ms p95=21.3ms p99=28.6ms

⚠️ Your exact numbers will differ, and that is the point of running it on your own endpoint rather than trusting mine. The ratio is what to look at: identical total work, wildly different tail latency, purely because of arrival distribution.

Turn the concurrency up until you see failures. That number, divided by your rotation interval, is not your capacity. Your capacity is that number in a single burst, because that is how the requests actually arrive.

4. What breaks when the key is late

This is worse than a late segment, and it is worth being precise about why.

Failure	Player behaviour	Recovery
Segment fetch times out	Buffer drains, player stalls, retries	Usually automatic
Manifest fetch times out	Player keeps playing buffered content	Usually automatic
Key fetch times out	Player cannot decrypt the next segment	Often a fatal error, client-dependent

hls.js emits a KEY_LOAD_ERROR under NETWORK_ERROR and will retry, but the retry budget is finite and the segments keep arriving that it cannot decrypt. Native players vary. On some devices the user has to hit play again, which for a live event means you have converted a 200ms latency spike into a measurable drop in concurrent viewers.

5. The three fixes that work today

Jitter the fetch on the client, if you control the client. If you are shipping hls.js you can hook key loading and add a random delay before the request. This is exactly what the spec-level fix does, just done by hand:

// only viable when you control the player build
class JitteredKeyLoader extends Hls.DefaultConfig.loader {
  load(context, config, callbacks) {
    const delay = context.url.includes('/keys/') ? Math.random() * 3000 : 0;
    setTimeout(() => super.load(context, config, callbacks), delay);
  }
}

const hls = new Hls({ loader: JitteredKeyLoader });

The catch is obvious: it only helps viewers on your player build. Anyone on a native player is unaffected.

Serve keys through your CDN. A content key is a small immutable object at a stable URI, which is the ideal cacheable object. Set a Cache-Control that covers the key's active lifetime and your origin sees a handful of requests per rotation instead of all of them. The tension is that a lot of teams do per-viewer entitlement checks at key fetch time, which makes the response non-cacheable by design. If that is you, the question worth asking is whether the key fetch is really where authorisation should happen, or whether it belongs at session establishment.

Lengthen the rotation interval, or at least justify it. Rotating more often does not reduce the size of each spike, it increases the number of spikes. Every rotation is a full-audience event. Write down what threat the current interval mitigates, then check whether the interval is actually load-bearing for that threat or was copied from a config example.

6. Where EXT-X-PRELOAD-HINT:TYPE=KEY fits

The WWDC 2026 HLS update (What's new in HTTP Live Streaming, June 2026) added TYPE=KEY to EXT-X-PRELOAD-HINT. Previously that tag took TYPE=PART and TYPE=MAP and was a Low-Latency HLS mechanism for hinting at resources that do not exist yet.

The new value hints at an upcoming decryption key. Apple's stated rationale names the problem directly: a key rotation during a high-concurrency live stream triggers a thundering herd of simultaneous key requests. Clients that support the hint preload the key at a randomly-selected point between first seeing the hint and the key's first use, spreading the requests across that window.

Which is to say: the fix is standardised jitter, moved from your player fork into the protocol.

⚠️ Do not treat this as shipped. It is a spec addition; client support arrives on each player's own schedule. hls.js is at 1.6.16 with 1.7.0 in development, and preload-hint handling has been an open thread in that project for a while. Smart TV and set-top players update on manufacturer timelines. Emit the hint if your packager supports it, but keep the CDN and interval work, because those help every client including the ones that will never be updated.

The spec details are in section 4.4.5.3 of the preliminary HLS draft, and the stable reference for everything else is draft-pantos-hls-rfc8216bis.

What's next

• Run stampede.js against your actual key endpoint in staging, at your real peak concurrency, in one burst. Not a sustained rate, a burst. That single number is more useful than the rest of this post.
• Check whether your key responses are cacheable. If they are not, find out why, and whether that reason still holds.
• Audit the rest of your stack for the same shape. Anything every viewer requests at the same instant belongs on a list: the key server, the DRM license server, the ad-decision call at the first break, the auth refresh if it is aligned to a fixed TTL.

The pattern generalises well beyond video, which is why "thundering herd" was already a name for it. Live streaming is just where you get to watch it happen in front of an audience.