A deep dive into the March 25th voice outage and the E2EE rollout that followed — two stories that together define Discord's most technically eventful year yet.
If you were trying to hop on a Discord voice call on the afternoon of March 25, 2026, you probably stared at that maddening "Awaiting Endpoint" spinner for a long time. Maybe you refreshed. Maybe you rage-quit. Maybe you blamed your ISP.
It wasn't your ISP.
What actually happened behind the scenes is one of the most fascinating distributed systems failure stories of the year — a cascade that started with a routine Kubernetes config tweak and ended with Discord engineers playing an urgent, multi-hour game of infrastructure whack-a-mole. And just weeks after that chaos, the same engineering org quietly shipped one of the most significant privacy features in the company's history: mandatory end-to-end encryption for every voice and video call on the platform.
Let's dig into both.
Part 1: The Outage — "You've Got (Too Much) Mail"
March 25, 2026. 12:13 PM PDT.
Discord's engineering team titled their postmortem with a dad joke: "You've Got (Too Much) Mail." It's funnier once you understand the punchline — and the punchline is a single Erlang supervisor process drowning in a million-message mailbox queue.
The Routine Change That Wasn't
Discord's Realtime Infrastructure team had been in the middle of a Kubernetes migration for their Elixir services. They'd been running stateful Elixir processes on VMs, and they were gradually moving these over to K8s pods. Standard stuff for a platform at scale.
On March 25th, they deployed a PR to adjust pod resources for their session management service — the service that manages a "session" for every device you connect with. If you've got Discord open on your phone, laptop, and smart TV simultaneously, that's three sessions. Sessions are basically the heartbeat of Discord's real-time infrastructure; they route every message, every presence update, every notification to the right client.
The plan was simple: vertically scale the pods (more CPU, more memory per pod) while reducing the pod count proportionally, and see if CPU utilization dropped. A reasonable experiment.
What happened instead: as Kubernetes applied the new config and terminated 50% of pods in one availability zone (us-east1-b), a safety check designed to wait for in-progress events before handoff ran longer than Kubernetes's termination grace period. Kubernetes assumed the pods were dead and killed them. Since us-east1-b hosts one of Discord's three zones of session servers, this ungracefully destroyed 17% of all active Discord sessions globally.
Seventeen percent sounds manageable until you do the math. Discord has hundreds of millions of users. Seventeen percent of sessions going dark simultaneously is an enormous shock.
The Thundering Herd Begins
Every GenServer process in Discord's Elixir systems uses process monitors (Process.monitor). When a session dies, every entity monitoring that session — guilds, presence trackers, gateway connections — gets a {:DOWN, ...} message in its mailbox. Seventeen percent of sessions dropping at once sent a tidal wave of these messages throughout the system.
The first thing to crack: the gateway service. Your gateway connection is how your client maintains the websocket to Discord. When sessions died, gateways immediately told clients to reconnect. Millions of clients began simultaneously reconnecting to us-east1-b. Memory on those gateway nodes spiked. OOM kills started. The zone went down harder.
Now users who weren't originally affected began reconnecting too, as their gateway connections were killed by the cascading OOM events. They all failed over to us-east1-c and us-east1-d. Those zones absorbed the surge, and most users eventually got their text-based experience back. Voice, however, was a different story.
Meet the Holster Pool: The Real Villain
Here's where it gets genuinely fascinating.
Discord's voice syncers service is responsible for routing voice and video calls. It constantly evaluates active voice states across every guild and every DM call, and it sends RPC commands to a fleet of 25,000+ Selective Forwarding Unit (SFU) instances scattered globally. The SFUs are what actually relay your encrypted media between call participants.
When sessions disconnect from voice calls, the voice syncer sends an RPC to the SFU to handle the disconnection. When sessions reconnect, it sends RPCs to recreate calls. With 17% of sessions dropping and then reconnecting, voice syncers suddenly had to fire an enormous flood of outgoing HTTPS connections to SFUs.
For those HTTPS connections, Discord uses an internal library called Holster, which pools connections through gun — an Erlang HTTP client. And here's the architectural landmine that nobody had tripped before:
Every outgoing HTTPS connection from voice syncers goes through two Erlang supervisor processes: one for the
Holster.Poolitself, and one for allgunconnections.
Each supervisor is a single GenServer process. When it spawns a child, it performs a selective receive — scanning its mailbox for a specific ACK from the child. This is where things go sideways in a subtle but devastating way.
Selective receive in Erlang has a performance characteristic that's not obvious until you're on call at 12:30 PM watching your services fall over: as the mailbox grows, selective receive gets slower. Discord's postmortem confirmed this empirically — a supervisor with a ~100k message mailbox adds ~1ms to each spawn. With a million-message mailbox and 100 spawn requests per second, catching up becomes mathematically impossible. The supervisor is a bottleneck that compounds under load.
So the supervisor mailboxes started growing. New connection requests began timing out. But critically — and this is the knife twist — the etcd connection that powers Discord's Elixir service discovery also goes through these same bottlenecking supervisors.
Discord's voice syncers announce their health to etcd periodically. If an instance doesn't refresh within its 60-second TTL, it drops out of the consistent hash ring that routes voice syncer traffic. With the supervisors jammed, etcd heartbeats couldn't get through. One by one, 14 of 15 voice syncer instances dropped off the hash ring.
By 12:13 PM, the aggregate process mailbox lengths across all voice syncers were growing, and RPC calls to SFUs had plummeted to near zero.
The Recovery Attempts (This Is the Whack-a-Mole Part)
12:43 PM — Engineers fully restarted the voice syncers application on one instance. It briefly recovered, re-announced in etcd, started routing calls... and then immediately got overwhelmed by the pent-up traffic from being the only healthy instance in the ring. Mailbox started growing again.
12:47 PM — They killed and restarted just the Holster.Pool DynamicSupervisor on another instance. Same pattern: brief recovery, immediate re-drowning.
13:05 PM — Full cluster restart. This is the nuclear option — "restarts the Discord" from A/V infrastructure's perspective, recreating millions of active calls simultaneously. Some calls recovered. By 13:09, all restarted nodes had ever-growing mailboxes again. The thundering herd of reconnecting calls was simply too much.
This is a genuinely hard problem: every restart attempt creates a fresh thundering herd because all the pent-up reconnection requests hit the newly-healthy instances at once.
13:43 PM — The team deployed aggressive rate limits on syncer creation across guilds, calls, and streams services. Another cluster restart. This time, things looked slightly better — the bottleneck shifted from Holster.Pool to the gun supervisor specifically, which meant instances could at least check out pooled connections and send some RPCs. All instances stayed announced in etcd.
Meanwhile, engineers were provisioning 15 additional voice syncer instances — a slow, manual process because these live on GCP VMs defined in Terraform, configured by Salt. Not a K8s deployment you can scale with one command.
14:03 PM — Targeted restart of instance 2-3. Success. It took on its voice syncers, mailbox lengths stayed low, connections worked. The combination of lower rate limits, reduced global syncer count (fewer users actively trying to connect), and a healthier cluster finally gave individual instances enough breathing room.
14:15 PM — Five instances restarted, four fully healthy.
14:26 PM — The 15 new instances came online simultaneously, cutting per-instance syncer load in half. The final backed-up instance was restarted. Full recovery.
Duration: ~3 hours and 17 minutes.
What Discord Fixed
Discord's postmortem action items are exactly what you'd hope to see from a well-run engineering org:
Kubernetes admission webhook — A validating webhook now rejects scale-down operations for Elixir workloads until pods have fully drained their processes. The initial cause of the outage, guarded at the API level.
PartitionSupervisorfor Holster.Pool — Instead of a singleDynamicSupervisorthat becomes a bottleneck, they've replaced it with aPartitionSupervisor, which distributes load across multiple independent supervisors. Concurrent connection creation is now possible.Gun connection lifecycle moved to Holster.Pool — Eliminates the second bottlenecking supervisor entirely.
Tuned rate limits with load shedding tooling — Rate limits that weren't re-tuned after the Kubernetes migration are now properly configured and enforceable.
Expanded observability — Monitoring on HTTP connection pool mailboxes, service discovery announcements, and voice syncer → SFU RPC traffic. You can't fix what you can't see.
Long-term: multi-region A/V control plane — A larger re-architecture to horizontally scale control plane services across regions, improving fault isolation and geographic distribution of connection spikes.
The "Guided by Constraints" section of their postmortem is worth bookmarking separately. The framing — every capacity constraint is an economics problem where you either increase supply or reduce demand — is the kind of clear mental model that makes incident response faster and more systematic.
Part 2: The Privacy Win — E2EE for Everyone
Here's the context switch. Three weeks before the March 25th outage, Discord had quietly crossed a different milestone.
Years in the Making
Discord's journey to end-to-end encryption started in earnest in August 2023 with a deliberately understated blog post. In September 2024, engineer Stephen Birarda introduced the DAVE protocol — Discord's open, Trail of Bits-audited E2EE protocol for audio and video. DAVE uses Messaging Layer Security (MLS) for key exchanges and WebRTC encoded transforms for media encryption.
Through 2025, engineer Clément Brisset extended DAVE to every remaining platform: web browsers, gaming consoles (PlayStation, Xbox), Discord bots, and the Social SDK. This was harder than it sounds — browsers don't implement WebRTC encoded transforms consistently. Discord actually worked with Mozilla to fix a Firefox compatibility issue rather than implementing a workaround or dropping browser support. That's the kind of detail that tells you this team takes the work seriously.
March 2, 2026: All Discord clients were required to support DAVE before joining any call. The unencrypted fallback was switched off.
May 19, 2026: Discord formally announced to the world that E2EE was now standard for every voice and video call — DMs, group DMs, voice channels, Go Live streams. No opt-in required. Stage channels remain the only exception (they're designed for large public broadcasts, not private conversations).
Discord is now in the process of removing the legacy code that supported unencrypted connections. Once that's done, falling back to unencrypted is architecturally impossible, not just disabled.
Why This Matters for the Dev Community
The timing is interesting. This announcement came shortly after Meta pulled E2EE from Instagram messaging, and TikTok signaled it wouldn't encrypt user messages after becoming a US company. Discord is swimming against that current, for hundreds of millions of users, including a massive developer and gaming community that actually cares about this stuff.
The technical story is also genuinely impressive. Running E2EE at Discord's scale — with the latency requirements of real-time voice — while supporting platforms as constrained as game consoles is non-trivial. DAVE is an open protocol with a public audit. Discord has an active bug bounty. This isn't privacy theater.
What's not encrypted: text messages. Discord was direct about this — text features were built from the ground up on unencrypted assumptions, and rebuilding them would be a massive engineering effort. No current plans to extend E2EE to text.
What These Two Stories Have in Common
On the surface, a catastrophic voice outage and a privacy milestone seem unrelated. But they tell the same story: Discord is running some of the most technically complex real-time infrastructure on the internet, and they're being honest about it.
The postmortem for the March 25th outage is exceptional. It names specific Erlang behaviors, walks through the failure cascade with timestamps, explains why each recovery attempt failed, and lists concrete architectural changes with rationale. Most companies write vague outage summaries. Discord wrote a distributed systems tutorial.
The E2EE announcement is similar — they published DAVE as an open protocol, got it audited, disclosed the Firefox bug they had to fix, and were explicit about the limits (no text E2EE, stage channels excluded).
For developers, both stories are worth reading carefully — not just as Discord news, but as case studies in how to build, fail, recover, and communicate at scale.
TL;DR for the Dev.to Crowd
What broke: A Kubernetes config tweak killed 17% of Discord's session servers. The thundering herd of reconnections overwhelmed an Erlang supervisor bottleneck in the voice syncer's HTTP connection pool, which also happened to be the path for etcd health checks, causing 14/15 voice syncer instances to drop off the hash ring.
How it was fixed: Rate limiting + doubling voice syncer cluster capacity + targeted restarts, after 3+ hours.
What changed: K8s admission webhook for graceful drain,
PartitionSupervisorreplacingDynamicSupervisor, gun supervisor eliminated, better observability.The other thing: E2EE via the DAVE protocol is now mandatory for all voice/video calls on Discord. Open protocol, externally audited, no opt-in needed.
If your system has never had a supervisor mailbox grow to 1 million messages and take down your service discovery, consider yourself lucky — and go read Discord's postmortem anyway.
Citations
Discord Engineering — "You've Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage" (April 29, 2026) — discord.com/blog
postmortem.io — Discord 2026-03-25 Voice Outage Archive — postmortem.io
Discord Engineering — "Every Voice and Video Call on Discord Is Now End-to-End Encrypted" (May 19, 2026) — discord.com/blog
BleepingComputer — "Discord rolls out end-to-end encryption on voice, video calls" (May 19, 2026) — bleepingcomputer.com
TechCrunch — "Discord enables end-to-end encrypted voice and video calling for every user" (May 19, 2026) — techcrunch.com
Discord Support — "End-to-End Encryption for Audio and Video" — support.discord.com
LavX News — "Discord Dissects a Hidden Circular Dependency Behind Its March Voice Outage" — news.lavx.hu
Lesty.tech — "Discord's 5 Hidden Circular Dependency Outage Lessons" — lesty.tech
LinkedIn / AB Satyaprakash — "Discord published an incredible behind-the-scenes post-mortem" (May 10, 2026) — linkedin.com
YouTube — "Discord's Global Incident Report" (summary video) — youtube.com
Top comments (0)