DEV Community

Cover image for Actors in Production: How Discord, LinkedIn and Halo Use the Actor Model
Hamed Yousefi
Hamed Yousefi

Posted on Originally published at hmdsefi.hashnode.dev AI-assisted

Actors in Production: How Discord, LinkedIn and Halo Use the Actor Model

#distributedsystems #softwareengineering #elixir #scalability

The actor model is easy to describe: small units that own their state, handle one message at a time, and talk only by messages. If you haven't met it yet, my previous article covers where it came from and how it works. This one is about what happens when real systems with millions of users are built that way.

I picked three systems that published detailed engineering write-ups, each on a different runtime:

  • Discord runs chat on Elixir, on the Erlang virtual machine (the BEAM).
  • LinkedIn built instant messaging and its "who's online" dots on Akka, on the JVM.
  • Halo's cloud services ran on Microsoft Orleans, on .NET.

For each one: what they needed, how actors fit, what broke, and the result. Every number below comes from the companies' own posts and papers, with the date they gave it.

Discord: one process per server, one per connection

What they needed

Discord is chat, voice and video organized into communities that Discord's code calls guilds (users call them servers). The company's 2017 engineering post describes most of Discord as publish/subscribe: when anything happens in a guild, every member who's online must see it, right away.

That gets expensive fast. A message in a guild with N people online goes to N sessions, and if all N people are active, the total work grows with N². Discord's 2023 post on very large servers works an example: with 100,000 people online, that's 10 billion notifications to deliver.

How actors fit

Discord built its first prototype in Elixir, and its 2017 post explains why: "The Erlang VM was the perfect candidate for the highly concurrent, real-time system". The design maps straight onto actors:

  • Every connected user gets a session process (a GenServer, Elixir's standard actor) that owns their WebSocket.
  • Every guild gets one guild process, on a separate set of nodes, that owns the guild's state, including which members' sessions are connected.
  • When something happens, the guild process fans it out by sending a message to the session process of every member online.

Discord's design. A guild process on a guild node keeps track of its members' sessions. A member's session process forwards a message to the guild process (step 1). The guild process handles it one message at a time (2), then Manifold sends one message per session node instead of one per member (3), and each node hands it to its local session processes, which push it to clients over WebSockets (4).
Follow the numbers: the guild process is the single place where a guild's state lives, and step 3 is the optimization that made it scale.

Discord's later posts, from 2023 and from 2026, still describe this shape: one process per guild as the central routing point, fanning out to sessions by message passing.

What broke

A guild process is an actor, so it handles one message at a time. That is what keeps its state consistent without locks. It's also its weak point. The 2017 post, written when Discord had nearly five million concurrent users, describes three failures, and each one is a lesson about actors:

  1. Mailboxes fell behind. Big guilds then had up to 30,000 people online. At peak, guild processes couldn't keep up with their message queues, and engineers had to turn off features by hand. A single Erlang send took 30 to 70 microseconds, so publishing one event from a large guild could take 0.9 to 2.1 seconds. The fix, an open-source library called Manifold, groups recipients by node and sends one message per remote node; a process on that node does the local sends. The work is spread out, and the guild process moves on sooner. Manifold's README reports that packets per second between nodes dropped by half right after it shipped.

  2. One process held data everyone needed. The hash ring that maps guilds to nodes was controlled by a single process, and it got so busy that it fell behind. Discord estimated that one session server restart cost about 30 seconds of ring lookups. Moving the ring into fast shared read-only storage (a library called FastGlobal) cut each lookup to 0.3 microseconds.

  3. A stampede. Nearly five million session processes all tried to call the same ten registry processes. The session processes' mailboxes ballooned until the virtual machine ran out of memory. The fix was a Semaphore library that caps how many callers can hit a busy process at once.

Notice what these fixes have in common: none of them abandoned actors. Each one controls how many messages reach a single actor. That paid off later in the same post: when Discord's presence services crashed for an unrelated reason, the session services carried on unaffected, and Discord credits the Semaphore library.

The work didn't stop in 2017. In 2019, Discord wrote a Rust data structure, called from Elixir, to keep member lists sorted in very large guilds. In 2023 it added relay processes between guilds and sessions, each handling up to 15,000 sessions, and cut the work for the roughly 90% of connections in large servers that it calls passive.

The result

  • July 2017: nearly 5 million concurrent users and millions of events per second.
  • May 2019: the Rust post is titled "Using Rust to Scale Elixir for 11 Million Concurrent Users".
  • October 2020: an Elixir case study, written by Elixir's creator José Valim with quotes from Discord engineers, reports more than 12 million concurrent users, more than 26 million WebSocket events per second, and 400–500 Elixir machines, run by a chat infrastructure team of five engineers.
  • October 2023: the largest single server, Midjourney, had over 10 million members, with over 1 million online.

The lesson: one actor per guild gives you a simple, lock-free owner for each guild's state. The price is that a hot actor's mailbox becomes your bottleneck, so the engineering goes into controlling what reaches it.

LinkedIn: one actor per connection, then one per online member

What they needed

In 2016 LinkedIn launched instant messaging with typing indicators and read receipts. Those need the server to push data to phones and browsers, not wait to be asked. LinkedIn's engineers chose Server-Sent Events over long-lived HTTP connections, and set a goal of holding hundreds of thousands of them on each machine.

Then, in 2018, they reused the same platform for presence: the green dot that shows a contact is online. Their post describes the load as roughly 500 million members, millions of them online at any moment, and thousands going online or offline every second.

How actors fit

LinkedIn's servers used the Play framework's integration with Akka, on the JVM.

For messaging, the 2016 post's design is: assign an Akka actor to each connection the server accepts. The actor owns that connection's lifecycle, and the only way for the rest of the system to reach the client is to send the actor a message.

For presence, they needed a cheap timer per online member, because "offline" is really the absence of a heartbeat. Their answer is a neat use of actors:

LinkedIn's presence design. While bob's app is connected, a heartbeat arrives every d seconds as a message in bob's actor's mailbox (1). With no live entry in a distributed key-value store, the platform publishes "online", and each heartbeat refreshes an entry that expires soon after (2). The actor schedules a Publish Offline message to itself, a little later, and each heartbeat resets it (3). When it fires and the entry has expired, it publishes "offline" on bob's presence topic, which reaches alice (4).
The amber loop is the whole trick: an actor sending a message to its own future self is a timer, and a heartbeat resets it.

  1. There is one actor per online member, which acts as that member's delayed trigger.
  2. While the member is connected, a heartbeat arrives every few seconds, and the actor treats it as a message in its mailbox.
  3. The actor uses Akka's scheduler to send itself a "Publish Offline" message, timed a little after the next heartbeat is due. Each heartbeat pushes that timer back.
  4. If the timer fires and the member's entry in a distributed key-value store has expired, the platform publishes an "offline" event on the member's presence topic. Everyone subscribed to that topic, like alice in the diagram, sees the dot change.

Two details from the post are worth copying. First, the shared state lives in the key-value store, not in the actors; each actor holds only what one member needs. Second, LinkedIn notes that the number of live actors equals the number of members online, and that a small number of threads, proportional to the number of cores, runs them all. Actors don't each need a thread.

The result

  • October 2016: LinkedIn reported holding 100,000 concurrent persistent connections on each node in production. Getting there was mostly operating-system and JVM tuning: they first ran out of memory at about 90,000 connections with an 8 GB heap, found each TLS connection cost about 20 KB, and moved to a 16 GB heap. CPU stayed well under 80%: the limits were memory and operating-system settings, not the actors.
  • January 2018: presence events reached viewers in under 200 milliseconds at the 99th percentile, and each presence node handled about 1,800 requests per second, scaling out by adding nodes.

The lesson: an actor doesn't have to be big to be useful. LinkedIn's presence actor is little more than a timer with a mailbox, and that's exactly why millions of them fit on a few machines.

Halo: virtual actors for a game launch

What they needed

Halo 4 launched on 6 November 2012. Behind the game sat cloud services for player statistics, presence (who is playing, in which game), daily challenges, playlists and cheat detection. Caitie McCaffrey, an engineer at 343 Industries, described the launch in a 2015 talk: Halo 4 went from zero to 1 million users on day one and 4 million in the first week, and had 11.6 million players by June 2013.

Game traffic is brutal in a specific way. It arrives almost all at once at launch, every player and every match is its own little bundle of state, and the services must answer with low latency while staying up.

How actors fit

All of Halo 4's cloud services ran on Orleans, the framework from Microsoft Research that the previous article introduced. Orleans changes one thing about classic actors: you never create or destroy them. Each actor, called a grain, is identified by an ID, and it always exists virtually. When code calls a grain that isn't in memory, Orleans activates it on one of the servers in the cluster (called silos); when a grain sits idle for a while, Orleans removes it from memory. Each activation runs one request at a time on a single thread, so its state needs no locks.

The 2014 Orleans paper describes Halo 4's Presence service in detail:

Halo 4 on Orleans. Consoles send heartbeats to a stateless Router actor, of which Orleans runs as many copies as needed (1). The router calls the Game Session actor by its ID, with no code to find or create it (2). For statistics, the game session sends each player's part to that player's Player actor (3). Calling an ID that is not in memory makes Orleans activate it on some silo, and idle ones are removed (4).
The silos are just where actors happen to be. Nothing in the code says "silo 2": a call by ID finds session 42 wherever it lives, or starts it.

  1. Every Halo 4 console sends regular heartbeats to Presence. A stateless Router actor takes them. Because it holds no state, Orleans can run several activations of it to spread the load.
  2. The router forwards each heartbeat to the Game Session actor for that session ID; there's one per session. Its state lives only in memory and is never written to storage, because, as the paper explains, the consoles always hold the real state of the game. If the actor is lost, the next heartbeats rebuild it.
  3. Every so often, not on every heartbeat, the game session calls the Player actor of each player in it. A Player actor is a meeting point: Halo's companion phone app asks a player's actor which game they're in, then subscribes to that Game Session actor for live updates. The Statistics service uses the same actors: when a game ends, the session sends each player's part of the statistics to that player's Player actor, and those are stored durably, because players hate losing their achievements.
  4. The paper points out that Halo's developers didn't write any code to find or create these actors. They called them by ID, and Orleans took care of where each one ran.

Notice what is not here: supervision. Because actors are virtual, the Orleans paper says the framework "has no need for supervision trees" like Erlang's and Akka's. If a server dies, no actor dies with it in any lasting sense: the next call to an actor that lived there creates a new activation on a surviving server. For a game session whose real state lives on the consoles, that's a perfect fit.

The result

This is where you have to read carefully. The Orleans paper says most of Halo's production scale and performance figures are confidential, so the famous numbers come from a lab. The researchers ran the actual Halo 4 Presence code on a test cluster:

  • 25 servers handled about 130,000 heartbeats per second.
  • Throughput scaled almost linearly as they grew the cluster to 125 servers.
  • The median latency was about 6.5 milliseconds at up to 19% CPU use.

What we know from production is the launch curve above, and that Microsoft kept using Orleans: Phil Bernstein, a co-author of the paper, said in a 2016 interview that it ran the cloud services for Halo 4 and Halo 5. Microsoft previewed Orleans publicly in April 2014 and released it as open source on GitHub in January 2015. Microsoft's documentation today lists Azure, Xbox, Skype, PlayFab and Gears of War among its users.

One caution if you read the 2014 paper today: some details have changed since. For example, the paper guaranteed a single activation of each actor only eventually during failures, and the default placement has since moved from random to resource-aware. Don't assume Halo 4 ran on today's Orleans.

The lesson: if your actors map to things that always exist conceptually, such as players and matches, letting the runtime handle their lifecycle removes a whole class of code: no spawning, no lookups, no supervisors. You trade that for less control over where and when each actor runs.

What the three have in common

Discord LinkedIn Halo
Runtime Elixir on the Erlang VM Akka on the JVM Orleans on .NET
An actor is a guild (server), a user session a connection, an online member a router, a game session, a player
Created by the code, explicitly the code, explicitly nobody: activated on first call
When one fails contained to that process; load control kept the rest up not covered in the posts re-activated on the next call; no supervision trees
Reported scale ~5M concurrent users (2017); 12M+ (2020) 100K connections per node (2016) 4M users in launch week (2012); lab: ~130K heartbeats/s on 25 servers

A few patterns repeat across all three:

  • One actor per thing in the domain. A guild, a connection, a member, a player, a game. The actor is the single owner of that thing's state, so nothing else needs a lock on it.
  • Messages are the only way in. That's what made it possible for Discord to batch sends per node, and for LinkedIn to treat a heartbeat as just another message.
  • Hot actors are the real limit. An actor handles one message at a time. Discord's big guilds show what happens when one actor gets too much traffic, and every fix there is about controlling what reaches that actor's mailbox.
  • Shared data doesn't belong in an actor's mailbox. Discord moved its hash ring to shared read-only storage; LinkedIn kept presence state in a key-value store. Actors own their state; data that everyone reads belongs somewhere everyone can read it.
  • The failures were operational, not conceptual. Memory per connection, mailbox growth, reconnect storms. None of these teams went back to threads and locks.

If you're building something where many independent things each have a little state and talk to each other, such as chat rooms, devices, players or shopping carts, these three systems are good evidence that actors scale a long way.

Sources

Discord

LinkedIn

Halo and Orleans

Top comments (0)