DEV Community

Cover image for The Actor Model: Why It Was Invented and How It Works
Hamed Yousefi
Hamed Yousefi

Posted on Originally published at hmdsefi.hashnode.dev AI-assisted

The Actor Model: Why It Was Invented and How It Works

#distributedsystems #systemdesign #elixir #architecture

If you have written concurrent code with threads, you know the routine. Several threads touch the same data, so you put a lock around it. Then you add a second lock, take the two in the wrong order on a busy day, and two threads wait for each other forever. Or you forget the lock once, and a counter is off by one in production every few weeks.

The actor model starts from a different question: what if no two pieces of code ever shared memory at all?

This article covers where the idea came from, the problem each step in its history answered, and how an actor works, in pictures. A follow-up article looks at three companies that run actors in production.

The problem: shared memory and locks

Here is a tiny shop with one item left in stock, written two ways.

Two panels. Left: three threads go through one mutex to a shared stock variable, with warnings that a forgotten lock is a data race, two locks in different orders deadlock, and every lock makes threads wait. Right: alice and bob send Buy messages to a shop actor's mailbox, and only the shop reads and writes its private stock.
Left: every thread must remember the lock. Right: only the shop touches stock, so there is nothing to lock.

On the left, three threads read and write one stock variable. Correctness depends on every thread taking the lock, every time, in the same order. The lock also makes threads that want the stock run one after another, which is the opposite of why you added threads. Akka's documentation sums up the second risk in five words: "Locks introduce a new menace: deadlocks."

On the right, the stock belongs to one actor, the shop. Nobody else can read or write it. Anyone who wants to buy sends the shop a message, and the shop handles its messages one at a time. There's no shared variable, so there is nothing to lock.

That is the whole idea. The rest of this article explains how the idea grew up, and what you need around it to build real systems.

Where actors came from

A timeline from 1973 to 2014: the actor formalism of Hewitt, Bishop and Steiger; Agha's definition of what an actor does; Erlang at Ericsson; the OTP project; Open Source Erlang; Armstrong's thesis on let it crash; Akka on the JVM; Orleans and virtual actors.
Navy: the theory. Violet: Erlang and OTP, where supervision comes from. Blue and amber: the model reaching the JVM and .NET.

1973: a theory with one kind of object

In August 1973, Carl Hewitt, Peter Bishop and Richard Steiger presented A Universal Modular ACTOR Formalism for Artificial Intelligence at the third International Joint Conference on Artificial Intelligence (IJCAI). Their proposal was radical in its simplicity: everything is an actor, and everything an actor does is send messages to other actors.

The problem they were answering was how to describe computation that happens in many places at once. The usual tools of the time were goto, interrupts and shared variables guarded by semaphores. The paper says its formalism "does not have or need goto, interrupt, or semaphore primitives". Message passing replaces all of them.

1985: what an actor does with a message

The 1973 paper was broad. Gul Agha's 1985 MIT technical report, ACTORS: A Model of Concurrent Computation in Distributed Systems (a book from MIT Press in 1986), pinned the model down. When an actor receives a message, it responds with three things:

  1. a set of messages it sends to other actors,
  2. a set of new actors it creates,
  3. a replacement behavior, which decides how it handles the next message.

Every actor has a mail address, and an actor can only send to addresses it knows. That is the definition most actor systems still follow. Hewitt's later papers state the same three actions and add a point worth remembering: the model itself does not require a mailbox or a thread. Those are implementation choices, though nearly every practical system adds a mailbox.

1986: Erlang reaches the same design by another road

At the Ericsson Computer Science Laboratory, Joe Armstrong, Robert Virding and Mike Williams started work on a new language in 1986. Armstrong's history of the language gives its goal: "a better way of programming telephony applications". A telephone switch handles a huge number of calls at once, must keep running for years, and can't stop because one call hit a bug.

Erlang's answer was many small processes that share no memory and communicate only by asynchronous messages. That is an actor in everything but name, and the name was missing on purpose. On a 2019 panel with Hewitt and Tony Hoare, Armstrong said: "I didn't know about the actor stuff at the time". In a 2014 mailing-list post he named Prolog and Smalltalk as Erlang's influences instead. Two groups, a theorist and a telecom lab, arrived at the same design from different problems.

1996 to 2003: supervision and "let it crash"

Ericsson started the Open Telecom Platform (OTP) project in 1996: a set of Erlang libraries for building telecom systems. It released Erlang and OTP as open source in December 1998.

Armstrong's 2003 PhD thesis, Making reliable distributed systems in the presence of software errors, explains the error-handling style behind OTP. Its slogan is "Let it crash." Instead of writing defensive code for every possible failure, a process that hits an unexpected error simply dies, and a separate process, its supervisor, recovers. Supervisors watch workers and other supervisors, so a whole system becomes a tree of them. We'll come back to this below.

The thesis also reports on Ericsson's AXD301 ATM switch, which had over 1.7 million lines of Erlang in 2003. You may have seen the famous "nine nines" (99.9999999%) availability figure for it. The thesis does quote a customer's figure of that size, and it also notes that how the figure was obtained was not documented. Treat it as a story about a very reliable system, not a measurement.

2009: actors on the JVM

Jonas Bonér started Akka in 2009 (its first commit is from February that year). Akka brought actors, mailboxes and supervision to the Java Virtual Machine, with Scala and Java APIs, so JVM teams could use the model without leaving their platform. Akka's documentation still opens with the problem it answers: the costs of locks and shared memory on modern CPUs.

2014: virtual actors

In 2014, Microsoft Research described Orleans, a .NET framework built around virtual actors. In classic actor systems you create an actor, keep its address, and stop it when you are done. Orleans actors are, in the paper's words, "purely logical entities that always exist, virtually". You never create or destroy them: you call an actor by its ID, and the runtime activates it on some server if it isn't already running. The problem it answered was cloud services with millions of objects, such as players, where managing each actor's lifecycle by hand is a burden. The second article shows Orleans running a real game backend.

How an actor works

With the history in place, here is the model in one picture.

An actor with its address, counter@node-1, a mailbox holding messages, and a behavior, receive(msg), with private state total: 6. Three arrows leave the behavior: send messages to actors whose addresses it has, create actors and get their addresses, and become a new behavior for the next message.
An actor is an address, a mailbox and a behavior with private state. For each message it can do only the three things on the right.

An actor has three parts:

  • An address. You can send a message only to an actor whose address you have. You get one by creating the actor, or by receiving the address in a message. The address is how the rest of the system refers to the actor; nobody holds a pointer to its insides.
  • A mailbox. Messages wait here until the actor takes them. Sending is asynchronous: the sender drops the message off and carries on.
  • A behavior with private state. The actor takes one message at a time, runs its behavior to completion, and only then takes the next. Only the actor reads or writes its state.

For each message, the behavior can do three things: send messages to addresses it knows, spawn new actors, and become a new behavior for the next message. Agha called the last one a replacement behavior; Akka's classic API calls it become. In practice, "become" is often just "update my own state", like setting total from 6 to 7.

Here is a counter actor in pseudocode:

actor Counter(total = 0):
    receive Add(n):
        total = total + n            # only this actor ever touches total
    receive Get(replyTo):
        send(replyTo, Total(total))  # answer with a message, not a return value
Enter fullscreen mode Exit fullscreen mode

Notice what's missing: no lock, no synchronized, no atomic. total can't be corrupted by two threads, because only one message is ever being handled, and only this actor can see total. Akka's docs make the same point: because at most one message is processed per actor at a time, the actor's invariants hold without synchronization.

Also notice Get. There is no return value, because there is no call stack shared between the two actors. The asker includes its own address, and the counter replies with a message.

One message at a time

The mailbox is what turns concurrent senders into sequential processing. Here, three people use one bank account at the same moment:

Animation: alice, bob and carol send Deposit 50, Withdraw 30 and Withdraw 130 to an account actor at the same moment. The messages queue in the mailbox in arrival order. Starting from 100, the account pays bob and leaves 70, adds alice's deposit for 120, and declines carol because 120 is less than 130.
Watch the balance box: it changes only when the account takes the next message from its mailbox, never while another message is in progress.

alice, bob and carol run concurrently, but the account handles their messages strictly one after another. It needs no lock on balance.

The last frame of the animation is the honest part. The order in which messages from different senders arrive is not fixed. In this run carol is declined; in another, bob would be. Actors remove data races, but they don't remove race conditions in your business logic. If order matters, you have to design for it, for example by putting the decision in one actor that sees all the requests, or by making the protocol explicit.

Location transparency

Because an actor is known only by its address, the sender's code doesn't depend on where the receiver runs.

Two nodes. On node-1 a checkout actor sends AddItem to a cart actor on the same node and Reserve to an inventory actor on node-2, with the same send call. The local message goes straight into the cart's mailbox; the remote one is serialized, crosses the network and lands in the inventory's mailbox.
Path 1 stays in memory. Path 2 is serialized and crosses the network. The send call looks the same for both.

Akka calls this location transparency. Its documentation says its functions work the same in a single JVM or on a cluster, because every interaction between actors is an asynchronous message anyway.

This is what lets an actor system grow from one machine to many without rewriting the code that sends messages. The warning icon in the picture is the catch. The code is the same, but the physics isn't: a remote message is slower and can be lost. Well-designed actor systems treat every message as one that might not arrive, and use timeouts and retries where it matters.

Let it crash: supervision trees

Actors that create other actors form a tree. In Erlang/OTP and Akka, each parent supervises its children: when a child fails, the parent decides what happens.

Animation of a supervision tree: a root supervisor, a supervisor named payments with the one_for_one strategy, and three workers. A bad message makes worker-2 crash. The supervisor gets the exit, its strategy picks restart, worker-2's old state is thrown away and it starts again with fresh state, while worker-1 and worker-3 keep counting.
Follow worker-2's count: 41 before the crash, 0 after the restart. Its siblings never stop counting.

The numbered steps in the animation are the whole recovery:

  1. A bad message makes worker-2 crash. It doesn't try to recover in place.
  2. Its supervisor, payments, is told that the child exited.
  3. The supervisor's strategy picks what to do. Here it's OTP's one_for_one: restart only the failed child.
  4. The worker's old, possibly corrupt, state is thrown away.
  5. It starts again from a known good state.
  6. New messages reach the restarted worker, and the siblings never noticed.

OTP's supervisor documentation defines other strategies too. one_for_all restarts every child when one fails, for children that only make sense together. And restarts are bounded: if a child needs more than a set number of restarts within a set number of seconds (by default, more than 1 restart in 5 seconds), the supervisor gives up, stops its children and itself, and its own parent decides what to do next. A local problem that a restart can't fix gets escalated up the tree.

Why is crashing a good idea? Because most production bugs are rare combinations of state and input. Code that tries to handle every unexpected case gets long and usually wrong. A fresh start from known state fixes the common cases, keeps the error-handling code in one place (the supervisor), and keeps the failure contained to one small actor instead of the whole process.

What actors buy you, compared with locks

Shared memory and locks Actors
Who touches the data Any thread with a reference Only the owning actor
Data races Possible whenever a lock is missed Impossible on actor state
Deadlocks from lock ordering Possible No locks to order
Failure An exception unwinds one thread's stack, often far from anyone who can fix it The actor dies and its supervisor decides
Scaling out A rewrite: shared memory doesn't cross machines The same send, to an address on another node

Actors are not free, and it's worth being clear about the costs:

  • Ordering still matters. As the account example showed, messages from different senders can arrive in any order.
  • Mailboxes can fill up. If an actor receives messages faster than it handles them, its mailbox grows. Production systems need back-pressure, bounded mailboxes, or both. The second article shows a real case of exactly this.
  • Two actors can still wait on each other. If A waits for a reply from B while B waits for a reply from A, neither progresses. Timeouts help; so does designing protocols that don't need round trips.
  • Debugging is different. A bug is a sequence of messages, not a stack trace, so good tracing and logging matter more.

When to reach for actors

Actors fit best when your problem already looks like many independent things with their own state that talk to each other: users, sessions, devices, game players, chat rooms, orders. Each of those becomes an actor, and the runtime spreads them over your cores and machines.

They fit less well for bulk number crunching over shared arrays, or for simple request-response services that keep no state between requests.

The model is fifty years old, and it has outlived many of the languages that first used it, because the problem it answers hasn't changed. Concurrent programs are hard when everything can touch everything. Actors make each piece of state belong to exactly one owner, and they turn failure into something a supervisor can deal with.

In the next article, we'll look at how Discord, LinkedIn and the team behind Halo used actors to serve millions of users.

Sources

Top comments (0)