<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Hamed Yousefi</title>
    <description>The latest articles on DEV Community by Hamed Yousefi (@hamed_yousefi).</description>
    <link>https://dev.to/hamed_yousefi</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4152817%2Fd8fb6028-5cf1-44a9-9971-2bc5303186d5.png</url>
      <title>DEV Community: Hamed Yousefi</title>
      <link>https://dev.to/hamed_yousefi</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/hamed_yousefi"/>
    <language>en</language>
    <item>
      <title>Actors in Go: How lockstep Runs One Actor in Your Program and in Tests</title>
      <dc:creator>Hamed Yousefi</dc:creator>
      <pubDate>Mon, 02 Nov 2026 17:02:24 +0000</pubDate>
      <link>https://dev.to/hamed_yousefi/actors-in-go-how-lockstep-runs-one-actor-in-your-program-and-in-tests-kk4</link>
      <guid>https://dev.to/hamed_yousefi/actors-in-go-how-lockstep-runs-one-actor-in-your-program-and-in-tests-kk4</guid>
      <description>&lt;p&gt;&lt;a href="https://hmdsefi.hashnode.dev/actor-model-why-it-was-invented-how-it-works" rel="noopener noreferrer"&gt;Part 1&lt;/a&gt; of this series explained actors in pictures, and &lt;a href="https://hmdsefi.hashnode.dev/actor-model-in-production-discord-linkedin-halo" rel="noopener noreferrer"&gt;part 2&lt;/a&gt; showed Discord, LinkedIn and Halo running them on Elixir, Akka and Orleans. None of those runtimes is Go. Go gives you goroutines and channels and leaves the rest to you. You decide which goroutine owns which state, what happens when a channel fills up, what restarts a goroutine that panicked, and how to test any of it.&lt;/p&gt;

&lt;p&gt;I've been building lockstep to answer those questions with actors. It's an actor runtime for Go, and v0.2.0 came out on 1 November 2026. You write actors with typed messages, bounded mailboxes, supervision and timers, and lockstep runs them in your program on a pool of worker goroutines. The same actor code also runs in &lt;code&gt;go test&lt;/code&gt;, in a simulation with its own clock and network. There, one number, the &lt;em&gt;seed&lt;/em&gt;, decides the order of every message.&lt;/p&gt;

&lt;p&gt;This article builds part 1's one-item shop with lockstep and runs it in both places.&lt;/p&gt;

&lt;h2&gt;
  
  
  The shop from part 1, in Go
&lt;/h2&gt;

&lt;p&gt;Part 1 opened with a shop that has one item left in stock. As an actor, the shop owns &lt;code&gt;stock&lt;/code&gt;, buyers send it &lt;code&gt;Buy&lt;/code&gt; messages, and there is nothing to lock. A real shop takes payment before it hands over the item, so this one charges a card. In lockstep the shop is a Go type with a &lt;code&gt;Receive&lt;/code&gt; method. All the code in this article ran on lockstep v0.2.0 and Go 1.27.1:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="c"&gt;// Msg is every message the shop accepts.&lt;/span&gt;
&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Msg&lt;/span&gt; &lt;span class="k"&gt;interface&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;isMsg&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;// Buy asks the shop to sell one item to User.&lt;/span&gt;
&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Buy&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;User&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;// Sold asks how many items the shop has sold.&lt;/span&gt;
&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Sold&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;Reply&lt;/span&gt; &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ReplyTo&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;// charged is the result of a Charge effect.&lt;/span&gt;
&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;charged&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;user&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;
    &lt;span class="n"&gt;err&lt;/span&gt;  &lt;span class="kt"&gt;error&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Buy&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;isMsg&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;     &lt;span class="p"&gt;{}&lt;/span&gt;
&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Sold&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;isMsg&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;    &lt;span class="p"&gt;{}&lt;/span&gt;
&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;charged&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;isMsg&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;

&lt;span class="c"&gt;// Shop sells a limited stock.&lt;/span&gt;
&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Shop&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;stock&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sold&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;// NewShop returns a factory for a shop with stock items.&lt;/span&gt;
&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;NewShop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stock&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Actor&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Msg&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Actor&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Msg&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;Shop&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;stock&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;stock&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Shop&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Receive&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Msg&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="n"&gt;Msg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;switch&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;Buy&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stock&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="c"&gt;// sold out&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stock&lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt; &lt;span class="c"&gt;// reserve the item before the slow payment call&lt;/span&gt;
        &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Do&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Charge&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;User&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;User&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{},&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Msg&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;charged&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;User&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;charged&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stock&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt; &lt;span class="c"&gt;// payment failed: put the item back&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sold&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;Sold&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Reply&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Send&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sold&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The parts of an actor from part 1 are all here, with Go types on them:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq43zedhx662ya4k9s785.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq43zedhx662ya4k9s785.png" alt="The shop actor in lockstep. Its address is a PID[Msg], lockstep://shop/user/shop#100000003. Buy messages from alice and bob wait in a bounded mailbox of 1024 messages. Receive(ctx, msg) takes one message at a time and owns the private state stock: 1 and sold: 0. Four arrows leave Receive: send messages with lockstep.Send or ReplyTo.Send, create actors with lockstep.Spawn, update its own fields for the next message, and start I/O with lockstep.Do, whose result comes back to the mailbox as a charged message." width="799" height="475"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Part 1's actor, with Go types and the PID the shop gets in the test below. The fourth arrow is new: the actor starts I/O and gets the result back as a message.&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The address is a typed PID.&lt;/strong&gt; &lt;code&gt;lockstep.Spawn&lt;/code&gt; returns a &lt;code&gt;PID[Msg]&lt;/code&gt;, which accepts only &lt;code&gt;Msg&lt;/code&gt; values, so a message of another type doesn't compile.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The mailbox is bounded.&lt;/strong&gt; It holds 1024 messages by default. When it's full, &lt;code&gt;Send&lt;/code&gt; returns &lt;code&gt;ErrMailboxFull&lt;/code&gt; instead of letting the queue grow. Part 1 listed unbounded mailboxes as a cost of Erlang's and Akka's defaults, and part 2 showed Discord paying it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;Receive&lt;/code&gt; handles one message at a time&lt;/strong&gt;, so &lt;code&gt;stock&lt;/code&gt; and &lt;code&gt;sold&lt;/code&gt; need no lock.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A reply is a message.&lt;/strong&gt; &lt;code&gt;Sold&lt;/code&gt; carries a &lt;code&gt;ReplyTo[int]&lt;/code&gt;, the typed version of part 1's &lt;code&gt;Get(replyTo)&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The shop takes the item out of stock before it charges the card. A charge takes tens of milliseconds, and another &lt;code&gt;Buy&lt;/code&gt; can arrive while it's in flight. Part 1's account animation ended on exactly that: the order in which messages arrive decides who is declined. Reserving first checks and takes the item in one &lt;em&gt;turn&lt;/em&gt;, lockstep's name for one call to &lt;code&gt;Receive&lt;/code&gt;, so no order of messages can sell it twice. Part 4 runs the version that takes the item after the charge.&lt;/p&gt;
&lt;h2&gt;
  
  
  Effects: I/O that doesn't block the actor
&lt;/h2&gt;

&lt;p&gt;Charging a card means waiting on the network, and an actor must not wait inside &lt;code&gt;Receive&lt;/code&gt;, because its mailbox doesn't move while it waits. &lt;code&gt;lockstep.Do&lt;/code&gt; starts an &lt;em&gt;effect&lt;/em&gt; and returns at once. When the payment provider answers, the function passed to &lt;code&gt;Do&lt;/code&gt; turns the result into a &lt;code&gt;charged&lt;/code&gt; message, and the shop handles it in a later turn. &lt;code&gt;Charge&lt;/code&gt; is the effect, and its &lt;code&gt;Run&lt;/code&gt; method makes the real call:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="c"&gt;// Charge is the effect that charges a user's card.&lt;/span&gt;
&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Charge&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;User&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Charge&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s"&gt;"payment.charge"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="n"&gt;Charge&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{},&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{}{},&lt;/span&gt; &lt;span class="n"&gt;chargeCard&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;User&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c"&gt;// the payment provider's API&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Effects are part of one rule for all actor code: it doesn't block, start goroutines, read the wall clock, or use global randomness. Time comes from &lt;code&gt;ctx.Now&lt;/code&gt;, &lt;code&gt;ctx.After&lt;/code&gt; and &lt;code&gt;ctx.Every&lt;/code&gt;, randomness from &lt;code&gt;ctx.Rand&lt;/code&gt;, and I/O from effects. When every delivery, timer, random number and I/O call goes through the runtime, the runtime decides all of them. That's what lets a second engine, the simulation below, decide them differently.&lt;/p&gt;

&lt;p&gt;lockstep ships an analyzer, &lt;code&gt;lockstep-vet&lt;/code&gt;, that checks the rule. Here it is on a version of the shop that charges the card in a goroutine and records each sale with &lt;code&gt;time.Now&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;shop.go:47:3: LS100: go statement starts a goroutine; use lockstep.Do for work outside the turn (actor code: (*Shop).Receive)
shop.go:56:21: LS100: time.Now uses the wall clock; use ctx.Now, ctx.After or ctx.Every (actor code: (*Shop).Receive)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It runs through &lt;code&gt;go vet&lt;/code&gt; or as a golangci-lint plugin. It also reports an ignored error from &lt;code&gt;Send&lt;/code&gt;, a message the sender keeps using after it sent it, and a reply that nobody waits for. And it reports a message type that &lt;code&gt;Receive&lt;/code&gt; has no case for.&lt;/p&gt;

&lt;h2&gt;
  
  
  The shop in your program
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;lockstep.New()&lt;/code&gt; starts the engine your program runs on, which lockstep calls the production engine. This program spawns the shop with one item, lets Alice buy, lets Bob buy 30 ms later, and asks how many items were sold:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;sys&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Shutdown&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Background&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="p"&gt;}()&lt;/span&gt;

    &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Spawn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"shop"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewShop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="c"&gt;// one item in stock&lt;/span&gt;
    &lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Println&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"spawned"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c"&gt;// Alice buys first; Bob buys 30 ms later.&lt;/span&gt;
    &lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Send&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Msg&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Buy&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;User&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"alice"&lt;/span&gt;&lt;span class="p"&gt;}))&lt;/span&gt;
    &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;30&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Send&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Msg&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Buy&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;User&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"bob"&lt;/span&gt;&lt;span class="p"&gt;}))&lt;/span&gt;
    &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c"&gt;// the payment answers in 10 to 59 ms&lt;/span&gt;

    &lt;span class="n"&gt;sold&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ReplyTo&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Msg&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Sold&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;Reply&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Await&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Background&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Println&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"sold:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sold&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;chargeCard&lt;/code&gt; here is a stand-in for the payment provider that answers after 10 to 59 ms, and &lt;code&gt;check&lt;/code&gt; exits on an error:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;spawned lockstep://local/user/shop#6494268c00000003
sold: 1
2026/11/02 17:07:44 INFO shutdown started node=local actors=1
2026/11/02 17:07:44 INFO shutdown finished node=local elapsed=552.709µs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;Ask&lt;/code&gt; puts a &lt;code&gt;ReplyTo[int]&lt;/code&gt; into the &lt;code&gt;Sold&lt;/code&gt; message and returns a reply that &lt;code&gt;Await&lt;/code&gt; waits for. &lt;code&gt;Await&lt;/code&gt; blocks, so it belongs outside actors. Inside an actor, &lt;code&gt;lockstep.Pipe&lt;/code&gt; delivers the reply as a message instead. The last two lines come from the runtime's &lt;code&gt;log/slog&lt;/code&gt; logger, which also logs failures, restarts and overload.&lt;/p&gt;

&lt;p&gt;The production engine runs actor turns on a pool of worker goroutines, &lt;code&gt;GOMAXPROCS&lt;/code&gt; of them by default:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Turns of different actors run in parallel; turns of one actor never overlap.&lt;/strong&gt; In one turn, an actor handles up to 32 messages before it gives its worker to another actor. A worker with nothing to do takes ready actors from a busier worker's queue.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Messages from one sender to one receiver arrive in send order.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Effects run on their own pool.&lt;/strong&gt; At most 64 run at once and 1024 more wait. A &lt;code&gt;Do&lt;/code&gt; that finds both full resolves with &lt;code&gt;ErrOverloaded&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A full mailbox has a policy.&lt;/strong&gt; The default fails the send. The others drop the newest or oldest message or block the sender.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Supervision follows Erlang's.&lt;/strong&gt; One-for-one and all-for-one strategies restart failed actors, with restart limits and backoff.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The same shop in a test
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;sim.Simulate&lt;/code&gt; runs a test body once per seed, each time in a fresh simulated world. This one has three nodes: a server for the shop, and two clients, &lt;code&gt;alice&lt;/code&gt; and &lt;code&gt;bob&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;TestOneItemTwoBuyers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;opts&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;sim&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Options&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;Options&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;faultline&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Options&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;Seeds&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="m"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;
    &lt;span class="n"&gt;sim&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Simulate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;oneItemTwoBuyers&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;oneItemTwoBuyers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;sim&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Sim&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;World&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;shop&lt;/span&gt; &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;PID&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Msg&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;spawnErr&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Node&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"shop"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;spawnErr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Spawn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"shop"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;NewShop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="c"&gt;// one item in stock&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"alice"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{})&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"bob"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{})&lt;/span&gt;

    &lt;span class="c"&gt;// The payment API answers after 10 to 60 ms, chosen by the seed.&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;HandleEffect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"payment.charge"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="n"&gt;sim&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;EffectCall&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;sim&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;EffectResult&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;sim&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;EffectResult&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{}{},&lt;/span&gt; &lt;span class="n"&gt;Latency&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Uniform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Rand&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="m"&gt;10&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;60&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;

    &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RunFor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c"&gt;// let the nodes boot&lt;/span&gt;

    &lt;span class="c"&gt;// Alice buys first; Bob buys 30 ms later.&lt;/span&gt;
    &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sim&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Send&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Msg&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"alice"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Buy&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;User&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"alice"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RunFor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;30&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sim&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Send&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Msg&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"bob"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Buy&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;User&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"bob"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RunFor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;sold&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;sim&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"alice"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;shop&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="n"&gt;lockstep&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ReplyTo&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;Msg&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;Sold&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;Reply&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Final&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"never sell more than the stock"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;spawnErr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;sold&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"sold %d items with 1 in stock"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sold&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The timing is the program's, in virtual time, the simulation's own clock: Alice buys at 10 ms and Bob 30 ms later. After a second, Alice asks how many items were sold, and the final check runs when the two seconds are up. &lt;code&gt;w.RunFor(time.Second)&lt;/code&gt; moves the clock forward without sleeping. All 100 seeds pass:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ok      example.com/shop    0.303s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The test calls the same &lt;code&gt;Receive&lt;/code&gt; as the program. What changes is the engine underneath:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7jplfixfimzzngu3kp7m.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7jplfixfimzzngu3kp7m.png" alt="Two engines run the same Receive. Left, the production engine behind lockstep.New(): a global queue of ready actors and three workers with local queues, where workers 1 and 2 run turns of actors and idle worker 3 steals half of worker 1's queue; Charge.Run calls the payment provider on a bounded pool. Right, the simulation engine behind sim.Simulate on faultline: one event queue in virtual-time order holding real events from the shop's run: Buy from alice at 13.8 ms, Buy from bob at 41.6 ms, the payment.charge result at 48.4 ms, and Sold from alice at 1.042 s. The seed orders events at the same time and draws every latency and fault, and a test handler answers payment.charge. A table compares turns, turn budget, time, randomness, effects and replay." width="800" height="637"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Both engines call the same &lt;code&gt;Receive&lt;/code&gt;. They differ in who picks the next turn, what time it is, and what answers an effect.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;sim.Simulate&lt;/code&gt; runs on &lt;a href="https://github.com/hmdsefi/faultline" rel="noopener noreferrer"&gt;faultline&lt;/a&gt;, my deterministic simulation library for Go. Every turn handles one message and is one event in a single queue ordered by virtual time. The seed breaks ties between events at the same time, and it draws every network latency and every effect's latency. &lt;code&gt;Charge.Run&lt;/code&gt; never runs, because a call to a real payment provider can't be replayed. The test answers the effect by its name, &lt;code&gt;payment.charge&lt;/code&gt;, with a handler instead.&lt;/p&gt;

&lt;p&gt;Here is the first of the 100 seeds, drawn from its trace:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxwnte3y9uir68430tpot.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxwnte3y9uir68430tpot.gif" alt="Animation of seed 0xb12302fa26394756 on the shop that reserves first, drawn from its trace. The shop starts with stock 1 and sold 0. Alice's Buy arrives at 13.8 ms; the shop reserves the item, so stock drops to 0, and starts a charge. Bob's Buy arrives at 41.6 ms while the charge is in flight; stock is 0, so the shop turns him away. Alice's charge succeeds at 48.4 ms and sold becomes 1. At 1.04 s Alice asks how many were sold and gets 1, and at 2 s the final check passes. The run has 57 trace records and the trace hash 0xa183b40775972eab." width="760" height="497"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Bob's &lt;code&gt;Buy&lt;/code&gt; arrives while Alice's charge is still in flight. The stock already reads 0, so the shop turns him away. Every time in the animation comes from the trace.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;On this seed, Bob's &lt;code&gt;Buy&lt;/code&gt; arrives at 41.6 ms, 7 ms before Alice's charge comes back. On other seeds the charge comes back sooner, and Bob's &lt;code&gt;Buy&lt;/code&gt; arrives after it. The shop sells one item either way.&lt;/p&gt;

&lt;p&gt;A run is a list of events, and the trace hash is a hash of all of them. Run this seed again with &lt;code&gt;FAULTLINE_TRACE=full&lt;/code&gt;, and its trace files show the same 57 records and the same hash, &lt;code&gt;0xa183b40775972eab&lt;/code&gt;. When a seed fails, the output includes a command that replays that exact run.&lt;/p&gt;
&lt;h2&gt;
  
  
  State that survives a restart
&lt;/h2&gt;

&lt;p&gt;The shop keeps its stock in memory. If it restarts, &lt;code&gt;NewShop(1)&lt;/code&gt; creates it again with one item in stock and nothing sold, so it can sell the same item twice. State that must survive goes in a persistent actor, which v0.2.0 adds in &lt;code&gt;lockstep/persist&lt;/code&gt;. This one is the bank account from part 1, written as event sourcing: a &lt;code&gt;Deposit&lt;/code&gt; command stores a &lt;code&gt;Deposited&lt;/code&gt; event, and the balance is the sum of the events:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Account&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Initial&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="kt"&gt;int64&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Account&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Handle&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Deposit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Deposited&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int64&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="kt"&gt;int64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="n"&gt;Deposit&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="n"&gt;Deposited&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Then&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Deposit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Deposited&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int64&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;balance&lt;/span&gt; &lt;span class="kt"&gt;int64&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Reply&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Send&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;balance&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c"&gt;// runs after the event is committed&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="n"&gt;Deposited&lt;/span&gt;&lt;span class="p"&gt;{{&lt;/span&gt;&lt;span class="n"&gt;Amount&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Amount&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Account&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Apply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;balance&lt;/span&gt; &lt;span class="kt"&gt;int64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="n"&gt;Deposited&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;int64&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;balance&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Amount&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The README's program spawns it with &lt;code&gt;persist.Spawn&lt;/code&gt; on a store in the directory &lt;code&gt;data&lt;/code&gt;, deposits 5, and prints the balance. Run it twice:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go run &lt;span class="nb"&gt;.&lt;/span&gt;
go run &lt;span class="nb"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;balance: 5
balance: 10
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second run starts a new system on the same directory. The account recovers the balance of 5 from its journal before it handles the new deposit:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzxz8whg2ikc952ch459y.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzxz8whg2ikc952ch459y.png" alt="A persistent actor handles Deposit 5. Handle returns the event Deposited 5. lockstep writes the event to the journal, then commits it with fsync, then applies it to the state, balance 5 to 10, and only then runs the Then callback that replies 10. On restart, the actor recovers its state from the newest snapshot and the events after it before it takes the next command. In production the store uses the operating system's files; in a simulation the same store runs on faultline's simulated disk." width="799" height="452"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The reply runs after the fsync, so a reply means the event is on disk. A restart rebuilds the state from the journal.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Handle&lt;/code&gt; returns events instead of changing the state, and &lt;code&gt;Apply&lt;/code&gt; folds one event into the state. lockstep writes the events to a journal, calls fsync, and only then applies them and runs the callbacks registered with &lt;code&gt;ctx.Then&lt;/code&gt;. While the actor recovers or commits, its commands wait in the mailbox. With a snapshot store, it also saves a snapshot every &lt;code&gt;SnapshotEvery&lt;/code&gt; events and recovers from the newest one.&lt;/p&gt;

&lt;p&gt;In your program, the store, &lt;code&gt;persist/filestore&lt;/code&gt;, keeps its files on the local disk. In a test, &lt;code&gt;filestore.New(s.Volume("n1"))&lt;/code&gt; runs the same store on node &lt;code&gt;n1&lt;/code&gt;'s simulated disk.&lt;/p&gt;
&lt;h2&gt;
  
  
  What lockstep is good for
&lt;/h2&gt;

&lt;p&gt;I'd reach for lockstep when correctness depends on the order of messages and failures: consensus, replication, and ledgers or inventories where a lost or doubled update costs money. It also fits services that retry and fail over, and workflows where many long-lived workers each keep state. Those are the shapes from part 2: one actor per guild, per online member, per game session, on one machine for now.&lt;/p&gt;

&lt;p&gt;It fits less well when one piece of state must be updated from many cores at once, because an actor handles one message at a time. A stateless request-response service doesn't need actors at all.&lt;/p&gt;
&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;You write actors.&lt;/strong&gt; lockstep simulates actors, not arbitrary goroutines, so existing goroutine code needs restructuring before a seed can decide its order.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your program is one process.&lt;/strong&gt; There is no remoting or clustering yet. A PID names its node, but nodes exchange messages only in a simulation, so part 1's location transparency exists only in tests for now.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A turn can't be preempted.&lt;/strong&gt; A &lt;code&gt;Receive&lt;/code&gt; that never returns holds its worker. lockstep logs a turn that has run for 5 seconds, with its worker's stack, but it can't stop it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Benchmarks compare it with three Go libraries.&lt;/strong&gt; The &lt;code&gt;bench&lt;/code&gt; module runs the same workloads on lockstep, protoactor-go, Hollywood and GoAkt. In the CI run that &lt;code&gt;bench/RESULTS.md&lt;/code&gt; reports, from 31 October 2026, ping-pong between one pair of actors took 783 ns per message on lockstep. It took 384 on Hollywood, 423 on GoAkt and 2,670 on protoactor-go. Hollywood and GoAkt also had lower send latency, and lower ask latency with one caller, than lockstep in that run. The file has every row.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The API can change before v1.0.&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;lockstep needs Go 1.26 or later:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go get github.com/hmdsefi/lockstep@v0.2.0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The README starts with part 1's counter, and the repository is at &lt;a href="https://github.com/hmdsefi/lockstep" rel="noopener noreferrer"&gt;github.com/hmdsefi/lockstep&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;In the next article, I'll take the item after the charge instead of before it, and a seed will find what goes wrong. Then I'll crash nodes, partition the network and retry requests.&lt;/p&gt;




&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;lockstep v0.2.0&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Release notes. &lt;a href="https://github.com/hmdsefi/lockstep/releases/tag/v0.2.0" rel="noopener noreferrer"&gt;https://github.com/hmdsefi/lockstep/releases/tag/v0.2.0&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;README: the counter, the production engine and the persistent account. &lt;a href="https://github.com/hmdsefi/lockstep/tree/v0.2.0#readme" rel="noopener noreferrer"&gt;https://github.com/hmdsefi/lockstep/tree/v0.2.0#readme&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;"Why lockstep": the shop example this article builds on. &lt;a href="https://github.com/hmdsefi/lockstep/blob/v0.2.0/docs/why-lockstep.md" rel="noopener noreferrer"&gt;https://github.com/hmdsefi/lockstep/blob/v0.2.0/docs/why-lockstep.md&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;"lockstep architecture": the two engines, effects, mailboxes and persistence. &lt;a href="https://github.com/hmdsefi/lockstep/blob/v0.2.0/docs/architecture.md" rel="noopener noreferrer"&gt;https://github.com/hmdsefi/lockstep/blob/v0.2.0/docs/architecture.md&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Benchmark results. &lt;a href="https://github.com/hmdsefi/lockstep/blob/v0.2.0/bench/RESULTS.md" rel="noopener noreferrer"&gt;https://github.com/hmdsefi/lockstep/blob/v0.2.0/bench/RESULTS.md&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Elsewhere&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;faultline, deterministic simulation testing for Go. &lt;a href="https://github.com/hmdsefi/faultline" rel="noopener noreferrer"&gt;https://github.com/hmdsefi/faultline&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Part 1: "The Actor Model: Why It Was Invented and How It Works". &lt;a href="https://hmdsefi.hashnode.dev/actor-model-why-it-was-invented-how-it-works" rel="noopener noreferrer"&gt;https://hmdsefi.hashnode.dev/actor-model-why-it-was-invented-how-it-works&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Part 2: "Actors in Production: How Discord, LinkedIn and Halo Use the Actor Model". &lt;a href="https://hmdsefi.hashnode.dev/actor-model-in-production-discord-linkedin-halo" rel="noopener noreferrer"&gt;https://hmdsefi.hashnode.dev/actor-model-in-production-discord-linkedin-halo&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>go</category>
      <category>opensource</category>
      <category>distributedsystems</category>
      <category>microservices</category>
    </item>
    <item>
      <title>Actors in Production: How Discord, LinkedIn and Halo Use the Actor Model</title>
      <dc:creator>Hamed Yousefi</dc:creator>
      <pubDate>Wed, 28 Oct 2026 17:06:16 +0000</pubDate>
      <link>https://dev.to/hamed_yousefi/actors-in-production-how-discord-linkedin-and-halo-use-the-actor-model-iio</link>
      <guid>https://dev.to/hamed_yousefi/actors-in-production-how-discord-linkedin-and-halo-use-the-actor-model-iio</guid>
      <description>&lt;p&gt;The actor model is easy to describe: small units that own their state, handle one message at a time, and talk only by messages. If you haven't met it yet, &lt;a href="https://hmdsefi.hashnode.dev/actor-model-why-it-was-invented-how-it-works" rel="noopener noreferrer"&gt;my previous article&lt;/a&gt; covers where it came from and how it works. This one is about what happens when real systems with millions of users are built that way.&lt;/p&gt;

&lt;p&gt;I picked three systems that published detailed engineering write-ups, each on a different runtime:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Discord&lt;/strong&gt; runs chat on Elixir, on the Erlang virtual machine (the BEAM).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;LinkedIn&lt;/strong&gt; built instant messaging and its "who's online" dots on Akka, on the JVM.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Halo&lt;/strong&gt;'s cloud services ran on Microsoft Orleans, on .NET.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For each one: what they needed, how actors fit, what broke, and the result. Every number below comes from the companies' own posts and papers, with the date they gave it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Discord: one process per server, one per connection
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What they needed
&lt;/h3&gt;

&lt;p&gt;Discord is chat, voice and video organized into communities that Discord's code calls &lt;em&gt;guilds&lt;/em&gt; (users call them servers). The company's 2017 engineering post describes most of Discord as publish/subscribe: when anything happens in a guild, every member who's online must see it, right away.&lt;/p&gt;

&lt;p&gt;That gets expensive fast. A message in a guild with N people online goes to N sessions, and if all N people are active, the total work grows with N². Discord's 2023 post on very large servers works an example: with 100,000 people online, that's 10 billion notifications to deliver.&lt;/p&gt;

&lt;h3&gt;
  
  
  How actors fit
&lt;/h3&gt;

&lt;p&gt;Discord built its first prototype in Elixir, and its 2017 post explains why: "The Erlang VM was the perfect candidate for the highly concurrent, real-time system". The design maps straight onto actors:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Every connected user gets a &lt;strong&gt;session process&lt;/strong&gt; (a GenServer, Elixir's standard actor) that owns their WebSocket.&lt;/li&gt;
&lt;li&gt;Every guild gets one &lt;strong&gt;guild process&lt;/strong&gt;, on a separate set of nodes, that owns the guild's state, including which members' sessions are connected.&lt;/li&gt;
&lt;li&gt;When something happens, the guild process fans it out by sending a message to the session process of every member online.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5uyd4q4ezj88s0jhs5uo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5uyd4q4ezj88s0jhs5uo.png" alt="Discord's design. A guild process on a guild node keeps track of its members' sessions. A member's session process forwards a message to the guild process (step 1). The guild process handles it one message at a time (2), then Manifold sends one message per session node instead of one per member (3), and each node hands it to its local session processes, which push it to clients over WebSockets (4)." width="800" height="522"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Follow the numbers: the guild process is the single place where a guild's state lives, and step 3 is the optimization that made it scale.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Discord's later posts, from 2023 and from 2026, still describe this shape: one process per guild as the central routing point, fanning out to sessions by message passing.&lt;/p&gt;

&lt;h3&gt;
  
  
  What broke
&lt;/h3&gt;

&lt;p&gt;A guild process is an actor, so it handles one message at a time. That is what keeps its state consistent without locks. It's also its weak point. The 2017 post, written when Discord had nearly five million concurrent users, describes three failures, and each one is a lesson about actors:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Mailboxes fell behind.&lt;/strong&gt; Big guilds then had up to 30,000 people online. At peak, guild processes couldn't keep up with their message queues, and engineers had to turn off features by hand. A single Erlang &lt;code&gt;send&lt;/code&gt; took 30 to 70 microseconds, so publishing one event from a large guild could take 0.9 to 2.1 seconds. The fix, an open-source library called &lt;strong&gt;Manifold&lt;/strong&gt;, groups recipients by node and sends &lt;em&gt;one&lt;/em&gt; message per remote node; a process on that node does the local sends. The work is spread out, and the guild process moves on sooner. Manifold's README reports that packets per second between nodes dropped by half right after it shipped.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;One process held data everyone needed.&lt;/strong&gt; The hash ring that maps guilds to nodes was controlled by a single process, and it got so busy that it fell behind. Discord estimated that one session server restart cost about 30 seconds of ring lookups. Moving the ring into fast shared read-only storage (a library called &lt;strong&gt;FastGlobal&lt;/strong&gt;) cut each lookup to 0.3 microseconds.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;A stampede.&lt;/strong&gt; Nearly five million session processes all tried to call the same ten registry processes. The session processes' mailboxes ballooned until the virtual machine ran out of memory. The fix was a &lt;strong&gt;Semaphore&lt;/strong&gt; library that caps how many callers can hit a busy process at once.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Notice what these fixes have in common: none of them abandoned actors. Each one controls how many messages reach a single actor. That paid off later in the same post: when Discord's presence services crashed for an unrelated reason, the session services carried on unaffected, and Discord credits the Semaphore library.&lt;/p&gt;

&lt;p&gt;The work didn't stop in 2017. In 2019, Discord wrote a Rust data structure, called from Elixir, to keep member lists sorted in very large guilds. In 2023 it added &lt;em&gt;relay&lt;/em&gt; processes between guilds and sessions, each handling up to 15,000 sessions, and cut the work for the roughly 90% of connections in large servers that it calls &lt;em&gt;passive&lt;/em&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  The result
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;July 2017:&lt;/strong&gt; nearly 5 million concurrent users and millions of events per second.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;May 2019:&lt;/strong&gt; the Rust post is titled "Using Rust to Scale Elixir for 11 Million Concurrent Users".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;October 2020:&lt;/strong&gt; an Elixir case study, written by Elixir's creator José Valim with quotes from Discord engineers, reports more than 12 million concurrent users, more than 26 million WebSocket events per second, and 400–500 Elixir machines, run by a chat infrastructure team of five engineers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;October 2023:&lt;/strong&gt; the largest single server, Midjourney, had over 10 million members, with over 1 million online.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;The lesson:&lt;/strong&gt; one actor per guild gives you a simple, lock-free owner for each guild's state. The price is that a hot actor's mailbox becomes your bottleneck, so the engineering goes into controlling what reaches it.&lt;/p&gt;

&lt;h2&gt;
  
  
  LinkedIn: one actor per connection, then one per online member
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What they needed
&lt;/h3&gt;

&lt;p&gt;In 2016 LinkedIn launched instant messaging with typing indicators and read receipts. Those need the server to &lt;em&gt;push&lt;/em&gt; data to phones and browsers, not wait to be asked. LinkedIn's engineers chose Server-Sent Events over long-lived HTTP connections, and set a goal of holding hundreds of thousands of them on each machine.&lt;/p&gt;

&lt;p&gt;Then, in 2018, they reused the same platform for &lt;em&gt;presence&lt;/em&gt;: the green dot that shows a contact is online. Their post describes the load as roughly 500 million members, millions of them online at any moment, and thousands going online or offline every second.&lt;/p&gt;

&lt;h3&gt;
  
  
  How actors fit
&lt;/h3&gt;

&lt;p&gt;LinkedIn's servers used the Play framework's integration with Akka, on the JVM.&lt;/p&gt;

&lt;p&gt;For messaging, the 2016 post's design is: assign an Akka actor to each connection the server accepts. The actor owns that connection's lifecycle, and the only way for the rest of the system to reach the client is to send the actor a message.&lt;/p&gt;

&lt;p&gt;For presence, they needed a cheap timer per online member, because "offline" is really the absence of a heartbeat. Their answer is a neat use of actors:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvqo3san1su85u5ljy0pw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvqo3san1su85u5ljy0pw.png" alt="LinkedIn's presence design. While bob's app is connected, a heartbeat arrives every d seconds as a message in bob's actor's mailbox (1). With no live entry in a distributed key-value store, the platform publishes &amp;quot;online&amp;quot;, and each heartbeat refreshes an entry that expires soon after (2). The actor schedules a Publish Offline message to itself, a little later, and each heartbeat resets it (3). When it fires and the entry has expired, it publishes &amp;quot;offline&amp;quot; on bob's presence topic, which reaches alice (4)." width="800" height="514"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The amber loop is the whole trick: an actor sending a message to its own future self is a timer, and a heartbeat resets it.&lt;/em&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;There is &lt;strong&gt;one actor per online member&lt;/strong&gt;, which acts as that member's delayed trigger.&lt;/li&gt;
&lt;li&gt;While the member is connected, a heartbeat arrives every few seconds, and the actor treats it as a message in its mailbox.&lt;/li&gt;
&lt;li&gt;The actor uses Akka's scheduler to send &lt;em&gt;itself&lt;/em&gt; a "Publish Offline" message, timed a little after the next heartbeat is due. Each heartbeat pushes that timer back.&lt;/li&gt;
&lt;li&gt;If the timer fires and the member's entry in a distributed key-value store has expired, the platform publishes an "offline" event on the member's presence topic. Everyone subscribed to that topic, like alice in the diagram, sees the dot change.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Two details from the post are worth copying. First, the shared state lives in the key-value store, not in the actors; each actor holds only what one member needs. Second, LinkedIn notes that the number of live actors equals the number of members online, and that a small number of threads, proportional to the number of cores, runs them all. Actors don't each need a thread.&lt;/p&gt;

&lt;h3&gt;
  
  
  The result
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;October 2016:&lt;/strong&gt; LinkedIn reported holding 100,000 concurrent persistent connections on each node in production. Getting there was mostly operating-system and JVM tuning: they first ran out of memory at about 90,000 connections with an 8 GB heap, found each TLS connection cost about 20 KB, and moved to a 16 GB heap. CPU stayed well under 80%: the limits were memory and operating-system settings, not the actors.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;January 2018:&lt;/strong&gt; presence events reached viewers in under 200 milliseconds at the 99th percentile, and each presence node handled about 1,800 requests per second, scaling out by adding nodes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;The lesson:&lt;/strong&gt; an actor doesn't have to be big to be useful. LinkedIn's presence actor is little more than a timer with a mailbox, and that's exactly why millions of them fit on a few machines.&lt;/p&gt;

&lt;h2&gt;
  
  
  Halo: virtual actors for a game launch
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What they needed
&lt;/h3&gt;

&lt;p&gt;Halo 4 launched on 6 November 2012. Behind the game sat cloud services for player statistics, presence (who is playing, in which game), daily challenges, playlists and cheat detection. Caitie McCaffrey, an engineer at 343 Industries, described the launch in a 2015 talk: Halo 4 went from zero to 1 million users on day one and 4 million in the first week, and had 11.6 million players by June 2013.&lt;/p&gt;

&lt;p&gt;Game traffic is brutal in a specific way. It arrives almost all at once at launch, every player and every match is its own little bundle of state, and the services must answer with low latency while staying up.&lt;/p&gt;

&lt;h3&gt;
  
  
  How actors fit
&lt;/h3&gt;

&lt;p&gt;All of Halo 4's cloud services ran on &lt;strong&gt;Orleans&lt;/strong&gt;, the framework from Microsoft Research that the previous article introduced. Orleans changes one thing about classic actors: you never create or destroy them. Each actor, called a &lt;em&gt;grain&lt;/em&gt;, is identified by an ID, and it always exists &lt;em&gt;virtually&lt;/em&gt;. When code calls a grain that isn't in memory, Orleans activates it on one of the servers in the cluster (called &lt;em&gt;silos&lt;/em&gt;); when a grain sits idle for a while, Orleans removes it from memory. Each activation runs one request at a time on a single thread, so its state needs no locks.&lt;/p&gt;

&lt;p&gt;The 2014 Orleans paper describes Halo 4's Presence service in detail:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd7o2jfy4z8kqq0ojncll.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd7o2jfy4z8kqq0ojncll.png" alt="Halo 4 on Orleans. Consoles send heartbeats to a stateless Router actor, of which Orleans runs as many copies as needed (1). The router calls the Game Session actor by its ID, with no code to find or create it (2). For statistics, the game session sends each player's part to that player's Player actor (3). Calling an ID that is not in memory makes Orleans activate it on some silo, and idle ones are removed (4)." width="800" height="529"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The silos are just where actors happen to be. Nothing in the code says "silo 2": a call by ID finds session 42 wherever it lives, or starts it.&lt;/em&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Every Halo 4 console sends regular heartbeats to Presence. A stateless &lt;strong&gt;Router&lt;/strong&gt; actor takes them. Because it holds no state, Orleans can run several activations of it to spread the load.&lt;/li&gt;
&lt;li&gt;The router forwards each heartbeat to the &lt;strong&gt;Game Session&lt;/strong&gt; actor for that session ID; there's one per session. Its state lives only in memory and is never written to storage, because, as the paper explains, the consoles always hold the real state of the game. If the actor is lost, the next heartbeats rebuild it.&lt;/li&gt;
&lt;li&gt;Every so often, not on every heartbeat, the game session calls the &lt;strong&gt;Player&lt;/strong&gt; actor of each player in it. A Player actor is a meeting point: Halo's companion phone app asks a player's actor which game they're in, then subscribes to that Game Session actor for live updates. The Statistics service uses the same actors: when a game ends, the session sends each player's part of the statistics to that player's Player actor, and those are stored durably, because players hate losing their achievements.&lt;/li&gt;
&lt;li&gt;The paper points out that Halo's developers didn't write any code to find or create these actors. They called them by ID, and Orleans took care of where each one ran.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Notice what is &lt;em&gt;not&lt;/em&gt; here: supervision. Because actors are virtual, the Orleans paper says the framework "has no need for supervision trees" like Erlang's and Akka's. If a server dies, no actor dies with it in any lasting sense: the next call to an actor that lived there creates a new activation on a surviving server. For a game session whose real state lives on the consoles, that's a perfect fit.&lt;/p&gt;

&lt;h3&gt;
  
  
  The result
&lt;/h3&gt;

&lt;p&gt;This is where you have to read carefully. The Orleans paper says most of Halo's production scale and performance figures are confidential, so the famous numbers come from a lab. The researchers ran the actual Halo 4 Presence code on a test cluster:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;25 servers handled about 130,000 heartbeats per second.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Throughput scaled almost linearly&lt;/strong&gt; as they grew the cluster to 125 servers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The median latency was about 6.5 milliseconds&lt;/strong&gt; at up to 19% CPU use.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What we know from production is the launch curve above, and that Microsoft kept using Orleans: Phil Bernstein, a co-author of the paper, said in a 2016 interview that it ran the cloud services for Halo 4 and Halo 5. Microsoft previewed Orleans publicly in April 2014 and released it as open source on GitHub in January 2015. Microsoft's documentation today lists Azure, Xbox, Skype, PlayFab and Gears of War among its users.&lt;/p&gt;

&lt;p&gt;One caution if you read the 2014 paper today: some details have changed since. For example, the paper guaranteed a single activation of each actor only eventually during failures, and the default placement has since moved from random to resource-aware. Don't assume Halo 4 ran on today's Orleans.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The lesson:&lt;/strong&gt; if your actors map to things that always exist conceptually, such as players and matches, letting the runtime handle their lifecycle removes a whole class of code: no spawning, no lookups, no supervisors. You trade that for less control over where and when each actor runs.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the three have in common
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Discord&lt;/th&gt;
&lt;th&gt;LinkedIn&lt;/th&gt;
&lt;th&gt;Halo&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Runtime&lt;/td&gt;
&lt;td&gt;Elixir on the Erlang VM&lt;/td&gt;
&lt;td&gt;Akka on the JVM&lt;/td&gt;
&lt;td&gt;Orleans on .NET&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;An actor is&lt;/td&gt;
&lt;td&gt;a guild (server), a user session&lt;/td&gt;
&lt;td&gt;a connection, an online member&lt;/td&gt;
&lt;td&gt;a router, a game session, a player&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Created by&lt;/td&gt;
&lt;td&gt;the code, explicitly&lt;/td&gt;
&lt;td&gt;the code, explicitly&lt;/td&gt;
&lt;td&gt;nobody: activated on first call&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;When one fails&lt;/td&gt;
&lt;td&gt;contained to that process; load control kept the rest up&lt;/td&gt;
&lt;td&gt;not covered in the posts&lt;/td&gt;
&lt;td&gt;re-activated on the next call; no supervision trees&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reported scale&lt;/td&gt;
&lt;td&gt;~5M concurrent users (2017); 12M+ (2020)&lt;/td&gt;
&lt;td&gt;100K connections per node (2016)&lt;/td&gt;
&lt;td&gt;4M users in launch week (2012); lab: ~130K heartbeats/s on 25 servers&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A few patterns repeat across all three:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One actor per thing in the domain.&lt;/strong&gt; A guild, a connection, a member, a player, a game. The actor is the single owner of that thing's state, so nothing else needs a lock on it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Messages are the only way in.&lt;/strong&gt; That's what made it possible for Discord to batch sends per node, and for LinkedIn to treat a heartbeat as just another message.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hot actors are the real limit.&lt;/strong&gt; An actor handles one message at a time. Discord's big guilds show what happens when one actor gets too much traffic, and every fix there is about controlling what reaches that actor's mailbox.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shared data doesn't belong in an actor's mailbox.&lt;/strong&gt; Discord moved its hash ring to shared read-only storage; LinkedIn kept presence state in a key-value store. Actors own &lt;em&gt;their&lt;/em&gt; state; data that everyone reads belongs somewhere everyone can read it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The failures were operational, not conceptual.&lt;/strong&gt; Memory per connection, mailbox growth, reconnect storms. None of these teams went back to threads and locks.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you're building something where many independent things each have a little state and talk to each other, such as chat rooms, devices, players or shopping carts, these three systems are good evidence that actors scale a long way.&lt;/p&gt;




&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Discord&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Vishnevskiy. "How Discord Scaled Elixir to 5,000,000 Concurrent Users", Discord blog, 6 July 2017. &lt;a href="https://discord.com/blog/how-discord-scaled-elixir-to-5-000-000-concurrent-users" rel="noopener noreferrer"&gt;https://discord.com/blog/how-discord-scaled-elixir-to-5-000-000-concurrent-users&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Nowack. "Using Rust to Scale Elixir for 11 Million Concurrent Users", Discord blog, 17 May 2019. &lt;a href="https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users" rel="noopener noreferrer"&gt;https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Valim. "Real time communication at scale with Elixir at Discord", Elixir blog, 8 October 2020. &lt;a href="https://elixir-lang.org/blog/2020/10/08/real-time-communication-at-scale-with-elixir-at-discord/" rel="noopener noreferrer"&gt;https://elixir-lang.org/blog/2020/10/08/real-time-communication-at-scale-with-elixir-at-discord/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Pisetsky. "Maxjourney: Pushing Discord's Limits with a Million+ Online Users in a Single Server", Discord blog, 25 October 2023. &lt;a href="https://discord.com/blog/maxjourney-pushing-discords-limits-with-a-million-plus-online-users-in-a-single-server" rel="noopener noreferrer"&gt;https://discord.com/blog/maxjourney-pushing-discords-limits-with-a-million-plus-online-users-in-a-single-server&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Krichevsky. "Tracing Discord's Elixir Systems (Without Melting Everything)", Discord blog, 4 March 2026. &lt;a href="https://discord.com/blog/tracing-discords-elixir-systems-without-melting-everything" rel="noopener noreferrer"&gt;https://discord.com/blog/tracing-discords-elixir-systems-without-melting-everything&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Manifold README. &lt;a href="https://github.com/discord/manifold" rel="noopener noreferrer"&gt;https://github.com/discord/manifold&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;LinkedIn&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Gupta, Snyder. "Instant Messaging at LinkedIn: Scaling to Hundreds of Thousands of Persistent Connections on One Machine", LinkedIn Engineering, 24 October 2016. &lt;a href="https://engineering.linkedin.com/blog/2016/10/instant-messaging-at-linkedin--scaling-to-hundreds-of-thousands-" rel="noopener noreferrer"&gt;https://engineering.linkedin.com/blog/2016/10/instant-messaging-at-linkedin--scaling-to-hundreds-of-thousands-&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Gupta, Lay. "Now You See Me, Now You Don't: LinkedIn's Real-Time Presence Platform", LinkedIn Engineering, 12 January 2018. &lt;a href="https://www.linkedin.com/blog/engineering/product-design/now-you-see-me-now-you-dont-linkedins-real-time-presence-platf" rel="noopener noreferrer"&gt;https://www.linkedin.com/blog/engineering/product-design/now-you-see-me-now-you-dont-linkedins-real-time-presence-platf&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Halo and Orleans&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Bernstein, Bykov, Geller, Kliot, Thelin. &lt;em&gt;Orleans: Distributed Virtual Actors for Programmability and Scalability&lt;/em&gt;. Microsoft Research MSR-TR-2014-41, March 2014. &lt;a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Orleans-MSR-TR-2014-41.pdf" rel="noopener noreferrer"&gt;https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Orleans-MSR-TR-2014-41.pdf&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;McCaffrey. "Architecting and Launching the Halo 4 Services", SREcon15, 17 March 2015 (abstract and slides). &lt;a href="https://www.usenix.org/conference/srecon15/program/presentation/mccaffrey" rel="noopener noreferrer"&gt;https://www.usenix.org/conference/srecon15/program/presentation/mccaffrey&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Microsoft News. "Master Chief Returns With Worldwide Launch of 'Halo 4' Beginning Nov. 6", 17 April 2012. &lt;a href="https://news.microsoft.com/source/2012/04/17/master-chief-returns-with-worldwide-launch-of-halo-4-beginning-nov-6/" rel="noopener noreferrer"&gt;https://news.microsoft.com/source/2012/04/17/master-chief-returns-with-worldwide-launch-of-halo-4-beginning-nov-6/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;ODBMS Industry Watch. "Orleans, the technology behind Xbox Halo4 and Halo5. Interview with Phil Bernstein", 9 February 2016. &lt;a href="https://www.odbms.org/blog/2016/02/orleans-the-technology-behind-xbox-halo4-and-halo5-interview-with-phil-bernstein/" rel="noopener noreferrer"&gt;https://www.odbms.org/blog/2016/02/orleans-the-technology-behind-xbox-halo4-and-halo5-interview-with-phil-bernstein/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Microsoft Learn. "Orleans overview". &lt;a href="https://learn.microsoft.com/en-us/dotnet/orleans/overview" rel="noopener noreferrer"&gt;https://learn.microsoft.com/en-us/dotnet/orleans/overview&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Microsoft Learn. "Grain placement". &lt;a href="https://learn.microsoft.com/en-us/dotnet/orleans/grains/grain-placement" rel="noopener noreferrer"&gt;https://learn.microsoft.com/en-us/dotnet/orleans/grains/grain-placement&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;.NET Blog. "Available Now: Preview of Project "Orleans" – Cloud Services at Scale", 2 April 2014. &lt;a href="https://devblogs.microsoft.com/dotnet/available-now-preview-of-project-orleans-cloud-services-at-scale/" rel="noopener noreferrer"&gt;https://devblogs.microsoft.com/dotnet/available-now-preview-of-project-orleans-cloud-services-at-scale/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Microsoft Research. "Orleans - Virtual Actors" project page. &lt;a href="https://www.microsoft.com/en-us/research/project/orleans-virtual-actors/" rel="noopener noreferrer"&gt;https://www.microsoft.com/en-us/research/project/orleans-virtual-actors/&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>distributedsystems</category>
      <category>softwareengineering</category>
      <category>elixir</category>
      <category>scalability</category>
    </item>
    <item>
      <title>The Actor Model: Why It Was Invented and How It Works</title>
      <dc:creator>Hamed Yousefi</dc:creator>
      <pubDate>Mon, 26 Oct 2026 17:15:54 +0000</pubDate>
      <link>https://dev.to/hamed_yousefi/the-actor-model-why-it-was-invented-and-how-it-works-mo1</link>
      <guid>https://dev.to/hamed_yousefi/the-actor-model-why-it-was-invented-and-how-it-works-mo1</guid>
      <description>&lt;p&gt;If you have written concurrent code with threads, you know the routine. Several threads touch the same data, so you put a lock around it. Then you add a second lock, take the two in the wrong order on a busy day, and two threads wait for each other forever. Or you forget the lock once, and a counter is off by one in production every few weeks.&lt;/p&gt;

&lt;p&gt;The actor model starts from a different question: what if no two pieces of code ever shared memory at all?&lt;/p&gt;

&lt;p&gt;This article covers where the idea came from, the problem each step in its history answered, and how an actor works, in pictures. A follow-up article looks at three companies that run actors in production.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem: shared memory and locks
&lt;/h2&gt;

&lt;p&gt;Here is a tiny shop with one item left in stock, written two ways.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff35xjb8rt3ea43q10tgo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ff35xjb8rt3ea43q10tgo.png" alt="Two panels. Left: three threads go through one mutex to a shared stock variable, with warnings that a forgotten lock is a data race, two locks in different orders deadlock, and every lock makes threads wait. Right: alice and bob send Buy messages to a shop actor's mailbox, and only the shop reads and writes its private stock." width="800" height="437"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Left: every thread must remember the lock. Right: only the shop touches &lt;code&gt;stock&lt;/code&gt;, so there is nothing to lock.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;On the left, three threads read and write one &lt;code&gt;stock&lt;/code&gt; variable. Correctness depends on every thread taking the lock, every time, in the same order. The lock also makes threads that want the stock run one after another, which is the opposite of why you added threads. Akka's documentation sums up the second risk in five words: "Locks introduce a new menace: deadlocks."&lt;/p&gt;

&lt;p&gt;On the right, the stock belongs to one &lt;em&gt;actor&lt;/em&gt;, the shop. Nobody else can read or write it. Anyone who wants to buy sends the shop a message, and the shop handles its messages one at a time. There's no shared variable, so there is nothing to lock.&lt;/p&gt;

&lt;p&gt;That is the whole idea. The rest of this article explains how the idea grew up, and what you need around it to build real systems.&lt;/p&gt;
&lt;h2&gt;
  
  
  Where actors came from
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7ztxbf18shgauajqlozf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7ztxbf18shgauajqlozf.png" alt="A timeline from 1973 to 2014: the actor formalism of Hewitt, Bishop and Steiger; Agha's definition of what an actor does; Erlang at Ericsson; the OTP project; Open Source Erlang; Armstrong's thesis on let it crash; Akka on the JVM; Orleans and virtual actors." width="800" height="368"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Navy: the theory. Violet: Erlang and OTP, where supervision comes from. Blue and amber: the model reaching the JVM and .NET.&lt;/em&gt;&lt;/p&gt;
&lt;h3&gt;
  
  
  1973: a theory with one kind of object
&lt;/h3&gt;

&lt;p&gt;In August 1973, Carl Hewitt, Peter Bishop and Richard Steiger presented &lt;em&gt;A Universal Modular ACTOR Formalism for Artificial Intelligence&lt;/em&gt; at the third International Joint Conference on Artificial Intelligence (IJCAI). Their proposal was radical in its simplicity: everything is an actor, and everything an actor does is send messages to other actors.&lt;/p&gt;

&lt;p&gt;The problem they were answering was how to describe computation that happens in many places at once. The usual tools of the time were goto, interrupts and shared variables guarded by semaphores. The paper says its formalism "does not have or need goto, interrupt, or semaphore primitives". Message passing replaces all of them.&lt;/p&gt;
&lt;h3&gt;
  
  
  1985: what an actor does with a message
&lt;/h3&gt;

&lt;p&gt;The 1973 paper was broad. Gul Agha's 1985 MIT technical report, &lt;em&gt;ACTORS: A Model of Concurrent Computation in Distributed Systems&lt;/em&gt; (a book from MIT Press in 1986), pinned the model down. When an actor receives a message, it responds with three things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;a set of messages it sends to other actors,&lt;/li&gt;
&lt;li&gt;a set of new actors it creates,&lt;/li&gt;
&lt;li&gt;a &lt;em&gt;replacement behavior&lt;/em&gt;, which decides how it handles the next message.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Every actor has a mail address, and an actor can only send to addresses it knows. That is the definition most actor systems still follow. Hewitt's later papers state the same three actions and add a point worth remembering: the model itself does not require a mailbox or a thread. Those are implementation choices, though nearly every practical system adds a mailbox.&lt;/p&gt;
&lt;h3&gt;
  
  
  1986: Erlang reaches the same design by another road
&lt;/h3&gt;

&lt;p&gt;At the Ericsson Computer Science Laboratory, Joe Armstrong, Robert Virding and Mike Williams started work on a new language in 1986. Armstrong's history of the language gives its goal: "a better way of programming telephony applications". A telephone switch handles a huge number of calls at once, must keep running for years, and can't stop because one call hit a bug.&lt;/p&gt;

&lt;p&gt;Erlang's answer was many small &lt;em&gt;processes&lt;/em&gt; that share no memory and communicate only by asynchronous messages. That is an actor in everything but name, and the name was missing on purpose. On a 2019 panel with Hewitt and Tony Hoare, Armstrong said: "I didn't know about the actor stuff at the time". In a 2014 mailing-list post he named Prolog and Smalltalk as Erlang's influences instead. Two groups, a theorist and a telecom lab, arrived at the same design from different problems.&lt;/p&gt;
&lt;h3&gt;
  
  
  1996 to 2003: supervision and "let it crash"
&lt;/h3&gt;

&lt;p&gt;Ericsson started the Open Telecom Platform (OTP) project in 1996: a set of Erlang libraries for building telecom systems. It released Erlang and OTP as open source in December 1998.&lt;/p&gt;

&lt;p&gt;Armstrong's 2003 PhD thesis, &lt;em&gt;Making reliable distributed systems in the presence of software errors&lt;/em&gt;, explains the error-handling style behind OTP. Its slogan is "Let it crash." Instead of writing defensive code for every possible failure, a process that hits an unexpected error simply dies, and a separate process, its &lt;em&gt;supervisor&lt;/em&gt;, recovers. Supervisors watch workers and other supervisors, so a whole system becomes a tree of them. We'll come back to this below.&lt;/p&gt;

&lt;p&gt;The thesis also reports on Ericsson's AXD301 ATM switch, which had over 1.7 million lines of Erlang in 2003. You may have seen the famous "nine nines" (99.9999999%) availability figure for it. The thesis does quote a customer's figure of that size, and it also notes that how the figure was obtained was not documented. Treat it as a story about a very reliable system, not a measurement.&lt;/p&gt;
&lt;h3&gt;
  
  
  2009: actors on the JVM
&lt;/h3&gt;

&lt;p&gt;Jonas Bonér started Akka in 2009 (its first commit is from February that year). Akka brought actors, mailboxes and supervision to the Java Virtual Machine, with Scala and Java APIs, so JVM teams could use the model without leaving their platform. Akka's documentation still opens with the problem it answers: the costs of locks and shared memory on modern CPUs.&lt;/p&gt;
&lt;h3&gt;
  
  
  2014: virtual actors
&lt;/h3&gt;

&lt;p&gt;In 2014, Microsoft Research described Orleans, a .NET framework built around &lt;em&gt;virtual actors&lt;/em&gt;. In classic actor systems you create an actor, keep its address, and stop it when you are done. Orleans actors are, in the paper's words, "purely logical entities that always exist, virtually". You never create or destroy them: you call an actor by its ID, and the runtime activates it on some server if it isn't already running. The problem it answered was cloud services with millions of objects, such as players, where managing each actor's lifecycle by hand is a burden. The second article shows Orleans running a real game backend.&lt;/p&gt;
&lt;h2&gt;
  
  
  How an actor works
&lt;/h2&gt;

&lt;p&gt;With the history in place, here is the model in one picture.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Floqy2as74ep0348udr1g.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Floqy2as74ep0348udr1g.png" alt="An actor with its address, counter@node-1, a mailbox holding messages, and a behavior, receive(msg), with private state total: 6. Three arrows leave the behavior: send messages to actors whose addresses it has, create actors and get their addresses, and become a new behavior for the next message." width="800" height="406"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;An actor is an address, a mailbox and a behavior with private state. For each message it can do only the three things on the right.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;An actor has three parts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;An address.&lt;/strong&gt; You can send a message only to an actor whose address you have. You get one by creating the actor, or by receiving the address in a message. The address is how the rest of the system refers to the actor; nobody holds a pointer to its insides.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A mailbox.&lt;/strong&gt; Messages wait here until the actor takes them. Sending is asynchronous: the sender drops the message off and carries on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A behavior with private state.&lt;/strong&gt; The actor takes one message at a time, runs its behavior to completion, and only then takes the next. Only the actor reads or writes its state.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For each message, the behavior can do three things: &lt;em&gt;send&lt;/em&gt; messages to addresses it knows, &lt;em&gt;spawn&lt;/em&gt; new actors, and &lt;em&gt;become&lt;/em&gt; a new behavior for the next message. Agha called the last one a replacement behavior; Akka's classic API calls it &lt;code&gt;become&lt;/code&gt;. In practice, "become" is often just "update my own state", like setting &lt;code&gt;total&lt;/code&gt; from 6 to 7.&lt;/p&gt;

&lt;p&gt;Here is a counter actor in pseudocode:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;actor Counter(total = 0):
    receive Add(n):
        total = total + n            # only this actor ever touches total
    receive Get(replyTo):
        send(replyTo, Total(total))  # answer with a message, not a return value
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notice what's missing: no lock, no &lt;code&gt;synchronized&lt;/code&gt;, no atomic. &lt;code&gt;total&lt;/code&gt; can't be corrupted by two threads, because only one message is ever being handled, and only this actor can see &lt;code&gt;total&lt;/code&gt;. Akka's docs make the same point: because at most one message is processed per actor at a time, the actor's invariants hold without synchronization.&lt;/p&gt;

&lt;p&gt;Also notice &lt;code&gt;Get&lt;/code&gt;. There is no return value, because there is no call stack shared between the two actors. The asker includes its own address, and the counter replies with a message.&lt;/p&gt;

&lt;h2&gt;
  
  
  One message at a time
&lt;/h2&gt;

&lt;p&gt;The mailbox is what turns concurrent senders into sequential processing. Here, three people use one bank account at the same moment:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbbbkv7l29m52k01lhoei.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbbbkv7l29m52k01lhoei.gif" alt="Animation: alice, bob and carol send Deposit 50, Withdraw 30 and Withdraw 130 to an account actor at the same moment. The messages queue in the mailbox in arrival order. Starting from 100, the account pays bob and leaves 70, adds alice's deposit for 120, and declines carol because 120 is less than 130." width="799" height="473"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Watch the balance box: it changes only when the account takes the next message from its mailbox, never while another message is in progress.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;alice, bob and carol run concurrently, but the account handles their messages strictly one after another. It needs no lock on &lt;code&gt;balance&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The last frame of the animation is the honest part. The &lt;em&gt;order&lt;/em&gt; in which messages from different senders arrive is not fixed. In this run carol is declined; in another, bob would be. Actors remove data races, but they don't remove race conditions in your business logic. If order matters, you have to design for it, for example by putting the decision in one actor that sees all the requests, or by making the protocol explicit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Location transparency
&lt;/h2&gt;

&lt;p&gt;Because an actor is known only by its address, the sender's code doesn't depend on where the receiver runs.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F394qocmvswnk85p5sh24.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F394qocmvswnk85p5sh24.png" alt="Two nodes. On node-1 a checkout actor sends AddItem to a cart actor on the same node and Reserve to an inventory actor on node-2, with the same send call. The local message goes straight into the cart's mailbox; the remote one is serialized, crosses the network and lands in the inventory's mailbox." width="799" height="452"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Path 1 stays in memory. Path 2 is serialized and crosses the network. The &lt;code&gt;send&lt;/code&gt; call looks the same for both.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Akka calls this &lt;em&gt;location transparency&lt;/em&gt;. Its documentation says its functions work the same in a single JVM or on a cluster, because every interaction between actors is an asynchronous message anyway.&lt;/p&gt;

&lt;p&gt;This is what lets an actor system grow from one machine to many without rewriting the code that sends messages. The warning icon in the picture is the catch. The code is the same, but the physics isn't: a remote message is slower and can be lost. Well-designed actor systems treat every message as one that might not arrive, and use timeouts and retries where it matters.&lt;/p&gt;

&lt;h2&gt;
  
  
  Let it crash: supervision trees
&lt;/h2&gt;

&lt;p&gt;Actors that create other actors form a tree. In Erlang/OTP and Akka, each parent &lt;em&gt;supervises&lt;/em&gt; its children: when a child fails, the parent decides what happens.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjy9a6pxup9y1r9m9y2vx.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjy9a6pxup9y1r9m9y2vx.gif" alt="Animation of a supervision tree: a root supervisor, a supervisor named payments with the one_for_one strategy, and three workers. A bad message makes worker-2 crash. The supervisor gets the exit, its strategy picks restart, worker-2's old state is thrown away and it starts again with fresh state, while worker-1 and worker-3 keep counting." width="800" height="507"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Follow worker-2's count: 41 before the crash, 0 after the restart. Its siblings never stop counting.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The numbered steps in the animation are the whole recovery:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A bad message makes &lt;code&gt;worker-2&lt;/code&gt; crash. It doesn't try to recover in place.&lt;/li&gt;
&lt;li&gt;Its supervisor, &lt;code&gt;payments&lt;/code&gt;, is told that the child exited.&lt;/li&gt;
&lt;li&gt;The supervisor's &lt;em&gt;strategy&lt;/em&gt; picks what to do. Here it's OTP's &lt;code&gt;one_for_one&lt;/code&gt;: restart only the failed child.&lt;/li&gt;
&lt;li&gt;The worker's old, possibly corrupt, state is thrown away.&lt;/li&gt;
&lt;li&gt;It starts again from a known good state.&lt;/li&gt;
&lt;li&gt;New messages reach the restarted worker, and the siblings never noticed.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;OTP's supervisor documentation defines other strategies too. &lt;code&gt;one_for_all&lt;/code&gt; restarts every child when one fails, for children that only make sense together. And restarts are bounded: if a child needs more than a set number of restarts within a set number of seconds (by default, more than 1 restart in 5 seconds), the supervisor gives up, stops its children and itself, and its own parent decides what to do next. A local problem that a restart can't fix gets escalated up the tree.&lt;/p&gt;

&lt;p&gt;Why is crashing a good idea? Because most production bugs are rare combinations of state and input. Code that tries to handle every unexpected case gets long and usually wrong. A fresh start from known state fixes the common cases, keeps the error-handling code in one place (the supervisor), and keeps the failure contained to one small actor instead of the whole process.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actors buy you, compared with locks
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Shared memory and locks&lt;/th&gt;
&lt;th&gt;Actors&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Who touches the data&lt;/td&gt;
&lt;td&gt;Any thread with a reference&lt;/td&gt;
&lt;td&gt;Only the owning actor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data races&lt;/td&gt;
&lt;td&gt;Possible whenever a lock is missed&lt;/td&gt;
&lt;td&gt;Impossible on actor state&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Deadlocks from lock ordering&lt;/td&gt;
&lt;td&gt;Possible&lt;/td&gt;
&lt;td&gt;No locks to order&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failure&lt;/td&gt;
&lt;td&gt;An exception unwinds one thread's stack, often far from anyone who can fix it&lt;/td&gt;
&lt;td&gt;The actor dies and its supervisor decides&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scaling out&lt;/td&gt;
&lt;td&gt;A rewrite: shared memory doesn't cross machines&lt;/td&gt;
&lt;td&gt;The same &lt;code&gt;send&lt;/code&gt;, to an address on another node&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Actors are not free, and it's worth being clear about the costs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ordering still matters.&lt;/strong&gt; As the account example showed, messages from different senders can arrive in any order.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mailboxes can fill up.&lt;/strong&gt; If an actor receives messages faster than it handles them, its mailbox grows. Production systems need back-pressure, bounded mailboxes, or both. The second article shows a real case of exactly this.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two actors can still wait on each other.&lt;/strong&gt; If A waits for a reply from B while B waits for a reply from A, neither progresses. Timeouts help; so does designing protocols that don't need round trips.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Debugging is different.&lt;/strong&gt; A bug is a sequence of messages, not a stack trace, so good tracing and logging matter more.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  When to reach for actors
&lt;/h2&gt;

&lt;p&gt;Actors fit best when your problem already looks like many independent things with their own state that talk to each other: users, sessions, devices, game players, chat rooms, orders. Each of those becomes an actor, and the runtime spreads them over your cores and machines.&lt;/p&gt;

&lt;p&gt;They fit less well for bulk number crunching over shared arrays, or for simple request-response services that keep no state between requests.&lt;/p&gt;

&lt;p&gt;The model is fifty years old, and it has outlived many of the languages that first used it, because the problem it answers hasn't changed. Concurrent programs are hard when everything can touch everything. Actors make each piece of state belong to exactly one owner, and they turn failure into something a supervisor can deal with.&lt;/p&gt;

&lt;p&gt;In the next article, we'll look at how Discord, LinkedIn and the team behind Halo used actors to serve millions of users.&lt;/p&gt;




&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Hewitt, Bishop, Steiger. &lt;em&gt;A Universal Modular ACTOR Formalism for Artificial Intelligence&lt;/em&gt;. IJCAI 1973, pp. 235–244. &lt;a href="https://www.ijcai.org/Proceedings/73/Papers/027B.pdf" rel="noopener noreferrer"&gt;https://www.ijcai.org/Proceedings/73/Papers/027B.pdf&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Agha. &lt;em&gt;ACTORS: A Model of Concurrent Computation in Distributed Systems&lt;/em&gt;. MIT AI Lab Technical Report 844, June 1985. &lt;a href="https://dspace.mit.edu/handle/1721.1/6952" rel="noopener noreferrer"&gt;https://dspace.mit.edu/handle/1721.1/6952&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Hewitt. &lt;em&gt;Actor Model of Computation: Scalable Robust Information Systems&lt;/em&gt;. arXiv:1008.1459. &lt;a href="https://arxiv.org/abs/1008.1459" rel="noopener noreferrer"&gt;https://arxiv.org/abs/1008.1459&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Armstrong. &lt;em&gt;A History of Erlang&lt;/em&gt;. HOPL III, 2007. &lt;a href="https://doi.org/10.1145/1238844.1238850" rel="noopener noreferrer"&gt;https://doi.org/10.1145/1238844.1238850&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Erlang FAQ, "Academic and Historical Questions". &lt;a href="https://www.erlang.org/faq/academic.html" rel="noopener noreferrer"&gt;https://www.erlang.org/faq/academic.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Erlang Solutions. &lt;em&gt;Let's #TalkConcurrency panel discussion with Sir Tony Hoare, Joe Armstrong and Carl Hewitt&lt;/em&gt;, 2019. &lt;a href="https://www.erlang-solutions.com/blog/lets-talkconcurrency-panel-discussion-with-sir-tony-hoare-joe-armstrong-and-carl-hewitt/" rel="noopener noreferrer"&gt;https://www.erlang-solutions.com/blog/lets-talkconcurrency-panel-discussion-with-sir-tony-hoare-joe-armstrong-and-carl-hewitt/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Armstrong. "Erlang is &lt;em&gt;not&lt;/em&gt; a implementation of the Actor model", erlang-questions mailing list, 25 June 2014. &lt;a href="http://erlang.org/pipermail/erlang-questions/2014-June/079891.html" rel="noopener noreferrer"&gt;http://erlang.org/pipermail/erlang-questions/2014-June/079891.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Armstrong. &lt;em&gt;Making reliable distributed systems in the presence of software errors&lt;/em&gt;. PhD thesis, KTH, December 2003. &lt;a href="https://erlang.org/download/armstrong_thesis_2003.pdf" rel="noopener noreferrer"&gt;https://erlang.org/download/armstrong_thesis_2003.pdf&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Erlang/OTP documentation, "Supervisor Behaviour". &lt;a href="https://www.erlang.org/doc/system/sup_princ.html" rel="noopener noreferrer"&gt;https://www.erlang.org/doc/system/sup_princ.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Akka documentation, "Why modern systems need a new programming model". &lt;a href="https://doc.akka.io/libraries/akka-core/current/typed/guide/actors-motivation.html" rel="noopener noreferrer"&gt;https://doc.akka.io/libraries/akka-core/current/typed/guide/actors-motivation.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Akka documentation, "How the Actor Model Meets the Needs of Modern, Distributed Systems". &lt;a href="https://doc.akka.io/libraries/akka-core/current/typed/guide/actors-intro.html" rel="noopener noreferrer"&gt;https://doc.akka.io/libraries/akka-core/current/typed/guide/actors-intro.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Akka documentation, "Location Transparency". &lt;a href="https://doc.akka.io/libraries/akka-core/current/general/remoting.html" rel="noopener noreferrer"&gt;https://doc.akka.io/libraries/akka-core/current/general/remoting.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Akka documentation, "Classic Actors" (Become/Unbecome). &lt;a href="https://doc.akka.io/libraries/akka-core/current/actors.html" rel="noopener noreferrer"&gt;https://doc.akka.io/libraries/akka-core/current/actors.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Akka's first commit, 16 February 2009. &lt;a href="https://github.com/akka/akka-core/commit/0a31ad718883ef0314de6dc8c6c2c9699eadfdae" rel="noopener noreferrer"&gt;https://github.com/akka/akka-core/commit/0a31ad718883ef0314de6dc8c6c2c9699eadfdae&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Bernstein, Bykov, Geller, Kliot, Thelin. &lt;em&gt;Orleans: Distributed Virtual Actors for Programmability and Scalability&lt;/em&gt;. Microsoft Research MSR-TR-2014-41, March 2014. &lt;a href="https://www.microsoft.com/en-us/research/publication/orleans-distributed-virtual-actors-for-programmability-and-scalability/" rel="noopener noreferrer"&gt;https://www.microsoft.com/en-us/research/publication/orleans-distributed-virtual-actors-for-programmability-and-scalability/&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>distributedsystems</category>
      <category>systemdesign</category>
      <category>elixir</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Cutting etcd/raft failover from 1.25 s to 343 ms, checked against 2,000 simulated leader crashes</title>
      <dc:creator>Hamed Yousefi</dc:creator>
      <pubDate>Mon, 19 Oct 2026 12:50:09 +0000</pubDate>
      <link>https://dev.to/hamed_yousefi/cutting-etcdraft-failover-from-125-s-to-343-ms-checked-against-2000-simulated-leader-crashes-4755</link>
      <guid>https://dev.to/hamed_yousefi/cutting-etcdraft-failover-from-125-s-to-343-ms-checked-against-2000-simulated-leader-crashes-4755</guid>
      <description>&lt;p&gt;When the leader of a three-node etcd/raft cluster dies, the other two servers wait between 1 and 1.9 seconds before one of them holds an election. That's with a tick every 100 ms and the settings from etcd/raft's README: &lt;code&gt;ElectionTick&lt;/code&gt; 10 and &lt;code&gt;HeartbeatTick&lt;/code&gt; 1. Until a new leader is in place, no write commits. I wanted that gap under a second.&lt;/p&gt;

&lt;p&gt;Lowering &lt;code&gt;ElectionTick&lt;/code&gt; is a one-line change. What I didn't know was what it does once servers crash and links drop. On a test cluster I could kill leaders and time the gap, but every attempt sees different timing. When one is slow, there's no way to run it again and see why.&lt;/p&gt;

&lt;p&gt;So I tried it with faultline, the deterministic simulation testing tool for Go I've been building. Version 0.1.1 came out today. A faultline test runs a distributed system in virtual time on a simulated network and disk. Every random choice in the simulation comes from a seed, so a run replays exactly.&lt;/p&gt;

&lt;p&gt;faultline includes a harness for etcd/raft v3.7.0. It runs a cluster of &lt;code&gt;RawNode&lt;/code&gt;s with a write-ahead log on the simulated disk and clients that write and read keys, and checks raft's safety properties after every event. It tests the raft library with your settings. The key-value store, the log and the transport on top are the harness's own, not your service's.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is a shorter timeout still safe?
&lt;/h2&gt;

&lt;p&gt;I started with safety. The first test runs 500 seeds with elections after 3 ticks instead of 10. Each seed runs 60 seconds of virtual time under partitions, an isolated leader, cut and lossy links, crashes, pauses, failed disk syncs and clock drift. The last 10 seconds are quiet, and the cluster must make progress again.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;TestFastElectionsStaySafe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;cfg&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;etcdraft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DefaultConfig&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ElectionTick&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt; &lt;span class="c"&gt;// elections after 3 to 5 ticks of 100 ms, instead of 10 to 19&lt;/span&gt;
    &lt;span class="n"&gt;etcdraft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;faultline&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Options&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;Seeds&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="m"&gt;500&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ go test -run '^TestFastElectionsStaySafe$' .
ok      example.com/failover    37.110s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No seed broke a safety check or left the cluster stuck. That makes the change safe in these runs. It says nothing yet about how fast failover got.&lt;/p&gt;

&lt;h2&gt;
  
  
  How long does failover take?
&lt;/h2&gt;

&lt;p&gt;Then I timed it. The next test crashes whichever server leads at 5 seconds, with no other faults, and checks the time until another server becomes leader. &lt;code&gt;etcdraft.Setup&lt;/code&gt; builds the cluster inside &lt;code&gt;faultline.Run&lt;/code&gt;, and &lt;code&gt;LeaderChanges&lt;/code&gt; lists every change of leadership in the run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="c"&gt;// failover crashes whichever server leads at 5s. check gets the time until the next leader and&lt;/span&gt;
&lt;span class="c"&gt;// the number of elections it took.&lt;/span&gt;
&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;failover&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt; &lt;span class="n"&gt;etcdraft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Config&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;seeds&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;check&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;uint64&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Faults&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;etcdraft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FaultsNone&lt;/span&gt; &lt;span class="c"&gt;// no faults except the crash below&lt;/span&gt;
    &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;20&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;
    &lt;span class="n"&gt;opts&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;faultline&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Options&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;Seeds&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;seeds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Net&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;etcdraft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NetConfig&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;
    &lt;span class="n"&gt;faultline&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;faultline&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;World&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;etcdraft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Setup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;crash&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;kernel&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Plan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fault&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Script&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fault&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;At&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;crash&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Kind&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fault&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;KindCrash&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Role&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"leader"&lt;/span&gt;&lt;span class="p"&gt;}))&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Final&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"new leader after the crash"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;oldTerm&lt;/span&gt; &lt;span class="kt"&gt;uint64&lt;/span&gt;
            &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ch&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;LeaderChanges&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;ch&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;State&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;raft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StateLeader&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="k"&gt;continue&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;ch&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;At&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;crash&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="n"&gt;oldTerm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ch&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Term&lt;/span&gt;
                    &lt;span class="k"&gt;continue&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ch&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;At&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;crash&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;ch&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Term&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;oldTerm&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"no new leader after the crash"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;TestFailoverUnderOneSecond&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;cfg&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;etcdraft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DefaultConfig&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ElectionTick&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt;
    &lt;span class="n"&gt;failover&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="kt"&gt;uint64&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"new leader %v after the crash"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It failed on the 45th seed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ go test -run '^TestFailoverUnderOneSecond$' .
--- FAIL: TestFailoverUnderOneSecond (1.14s)
    failover_test.go:30: faultline: running 200 seeds from base 0xf41bc040834225b9 (test name)
    --- FAIL: TestFailoverUnderOneSecond/seed=0xe05bdf977f330e42 (0.19s)
        faultline: final check "new leader after the crash" failed after the run at t=20.000000000s
          new leader 1.783042452s after the crash
        replay:    FAULTLINE_SEED=0xe05bdf977f330e42 go test -v -run '^TestFailoverUnderOneSecond$' example.com/failover
        artifacts: /tmp/faultline-501/example.com_failover/TestFailoverUnderOneSecond/e05bdf977f330e42/
    failover_test.go:30: faultline: stopping after failing seed 0xe05bdf977f330e42; 155 of 200 seeds not run (set Options.KeepGoing to run all)
FAIL
FAIL    example.com/failover    1.678s
FAIL
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Reading the slow seed
&lt;/h2&gt;

&lt;p&gt;This time I could run the slow one again as often as I liked. The replay command runs the identical run, with the same events at the same virtual times. The failing seed also left an artifact directory, and its text timeline shows what happened after the crash:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-E&lt;/span&gt; &lt;span class="s2"&gt;"fault.crash|became (candidate|leader)"&lt;/span&gt; timeline.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;    254  0.220702122s n3#1    etcdraft.raft_log    3 became candidate at term 2 level=info &amp;lt;-253
    300  0.225595569s n3#1    etcdraft.raft_log    3 became leader at term 2 level=info &amp;lt;-299
  18833  5.000000000s n3#1    fault.crash          crash n3 id=1 source=script node=n3 effect=applied &amp;lt;-18832
  18905  5.473915956s n2#1    etcdraft.raft_log    2 became candidate at term 3 level=info &amp;lt;-18904
  18916  5.474188777s n1#1    etcdraft.raft_log    1 became candidate at term 3 level=info &amp;lt;-18915
  19083  5.873566634s n1#1    etcdraft.raft_log    1 became candidate at term 4 level=info &amp;lt;-19082
  19094  5.874618666s n2#1    etcdraft.raft_log    2 became candidate at term 4 level=info &amp;lt;-19093
  19271  6.374222166s n1#1    etcdraft.raft_log    1 became candidate at term 5 level=info &amp;lt;-19270
  19290  6.375359195s n2#1    etcdraft.raft_log    2 became candidate at term 5 level=info &amp;lt;-19289
  19457  6.774700502s n1#1    etcdraft.raft_log    1 became candidate at term 6 level=info &amp;lt;-19456
  19505  6.783042452s n1#1    etcdraft.raft_log    1 became leader at term 6 level=info &amp;lt;-19504
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;n1 and n2 became candidates within about a millisecond of each other, three rounds in a row. Each voted for itself, so neither got the second vote it needed, and both waited out another timeout. In the fourth round only n1 ran, and it won.&lt;/p&gt;

&lt;p&gt;etcd/raft picks each server's timeout in whole ticks: &lt;a href="https://github.com/etcd-io/raft/blob/v3.7.0/raft.go#L2053-L2055" rel="noopener noreferrer"&gt;&lt;code&gt;electionTimeout + rand.Intn(electionTimeout)&lt;/code&gt;&lt;/a&gt;, drawn from &lt;code&gt;crypto/rand&lt;/code&gt;, which faultline seeds for every run. With &lt;code&gt;ElectionTick&lt;/code&gt; 3, a server waits 3, 4 or 5 ticks. The harness starts each server's ticks at a random point within the first interval, as real servers that started at different times would. In this seed, n1's and n2's ticks landed less than half a millisecond apart, and they kept drawing the same number of ticks.&lt;/p&gt;

&lt;h2&gt;
  
  
  How often does it happen?
&lt;/h2&gt;

&lt;p&gt;One slow seed doesn't give a rate. The next test runs 2,000 crashes per setting and records the time to the next leader and the number of election rounds it took. More than one round means at least one round elected nobody, as in the seed above:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;TestFailoverDistribution&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;name&lt;/span&gt;  &lt;span class="kt"&gt;string&lt;/span&gt;
        &lt;span class="n"&gt;tick&lt;/span&gt;  &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;
        &lt;span class="n"&gt;ticks&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="p"&gt;}{&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;"10 ticks of 100ms"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;10&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;"3 ticks of 100ms"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;"10 ticks of 30ms"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;30&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;10&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;took&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;
        &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;repeated&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;slow&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;
        &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;cfg&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;etcdraft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DefaultConfig&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
            &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TickInterval&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ElectionTick&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tick&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ticks&lt;/span&gt;
            &lt;span class="n"&gt;failover&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;elections&lt;/span&gt; &lt;span class="kt"&gt;uint64&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;took&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;took&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;elections&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="n"&gt;repeated&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt; &lt;span class="c"&gt;// at least one election round elected nobody&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="n"&gt;slow&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
            &lt;span class="p"&gt;})&lt;/span&gt;
        &lt;span class="p"&gt;})&lt;/span&gt;
        &lt;span class="n"&gt;slices&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Sort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;took&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;took&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;ms&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Logf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"%-17s median %v, p99 %v, max %v; repeated elections in %d, over 1s in %d of %d"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ms&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;took&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="n"&gt;ms&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;took&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="m"&gt;99&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="m"&gt;100&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="n"&gt;ms&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;took&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="n"&gt;repeated&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;slow&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ go test -v -run '^TestFailoverDistribution$' . | grep repeated
    failover_test.go:91: 10 ticks of 100ms median 1.249s, p99 1.893s, max 3.155s; repeated elections in 16, over 1s in 1797 of 2000
    failover_test.go:91: 3 ticks of 100ms  median 343ms, p99 622ms, max 1.177s; repeated elections in 27, over 1s in 1 of 2000
    failover_test.go:91: 10 ticks of 30ms  median 381ms, p99 814ms, max 1.418s; repeated elections in 32, over 1s in 7 of 2000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With 3 ticks, half the failovers finished within 343 ms and 99% within 622 ms, against 1.249 s and 1.893 s with the default. One failover in 2,000 took more than a second. About 1.4% needed more than one round.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fewer ticks or shorter ticks?
&lt;/h2&gt;

&lt;p&gt;The third line is the other way to reach a 300 ms timeout: keep 10 ticks and make each one 30 ms. I expected it to repeat elections less often, because a server then picks from 10 timeouts instead of 3. It didn't. Its median was close, and its tail was worse: 7 failovers over a second, and a p99 of 814 ms.&lt;/p&gt;

&lt;p&gt;Three more settings, 2,000 crashes each, point at the reason. Settings whose shortest timeout is 300 ms repeated an election in 24 to 32 of 2,000 failovers, whether they used 3, 10 or 20 ticks. Settings that start at 600 ms or later repeated one in 14 to 20. So the timeout in milliseconds, not the number of ticks, seems to set how often two servers start an election together.&lt;/p&gt;

&lt;p&gt;30 ms ticks also cost something that 3 ticks don't. With &lt;code&gt;HeartbeatTick&lt;/code&gt; 1, the leader sends a heartbeat every tick, so 30 ms ticks send heartbeats more than three times as often. The same settings passed 500 seeds of the full fault mix, like the 3-tick version:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;TestShortTicksStaySafe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;cfg&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;etcdraft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DefaultConfig&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TickInterval&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;30&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;
    &lt;span class="n"&gt;etcdraft&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;faultline&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Options&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;Seeds&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="m"&gt;500&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ go test -run '^TestShortTicksStaySafe$' .
ok      example.com/failover    43.569s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What a shorter timeout costs
&lt;/h2&gt;

&lt;p&gt;A shorter timeout reacts faster to every missed heartbeat, not only to a dead leader. I counted the elections in 200 runs of 60 seconds for each setting. With no faults, all three settings held exactly one election per run, the first. Under the default faults, the default setting held 8.95 per run, and the two 300 ms settings 12.7 and 12.4. Each election is a short window with no leader. Some of the extra ones are faster reactions to real failures; others are elections a longer timeout would have ridden out, and these runs don't separate the two.&lt;/p&gt;

&lt;p&gt;etcd/raft's documentation suggests &lt;code&gt;ElectionTick = 10 * HeartbeatTick&lt;/code&gt; "to avoid unnecessary leader switching". In these runs, the 3:1 setting and the 10:1 setting held the same number of extra elections, so the timeout in milliseconds mattered and the ratio didn't. On a network that loses messages in bursts, the ratio may still matter. The harness doesn't model that kind of loss.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this doesn't tell you
&lt;/h2&gt;

&lt;p&gt;The harness runs etcd/raft with its own state machine, log and transport. Your service's code is not in the run. To test it, you port it to faultline's node API, and the harness is one worked example of such a port.&lt;/p&gt;

&lt;p&gt;The network is a model with 1 ms of latency and up to 2 ms of jitter. One message in 100 is delayed by up to 50 ms more, one in 1,000 is lost, and one in 1,000 is duplicated. A real network with longer stalls, or a process with long garbage collection pauses, needs more headroom than these runs show. Inside the simulation a callback takes no time, so CPU load doesn't slow the cluster down.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;Every test above runs with &lt;code&gt;go test&lt;/code&gt;, against the released modules. All four took under four minutes on my laptop, an Apple M4 Max with Go 1.27.1:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go get github.com/hmdsefi/faultline/harness/etcdraft@v0.1.1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I first ran these tests on v0.1.0, and v0.1.1 gives the same numbers. v0.1.1 also adds &lt;code&gt;faultline.RunSeeds&lt;/code&gt;, which keeps a failing seed like the slow one above as a regression test.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/hmdsefi/faultline/blob/v0.1.1/docs/guides/etcd-raft.md" rel="noopener noreferrer"&gt;Test etcd/raft with the harness&lt;/a&gt; covers the rest of the harness: cluster sizes, fault presets, snapshots and membership changes. The &lt;a href="https://github.com/hmdsefi/faultline/tree/v0.1.1/docs" rel="noopener noreferrer"&gt;documentation&lt;/a&gt; starts with how an AI coding agent runs the same loop, from a failing seed to a pinned regression test, and the &lt;a href="https://github.com/hmdsefi/faultline" rel="noopener noreferrer"&gt;repository&lt;/a&gt; is on GitHub.&lt;/p&gt;

</description>
      <category>go</category>
      <category>testing</category>
      <category>raft</category>
      <category>distributedsystems</category>
    </item>
    <item>
      <title>A Go test that passed 40,000 times, then lost a write under load</title>
      <dc:creator>Hamed Yousefi</dc:creator>
      <pubDate>Tue, 13 Oct 2026 16:51:28 +0000</pubDate>
      <link>https://dev.to/hamed_yousefi/a-go-test-that-passed-40000-times-then-lost-a-write-under-load-h6l</link>
      <guid>https://dev.to/hamed_yousefi/a-go-test-that-passed-40000-times-then-lost-a-write-under-load-h6l</guid>
      <description>&lt;p&gt;Say a client writes a value and gets a success back. Later, the value is gone. It happens once in production, around the time a node restarts, and every test is green. A week later a CI job fails on a test nobody touched. The re-run passes, so someone adds a retry and moves on.&lt;/p&gt;

&lt;p&gt;Bugs like this need a rare timing, such as a partition during a leader election, or a crash after a write was acknowledged but before it reached the disk. Production hits that timing eventually, because it runs all day on busy machines.&lt;/p&gt;

&lt;h2&gt;
  
  
  A test that passes 40,000 times
&lt;/h2&gt;

&lt;p&gt;Here is a write path cut down to one test, which you can save as &lt;code&gt;wal_test.go&lt;/code&gt; in a new module (&lt;code&gt;go mod init wal&lt;/code&gt;) and run. The caller puts an entry on a queue and is told the write is safe. A background goroutine moves queued entries to disk, and shutdown tells it to stop.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;wal&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"testing"&lt;/span&gt;
    &lt;span class="s"&gt;"time"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;TestAckedWriteSurvivesShutdown&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;queue&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;stop&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;done&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{}),&lt;/span&gt; &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{})&lt;/span&gt;
    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;disk&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;

    &lt;span class="k"&gt;go&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="c"&gt;// the writer moves queued entries to disk&lt;/span&gt;
        &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="nb"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;queue&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;disk&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;disk&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;stop&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="c"&gt;// bug: anything still in the queue is lost&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}()&lt;/span&gt;

    &lt;span class="n"&gt;queue&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt; &lt;span class="s"&gt;"x=1"&lt;/span&gt;                    &lt;span class="c"&gt;// the caller is told the write is safe&lt;/span&gt;
    &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;50&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Microsecond&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c"&gt;// "give the writer a moment"&lt;/span&gt;
    &lt;span class="nb"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stop&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;done&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;disk&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatalf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"acknowledged write lost: disk=%v"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;disk&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The bug is in the writer's &lt;code&gt;select&lt;/code&gt;. If an entry is still in the queue when the stop signal arrives, both cases are ready. The &lt;a href="https://go.dev/ref/spec#Select_statements" rel="noopener noreferrer"&gt;Go spec&lt;/a&gt; says &lt;code&gt;select&lt;/code&gt; then picks one of them by uniform pseudo-random selection. If &lt;code&gt;stop&lt;/code&gt; wins, the writer returns and the acknowledged entry is lost.&lt;/p&gt;

&lt;p&gt;On my 16-core Apple M4 Max laptop with Go 1.27.1, the test passed 40,000 runs in a row while the machine was idle (&lt;code&gt;go test -count=10000&lt;/code&gt;, four times). Then I kept every core busy with 32 copies of &lt;code&gt;yes &amp;gt; /dev/null&lt;/code&gt; and ran 60,000 more. The test failed 23 times, between 1 and 7 times per 10,000 runs.&lt;/p&gt;

&lt;p&gt;The race detector does not catch it. &lt;code&gt;go test -race&lt;/code&gt; stayed quiet for 2,200 runs, because there is no data race: closing &lt;code&gt;done&lt;/code&gt; orders the writer's last append before the test reads &lt;code&gt;disk&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;When the test does fail, this is all you get:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;--- FAIL: TestAckedWriteSurvivesShutdown (0.00s)
    wal_test.go:30: acknowledged write lost: disk=[]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is nothing in that output you can run again. The next run makes its own random choices, and in the 40,000 idle runs, no run lost the write.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why normal tests miss these bugs
&lt;/h2&gt;

&lt;p&gt;A test controls its inputs. It does not control the clock, goroutine scheduling, the network or the disk, and in a distributed system those decide when things happen:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;code&gt;time.Now&lt;/code&gt; and timers follow the wall clock, so whether a timeout fires first depends on how busy the machine is.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The Go runtime decides which goroutine runs next, on several OS threads, and since &lt;a href="https://go.dev/doc/go1.14#runtime" rel="noopener noreferrer"&gt;Go 1.14&lt;/a&gt; it can preempt goroutines asynchronously. There is no way to hand the scheduler a seed.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;A TCP connection over localhost does not lose or reorder bytes, and it never runs into a partition. On a real network, packets get lost and reordered, connections stall and break, and partitions happen.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;A write that returned is not durable until it is synced, and a test process does not crash between the two.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So a normal test explores a small part of the timings your system can see, mostly the ones an idle laptop produces, and sleeps and retries make this worse. Say someone added the sleep in the example after the test failed once. Without it, the test failed 971 times in 2,000 runs on the same laptop. The sleep did not fix the bug. It made the losing timing rarer, which made the bug harder to find. A retry in CI does the same for the whole suite: it turns a test that hit a bad timing into a green check mark.&lt;/p&gt;

&lt;p&gt;That is why I treat a flaky test in a distributed system as a bug report until I can show otherwise. It may have hit a timing the code mishandles, and it came with no steps to reproduce it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What deterministic simulation testing is
&lt;/h2&gt;

&lt;p&gt;Deterministic simulation testing (DST) takes those sources of nondeterminism away from the machine and gives them to the test. In its usual form, you run the whole system, every node, inside one process, in a simulated world with its own clock, network and disk. The clock there runs on virtual time: it jumps from one event to the next and never waits. Every random choice comes from one number, the seed: which message arrives next, how long it takes, which ready task runs first, whether a disk write survives a crash.&lt;/p&gt;

&lt;p&gt;Then you add faults on purpose. The simulator delays and drops messages or delivers them out of order, cuts the network into partitions, crashes and restarts nodes, and fails disk writes. A test runs many seeds and checks, after each run, that the system kept its promises, for example that no acknowledged write was lost.&lt;/p&gt;

&lt;p&gt;When a seed fails, you replay it: you run the same seed again and get the same run. Every message and every fault lands at the same point as before, and the run ends in the same failure. The seed is the reproduction. You can add logging or step through the run in a debugger, and once you have a fix, you run the same seed to see whether it passes. After a code change, though, the same seed can take a different path, so one passing seed does not prove the fix.&lt;/p&gt;

&lt;p&gt;Apply that to the example. A simulator that controlled the writer's &lt;code&gt;select&lt;/code&gt; would make its choice from the seed, so a losing seed would lose the write on every replay.&lt;/p&gt;

&lt;p&gt;None of this is new. FoundationDB's &lt;a href="https://apple.github.io/foundationdb/testing.html" rel="noopener noreferrer"&gt;simulation&lt;/a&gt; runs a whole cluster deterministically in a single-threaded process. It injects faults from broken connections and rebooting machines up to whole datacenters, and the team estimates they have run the equivalent of about a trillion CPU-hours of simulation. TigerBeetle's simulator, the &lt;a href="https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/internals/vopr.md" rel="noopener noreferrer"&gt;VOPR&lt;/a&gt;, stubs out the clock, the network and the disk. It drops and reorders packets, partitions the network, corrupts disk reads and writes, and replays any failure from its seed and Git commit.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://antithesis.com/product/how_does_antithesis_work/" rel="noopener noreferrer"&gt;Antithesis&lt;/a&gt; takes a different route. It built a deterministic hypervisor that runs your whole system on a single CPU core, so the software inside does not have to change. Any bug it finds can be reproduced. FoundationDB and TigerBeetle went the other way: their code is written to run inside their own simulators. FoundationDB even built its own runtime, Flow, which runs the same code in production or in simulation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this is hard in Go
&lt;/h2&gt;

&lt;p&gt;A simulator has to make every choice itself, and the scheduler's choices are not the only ones a Go test cannot control:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Besides &lt;code&gt;select&lt;/code&gt;, the spec leaves &lt;a href="https://go.dev/ref/spec#For_range" rel="noopener noreferrer"&gt;map iteration order&lt;/a&gt; unspecified, and since &lt;a href="https://go.dev/doc/go1.20#mathrandpkgmathrand" rel="noopener noreferrer"&gt;Go 1.20&lt;/a&gt; the global &lt;code&gt;math/rand&lt;/code&gt; generator gets a random seed at startup.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;code&gt;net&lt;/code&gt; and &lt;code&gt;os&lt;/code&gt; call into the operating system, which a test cannot partition or crash.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Since Go 1.25, the clock is the exception. That release added the &lt;a href="https://pkg.go.dev/testing/synctest" rel="noopener noreferrer"&gt;&lt;code&gt;testing/synctest&lt;/code&gt;&lt;/a&gt; package to the standard library. &lt;code&gt;synctest.Test&lt;/code&gt; runs a function in a "bubble" whose goroutines see a fake clock. That clock moves forward only when every goroutine in the bubble is durably blocked. Durably blocked means only another goroutine in the bubble can wake it, for example because it waits on a channel created inside the bubble. Tests with timeouts run fast, and the times they read are the same on every run.&lt;/p&gt;

&lt;p&gt;I put the example in a bubble with a second file in the same package:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;wal&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"testing"&lt;/span&gt;
    &lt;span class="s"&gt;"testing/synctest"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;TestAckedWriteInBubble&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;synctest&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;TestAckedWriteSurvivesShutdown&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With the sleep, &lt;code&gt;go test -count=10000 -run TestAckedWriteInBubble&lt;/code&gt; passed 10,000 runs out of 10,000, both idle and with every core busy. The fake clock does not move until the writer has emptied the queue and blocked again, so the window for the bug never opens. With the &lt;code&gt;time.Sleep&lt;/code&gt; line and the &lt;code&gt;time&lt;/code&gt; import deleted, it failed 5,017 times in 10,000: &lt;code&gt;select&lt;/code&gt; still picks a ready case at random, and nothing records which one it picked.&lt;/p&gt;

&lt;p&gt;Apart from the clock, synctest leaves a test without the controls DST needs. Its API takes no seed, and its documentation says nothing about goroutine order. It has no simulated network or disk and no way to inject faults such as crashes, and the same documentation tells you to avoid the network and use a fake.&lt;/p&gt;

&lt;p&gt;So DST in Go today means picking a trade-off. One way is to write the core of a system as a state machine that a simulator drives. The &lt;a href="https://github.com/etcd-io/raft" rel="noopener noreferrer"&gt;etcd/raft&lt;/a&gt; library is built like this: it implements only the Raft algorithm and leaves network and disk I/O to the caller. That gives exact replay, but only for code written that way. The other way is to run ordinary goroutine code and take control of the runtime underneath it. That works with the code you already have. Exact replay is the hard part, because the simulator would have to take over the scheduler and every item in the list at the start of this section.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I am building
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/hmdsefi/faultline" rel="noopener noreferrer"&gt;faultline&lt;/a&gt; is an open-source deterministic simulation testing tool for Go that runs inside &lt;code&gt;go test&lt;/code&gt;. Its first release, &lt;a href="https://github.com/hmdsefi/faultline/releases/tag/v0.1.0" rel="noopener noreferrer"&gt;v0.1.0&lt;/a&gt;, is out. It runs code written for its simulator, plus etcd/raft through a ready-made harness. Running ordinary goroutine code like the example above is the next phase on the &lt;a href="https://github.com/hmdsefi/faultline/issues/167" rel="noopener noreferrer"&gt;roadmap&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>go</category>
      <category>testing</category>
      <category>distributedsystems</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Finding cycles and hidden dependencies in go mod graph</title>
      <dc:creator>Hamed Yousefi</dc:creator>
      <pubDate>Mon, 12 Oct 2026 17:27:57 +0000</pubDate>
      <link>https://dev.to/hamed_yousefi/finding-cycles-and-hidden-dependencies-in-go-mod-graph-1j87</link>
      <guid>https://dev.to/hamed_yousefi/finding-cycles-and-hidden-dependencies-in-go-mod-graph-1j87</guid>
      <description>&lt;p&gt;&lt;code&gt;go mod graph&lt;/code&gt; prints every requirement in your module graph, one per line. For Prometheus that's 2,510 lines covering 893 module versions. All the information is there, but not in a form you can read. Questions like "do any of these modules require each other?", "what depends on this module?" or "what did this pull request change in my dependencies?" turn into a lot of grep.&lt;/p&gt;

&lt;p&gt;The output is a graph, so I loaded it into &lt;a href="https://github.com/hmdsefi/gograph" rel="noopener noreferrer"&gt;gograph&lt;/a&gt;, the graph library I maintain, and wrote a small program, &lt;a href="https://github.com/hmdsefi/gograph/tree/master/examples/gomodgraph" rel="noopener noreferrer"&gt;gomodgraph&lt;/a&gt;, that answers those questions. Each command turns out to be a few lines on top of a standard graph algorithm. This post walks through them, with real output from Prometheus. The same algorithms are on &lt;a href="https://gograph.dev" rel="noopener noreferrer"&gt;gograph.dev&lt;/a&gt;, on graphs small enough to step through.&lt;/p&gt;

&lt;p&gt;You can run it on your own module with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go mod graph | go run github.com/hmdsefi/gograph/examples/gomodgraph@latest cycles
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Loading the graph
&lt;/h2&gt;

&lt;p&gt;Each line of &lt;code&gt;go mod graph&lt;/code&gt; is &lt;code&gt;from to&lt;/code&gt;, meaning &lt;code&gt;from&lt;/code&gt; requires &lt;code&gt;to&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;github.com/prometheus/prometheus cloud.google.com/go/auth@v0.23.3
golang.org/x/crypto@v0.57.0 golang.org/x/net@v0.58.0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The main module has no version, and every other module is written as &lt;code&gt;path@version&lt;/code&gt;. Each line becomes one edge:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;g&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;gograph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="n"&gt;gograph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Directed&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="n"&gt;scanner&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;bufio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewScanner&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Scan&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;fields&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fields&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scanner&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Text&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AddEdge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vertex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;vertex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;vertex&lt;/code&gt; returns the vertex with that label, or adds it if it's new. The real program also skips the &lt;code&gt;go@&lt;/code&gt; and &lt;code&gt;toolchain@&lt;/code&gt; lines, which aren't modules.&lt;/p&gt;

&lt;p&gt;The direction matters for everything that follows. With an edge &lt;code&gt;from -&amp;gt; to&lt;/code&gt;, the dependencies of a module are the vertices it can reach, and the modules that depend on it are the vertices that can reach it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cycles
&lt;/h2&gt;

&lt;p&gt;Requirement cycles are allowed in Go, and they're common between modules that are released together. They're usually harmless, but they mean that upgrading any module in the cycle raises the others too. That's good to know when an upgrade pulls in more than you expected.&lt;/p&gt;

&lt;p&gt;In a directed graph, a cycle is a strongly connected component with more than one vertex. gograph has three algorithms for these. The example uses Tarjan's:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;component&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;connectivity&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Tarjan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;component&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;labels&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;([]&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;component&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;component&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;sort&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Strings&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Println&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On Prometheus it finds one cycle:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ gomodgraph -f graph.txt cycles
go.opentelemetry.io/otel/metric@v1.46.0 go.opentelemetry.io/otel/trace@v1.46.0 go.opentelemetry.io/otel@v1.46.0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;otel&lt;/code&gt; requires &lt;code&gt;otel/metric&lt;/code&gt; and &lt;code&gt;otel/trace&lt;/code&gt;, and both of them require &lt;code&gt;otel&lt;/code&gt; back. They're released from the same repository with the same version, so that's expected. Loading the graph and finding the cycle takes about 10 ms on my laptop. On a small call graph, the same finder is &lt;a href="https://gograph.dev/algorithms/tarjan/service-calls" rel="noopener noreferrer"&gt;Tarjan on gograph.dev&lt;/a&gt;. The number on that page is measured on the server, separate from this laptop run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Dependents
&lt;/h2&gt;

&lt;p&gt;"What depends on this module?" means "which vertices can reach it?" The example builds a reversed copy of the graph and walks it breadth-first from each version of the module:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;reversed&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;gograph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="n"&gt;gograph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Directed&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AllEdges&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;reversed&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AddEdge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;vertex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reversed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Destination&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
        &lt;span class="n"&gt;vertex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reversed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Source&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;traverse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewBreadthFirstIterator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reversed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"github.com/golang/protobuf@v1.3.2"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;HasNext&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Println&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Next&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;gograph 0.8 also has a &lt;code&gt;dag&lt;/code&gt; package, and &lt;code&gt;dag.Ancestors(g, label)&lt;/code&gt; does the same walk without building a second graph. Its names follow the build graph convention, where &lt;code&gt;A -&amp;gt; B&lt;/code&gt; means "B depends on A". For a requirement graph like this one the meanings swap, so &lt;code&gt;Ancestors&lt;/code&gt; returns the dependents.&lt;/p&gt;

&lt;p&gt;In Prometheus, 39 modules require some version of &lt;code&gt;github.com/golang/protobuf&lt;/code&gt;, directly or indirectly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ gomodgraph -f graph.txt dependents github.com/golang/protobuf
github.com/armon/go-metrics@v0.4.1
github.com/envoyproxy/go-control-plane/envoy@v1.39.0
github.com/google/s2a-go@v0.1.10
github.com/hashicorp/consul/api@v1.34.5
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A shorter walk of the same kind is &lt;a href="https://gograph.dev/algorithms/bfs/office-network" rel="noopener noreferrer"&gt;breadth-first search on gograph.dev&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is this module here?
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;go mod why -m&lt;/code&gt; answers a related question, but it looks at the package import graph. It shows which chain of imports needs a module. For a module that no package imports, it prints &lt;code&gt;(main module does not need module ...)&lt;/code&gt;, even though the module is still in the graph and in &lt;code&gt;go.sum&lt;/code&gt;. The requirement graph answers the other half: which chain of &lt;code&gt;go.mod&lt;/code&gt; files brought it in.&lt;/p&gt;

&lt;p&gt;In a graph where every edge has the same cost, the shortest path comes from a breadth-first search. The iterator returns vertices in order of distance, so the first vertex that has an edge to a new vertex is on a shortest path to it. &lt;code&gt;matches&lt;/code&gt; accepts either a module path, which matches every version, or &lt;code&gt;path@version&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;parent&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;map&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;HasNext&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;it&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Next&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;module&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;label&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="n"&gt;label&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;label&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;append&lt;/span&gt;&lt;span class="p"&gt;([]&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Neighbors&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;()];&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The output uses the format of &lt;code&gt;go mod why -m&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ gomodgraph -f graph.txt why github.com/golang/protobuf
# github.com/golang/protobuf
github.com/prometheus/prometheus
github.com/armon/go-metrics@v0.4.1
github.com/golang/protobuf@v1.3.2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://gograph.dev/algorithms/ancestors/go-mod-graph" rel="noopener noreferrer"&gt;Ancestors on gograph.dev&lt;/a&gt; answers the same question on a module graph small enough to draw.&lt;/p&gt;

&lt;h2&gt;
  
  
  What changed between two versions?
&lt;/h2&gt;

&lt;p&gt;Module versions are vertices and requirements are edges, so comparing two graphs takes two set differences. A module version is removed if the new graph doesn't contain its vertex, and a requirement is removed if the new graph doesn't contain its edge:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;oldGraph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;GetAllVertices&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;newGraph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ContainsVertex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Println&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"- module"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;oldGraph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AllEdges&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;newGraph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ContainsEdge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Source&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Destination&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Println&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"- require"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Source&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Destination&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Label&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same loops with the graphs swapped print what was added. Between Prometheus v3.14.0 and v3.15.0, 277 module versions left the graph and 235 came in, and 1,472 requirements were removed while 1,504 were added. Most of that is version bumps deep in the graph. The lines for the main module show the direct changes. Here are the &lt;code&gt;golang.org/x&lt;/code&gt; ones:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ gomodgraph diff v3.14.0.txt v3.15.0.txt | grep 'require github.com/prometheus/prometheus golang.org/x/'
- require github.com/prometheus/prometheus golang.org/x/crypto@v0.54.0
- require github.com/prometheus/prometheus golang.org/x/mod@v0.38.0
- require github.com/prometheus/prometheus golang.org/x/net@v0.57.0
- require github.com/prometheus/prometheus golang.org/x/text@v0.40.0
- require github.com/prometheus/prometheus golang.org/x/tools@v0.48.0
+ require github.com/prometheus/prometheus golang.org/x/crypto@v0.56.0
+ require github.com/prometheus/prometheus golang.org/x/net@v0.58.0
+ require github.com/prometheus/prometheus golang.org/x/text@v0.41.0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three of them are upgrades. &lt;code&gt;golang.org/x/mod&lt;/code&gt; and &lt;code&gt;golang.org/x/tools&lt;/code&gt; have a &lt;code&gt;-&lt;/code&gt; line and no &lt;code&gt;+&lt;/code&gt; line, so v3.15.0 doesn't list them in its &lt;code&gt;go.mod&lt;/code&gt; any more. That's easy to miss in a &lt;code&gt;go.mod&lt;/code&gt; diff with dozens of changed lines.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running it on pull requests
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://github.com/hmdsefi/gograph/tree/master/examples/gomodgraph#comment-on-pull-requests-that-change-gomod" rel="noopener noreferrer"&gt;example's README&lt;/a&gt; has a GitHub Actions workflow that runs &lt;code&gt;go mod graph&lt;/code&gt; on the base and head commits of a pull request and posts the &lt;code&gt;diff&lt;/code&gt; as a comment. That way, reviewers see what a dependency bump pulls in before they approve it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Watching it run
&lt;/h2&gt;

&lt;p&gt;The Prometheus graph is too big to draw step by step. &lt;a href="https://gograph.dev" rel="noopener noreferrer"&gt;gograph.dev&lt;/a&gt; runs the same algorithms on small graphs, and the timing on each page is measured on the server when the site starts, on the released library.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://gograph.dev/algorithms/tarjan/service-calls" rel="noopener noreferrer"&gt;Tarjan on a call graph&lt;/a&gt;, the cycle finder from the first section.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://gograph.dev/algorithms/bfs/office-network" rel="noopener noreferrer"&gt;Breadth-first on a network&lt;/a&gt;, the walk behind dependents and why.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://gograph.dev/algorithms/ancestors/go-mod-graph" rel="noopener noreferrer"&gt;Ancestors on a module graph&lt;/a&gt;, the same question as dependents, on a graph small enough to draw.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://gograph.dev/algorithms/topological-sort/go-mod-graph" rel="noopener noreferrer"&gt;Topological sort on that module graph&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Wrapping up
&lt;/h2&gt;

&lt;p&gt;The whole program is about 340 lines, and most of that is argument parsing and output. The graph work is a Tarjan call, two breadth-first searches and a few set checks.&lt;/p&gt;

&lt;p&gt;gograph is a generic graph library with no dependencies outside the standard library. It covers dependency graphs (&lt;code&gt;Acyclic()&lt;/code&gt; graphs that reject cycles, &lt;code&gt;TopologySort&lt;/code&gt;, and the &lt;code&gt;dag&lt;/code&gt; package), traversal, shortest paths, strongly connected components and partitioning, and &lt;code&gt;encoding/mermaid&lt;/code&gt; writes any graph as a Mermaid flowchart. Version 0.8 made the order of vertices and edges predictable, so results are the same on every run. I wrote about how that works in &lt;a href="https://hmdsefi.hashnode.dev/why-your-topological-sort-gives-a-different-answer-on-every-run" rel="noopener noreferrer"&gt;an earlier post&lt;/a&gt;.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Repository: &lt;a href="https://github.com/hmdsefi/gograph" rel="noopener noreferrer"&gt;https://github.com/hmdsefi/gograph&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Live runs: &lt;a href="https://gograph.dev" rel="noopener noreferrer"&gt;https://gograph.dev&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Documentation: &lt;a href="https://pkg.go.dev/github.com/hmdsefi/gograph" rel="noopener noreferrer"&gt;https://pkg.go.dev/github.com/hmdsefi/gograph&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Roadmap: &lt;a href="https://github.com/hmdsefi/gograph/issues/136" rel="noopener noreferrer"&gt;https://github.com/hmdsefi/gograph/issues/136&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you run it on your own module and find something surprising, I'd like to hear about it.&lt;/p&gt;

</description>
      <category>go</category>
      <category>opensource</category>
      <category>devops</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Why your topological sort gives a different answer on every run</title>
      <dc:creator>Hamed Yousefi</dc:creator>
      <pubDate>Mon, 05 Oct 2026 13:21:27 +0000</pubDate>
      <link>https://dev.to/hamed_yousefi/why-your-topological-sort-gives-a-different-answer-on-every-run-2la4</link>
      <guid>https://dev.to/hamed_yousefi/why-your-topological-sort-gives-a-different-answer-on-every-run-2la4</guid>
      <description>&lt;p&gt;You have a few tasks, and some of them depend on others. You put them in a graph, sort it topologically, and print the order. Then you run the program again and get a different order. Both are correct, but your test that compares against an expected slice fails every few runs. And the file you generate from the order changes every time you rebuild it, even though nothing changed.&lt;/p&gt;

&lt;p&gt;This happened in &lt;a href="https://github.com/hmdsefi/gograph" rel="noopener noreferrer"&gt;gograph&lt;/a&gt;, the Go graph library I maintain, until version 0.8. This post explains where the randomness came from, how the library got rid of it without changing its API, and what to do when insertion order isn't enough. The pipeline from the next section is on &lt;a href="https://gograph.dev/algorithms/topological-sort/build-pipeline" rel="noopener noreferrer"&gt;gograph.dev&lt;/a&gt;, where you can step through the sort instead of only reading the code.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the randomness comes from
&lt;/h2&gt;

&lt;p&gt;Here's a small build pipeline. Four tasks can run in any order, and &lt;code&gt;release&lt;/code&gt; has to wait for all of them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;g&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;gograph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="n"&gt;gograph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Acyclic&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="n"&gt;release&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AddVertexByLabel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"release"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;"lint"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"vet"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"test"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"build"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AddEdge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AddVertexByLabel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;release&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="m"&gt;5&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;gograph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TopologySort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c"&gt;// print the labels&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With gograph v0.7.2, five calls in the same process gave this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;vet test build lint release
lint vet test build release
lint vet test build release
test build lint vet release
lint vet test build release
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A topological sort only promises that every vertex comes after the vertices it depends on. Whenever more than one vertex is ready at the same time, the algorithm has to pick one, and any choice is valid. Here, all four tasks are ready from the start.&lt;/p&gt;

&lt;p&gt;The graph stored its vertices in a &lt;code&gt;map[T]*Vertex[T]&lt;/code&gt;, and &lt;code&gt;GetAllVertices&lt;/code&gt; built its result by ranging over it. &lt;code&gt;TopologySort&lt;/code&gt; then counted the incoming edges of each vertex in another map and found the starting vertices by ranging over that one. The Go spec says the iteration order of a map "is not specified and is not guaranteed to be the same from one iteration to the next", and the runtime randomizes it on purpose, so code can't come to depend on it. So the tie-break was random, and so was the result. That's also why the fix needed more than one change. The maps are still there for lookups, but every loop over a map on that path had to become a loop over a slice.&lt;/p&gt;

&lt;p&gt;The same thing happened in every function built on top of the vertex list: the strongly connected component algorithms (Tarjan, Kosaraju and Gabow), maximal cliques, Girvan-Newman communities and transitive reduction. They all gave correct results, but not the same correct result twice. &lt;a href="https://gograph.dev" rel="noopener noreferrer"&gt;gograph.dev&lt;/a&gt; runs those too. &lt;a href="https://gograph.dev/algorithms/tarjan/service-calls" rel="noopener noreferrer"&gt;Tarjan on a call graph&lt;/a&gt; is the one that finds the groups of vertices that can all reach each other.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keeping the insertion order
&lt;/h2&gt;

&lt;p&gt;The fix is to remember the order the vertices were added. The graph still needs the map, because finding a vertex by its label has to stay fast. Next to it, it now keeps a slice, and each vertex stores its index in that slice:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;baseGraph&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;addVertex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Vertex&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Vertex&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c"&gt;// ...&lt;/span&gt;
    &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;vertices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;
    &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;position&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c"&gt;// ...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Adding is an append. Removing is the only tricky part, since deleting from the middle of a slice means moving everything after it. Instead, a removed vertex leaves an empty slot, and once more than half of the slots are empty, the remaining vertices move to the front:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;baseGraph&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;removeFromOrder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Vertex&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;position&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
    &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;removedCount&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;removedCount&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;position&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
            &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;
            &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;clear&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;removedCount&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each compaction moves at most as many vertices as were removed since the last one, so a removal still costs O(1) on average.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;GetAllVertices&lt;/code&gt; now walks the slice and skips the empty slots. Edges got the same treatment: &lt;code&gt;AllEdges&lt;/code&gt; groups them by source vertex in insertion order, and within each vertex they're in the order they were added. After that, the algorithms that ranged over maps were changed to range over these slices.&lt;/p&gt;

&lt;p&gt;The same program with v0.8.1:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;lint vet test build release
lint vet test build release
lint vet test build release
lint vet test build release
lint vet test build release
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;lint&lt;/code&gt; was added first, so it comes first. When several vertices are ready, &lt;code&gt;TopologySort&lt;/code&gt; now takes them in the order they were added. All the algorithms listed above give the same result on every run for a graph built the same way. The &lt;a href="https://gograph.dev/algorithms/topological-sort/build-pipeline" rel="noopener noreferrer"&gt;build pipeline on gograph.dev&lt;/a&gt; is this case: lint, vet, test, build, then release.&lt;/p&gt;

&lt;p&gt;The change was also faster. Walking a slice beats ranging over a map, and in my benchmarks &lt;code&gt;GetAllVertices&lt;/code&gt; got about 6 times faster, &lt;code&gt;AllEdges&lt;/code&gt; 3 times and &lt;code&gt;EdgesOf&lt;/code&gt; 5 times. On a graph with 100,000 vertices, &lt;code&gt;TopologySort&lt;/code&gt; takes about 7 ms.&lt;/p&gt;

&lt;p&gt;None of the signatures changed. The order was never documented before, so code that didn't depend on it keeps working, and the docs now say what the order is.&lt;/p&gt;

&lt;h2&gt;
  
  
  When insertion order isn't enough
&lt;/h2&gt;

&lt;p&gt;Insertion order moves the problem to whoever builds the graph. If you fill the graph from a map, from files listed in directory order, or from results that come back from goroutines, the insertion order is random again, and so is the sort. Here's the same pipeline built twice, with the tasks added in a different order:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;build&lt;/span&gt;&lt;span class="p"&gt;([]&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;"lint"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"vet"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"test"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"build"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;build&lt;/span&gt;&lt;span class="p"&gt;([]&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;"build"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"test"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"vet"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"lint"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;TopologySort&lt;/code&gt; follows each graph's insertion order, so the results differ:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;lint vet test build release
build test vet lint release
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For this case, 0.8 added &lt;code&gt;StableTopologySort&lt;/code&gt;. It takes a compare function, and whenever several vertices are ready, it picks the smallest one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;stableA&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;gograph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StableTopologySort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cmp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Compare&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="n"&gt;stableB&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;gograph&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StableTopologySort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cmp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Compare&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;build lint test vet release
build lint test vet release
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The ready vertices go in a heap ordered by the compare function, so the sort runs in O((V + E) log V) instead of O(V + E). On the 100,000 vertex graph it takes about 15 ms, twice as long as &lt;code&gt;TopologySort&lt;/code&gt;. That's the price of an order that doesn't depend on how the graph was built. If you build the graph in a fixed order anyway, &lt;code&gt;TopologySort&lt;/code&gt; is enough. &lt;a href="https://gograph.dev/algorithms/stable-topological-sort/build-pipeline" rel="noopener noreferrer"&gt;gograph.dev runs that stable sort&lt;/a&gt; on the same pipeline, and both graphs come out &lt;code&gt;build lint test vet release&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The general lesson
&lt;/h2&gt;

&lt;p&gt;Any algorithm that has to make an arbitrary choice, like which ready vertex goes next or which neighbor to visit first, takes that choice from whatever order its data is in. If the data is in a map, the choice is random. The algorithm is still correct, but its output can't be tested with a simple comparison, cached or diffed.&lt;/p&gt;

&lt;p&gt;The fix doesn't have to be expensive. Keep a slice next to the map when you need insertion order, or sort the candidates when you need an order that doesn't depend on history. It's worth deciding which one your API promises and writing it in the docs, because someone will eventually depend on it.&lt;/p&gt;

&lt;p&gt;gograph is a generic graph library for Go with no dependencies outside the standard library. Besides topological sorting, it covers traversal, shortest paths, strongly connected components and partitioning. Version 0.8 also added a &lt;code&gt;dag&lt;/code&gt; package for dependency questions and &lt;code&gt;encoding/mermaid&lt;/code&gt; for drawing graphs. &lt;a href="https://gograph.dev" rel="noopener noreferrer"&gt;gograph.dev&lt;/a&gt; runs the ones in the current release, step by step, and each page shows how long that run took on the server.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The sort from this post: &lt;a href="https://gograph.dev/algorithms/topological-sort/build-pipeline" rel="noopener noreferrer"&gt;https://gograph.dev/algorithms/topological-sort/build-pipeline&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The stable sort: &lt;a href="https://gograph.dev/algorithms/stable-topological-sort/build-pipeline" rel="noopener noreferrer"&gt;https://gograph.dev/algorithms/stable-topological-sort/build-pipeline&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;All of the runs: &lt;a href="https://gograph.dev" rel="noopener noreferrer"&gt;https://gograph.dev&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Repository: &lt;a href="https://github.com/hmdsefi/gograph" rel="noopener noreferrer"&gt;https://github.com/hmdsefi/gograph&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Documentation: &lt;a href="https://pkg.go.dev/github.com/hmdsefi/gograph" rel="noopener noreferrer"&gt;https://pkg.go.dev/github.com/hmdsefi/gograph&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The change itself: &lt;a href="https://github.com/hmdsefi/gograph/pull/167" rel="noopener noreferrer"&gt;https://github.com/hmdsefi/gograph/pull/167&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>go</category>
      <category>algorithms</category>
      <category>opensource</category>
      <category>performance</category>
    </item>
  </channel>
</rss>
