Building Distributed Systems in Elixir: Part 3 — Process Monitoring

In the previous parts of this series, we built a stateful process using a mailbox and then added correlated request–reply using references.

Now we have another problem.

What happens when the process we are communicating with dies?

A worker might finish normally, crash because of an exception, or terminate for some other reason.

Instead of repeatedly checking whether a process is alive, the BEAM provides process monitoring.

In this part, we'll build process monitoring directly using:

Process.monitor/1

No GenServer.

No Supervisor.

The goal is to understand the primitive underneath OTP's failure-handling mechanisms.

---

The Problem

Imagine we start a worker:

worker = Worker.start("worker-1")

That worker is an independent BEAM process.

At some point it could stop:

send(worker, :stop)

or crash:

send(worker, :crash)

Another process may need to know that the worker is gone.

One option would be repeatedly checking:

Process.alive?(worker)

But polling isn't what we want.

Instead, we can ask the BEAM to notify us when the process terminates.

---

Monitoring a Process

Monitoring starts with one line:

ref = Process.monitor(worker)

The important detail is:

The process calling Process.monitor/1 becomes the monitoring process.

Process.monitor/1 returns a unique reference.

If the worker later terminates, the BEAM sends the monitoring process a message:

{:DOWN, ref, :process, worker, reason}

This is just a message.

That means we can handle it using the same receive primitive we've already been using.

---

A Simple Worker

Let's create a worker that understands three messages:

defmodule Worker do
  def start(name) do
    spawn(fn -> loop(name) end)
  end

  defp loop(name) do
    receive do
      {:work, from, value} ->
        IO.puts("#{name} received #{value}")
        send(from, {:done, name, value})
        loop(name)

      :stop ->
        IO.puts("#{name} stopping normally")
        :ok

      :crash ->
        raise "#{name} crashed"
    end
  end
end

The process starts here:

spawn(fn -> loop(name) end)

The newly spawned process executes loop/1 and waits at:

receive do

If it receives work:

{:work, from, value}

it handles the message and calls:

loop(name)

again.

That keeps the process alive.

But look at :stop:

:stop ->
  IO.puts("#{name} stopping normally")
  :ok

There is no call to loop/1.

The function finishes, so the process terminates normally.

---

Detecting a Normal Exit

Now let's monitor the worker:

def normal_exit do
  worker = Worker.start("worker-1")

  ref = Process.monitor(worker)

  send(worker, :stop)

  receive do
    {:DOWN, ^ref, :process, ^worker, reason} ->
      IO.puts("worker-1 terminated")
      IO.puts("reason: #{inspect(reason)}")
  end
end

The execution flow looks like this:

Monitoring Process                 Worker Process

       |                                |
       | -------- spawn ------------->  |
       |                                |
       |                             receive
       |                             waiting
       |
       | Process.monitor(worker)
       |
       | -------- :stop ------------->  |
       |                                |
       |                         receives :stop
       |                                |
       |                          returns :ok
       |                                |
       |                           terminates
       |                                X
       |
       |      BEAM detects termination
       |
       | <---- {:DOWN, ..., :normal}
       |
    receive
       |
       v
 reason = :normal

There are two separate messages here.

First:

:stop

is sent to the worker's mailbox.

Later:

{:DOWN, ref, :process, worker, :normal}

is sent by the BEAM to the monitoring process's mailbox.

This distinction is important.

---

Understanding the :DOWN Message

The notification has this shape:

{:DOWN, ref, :process, pid, reason}

Let's break it down.

:DOWN

Identifies the message as a monitor notification.

ref

The unique monitor reference returned by:

Process.monitor(worker)

:process

Indicates that the monitored entity was a process.

pid

The PID of the process that terminated.

reason

Describes why the process terminated.

For a normal exit:

:normal

For a crash, it contains information about the failure.

---

Why Use ^ref?

Our receive pattern contains:

{:DOWN, ^ref, :process, ^worker, reason}

The pin operator ^ tells Elixir to match against an existing value.

We already have:

ref = Process.monitor(worker)

So:

^ref

means:

Match the :DOWN message belonging to this particular monitor.

Likewise:

^worker

means:

Match the PID of this particular worker.

This lets us precisely identify the termination notification we're waiting for.

---

Detecting a Crash

Monitoring also works when a worker crashes.

def crash_exit do
  worker = Worker.start("worker-2")

  ref = Process.monitor(worker)

  send(worker, :crash)

  receive do
    {:DOWN, ^ref, :process, ^worker, reason} ->
      IO.puts("worker-2 terminated")
      IO.puts("reason: #{inspect(reason)}")
  end
end

The worker receives:

:crash

and executes:

raise "#{name} crashed"

The worker dies.

But the monitoring process does not die with it.

Instead, the BEAM sends it a :DOWN message containing the failure reason.

This gives us an important property:

Monitoring lets one process observe another process's failure without automatically propagating that failure.

---

Monitoring Multiple Workers

Real systems usually have more than one worker.

Let's start three:

worker1 = Worker.start("worker-1")
worker2 = Worker.start("worker-2")
worker3 = Worker.start("worker-3")

Then monitor each one:

ref1 = Process.monitor(worker1)
ref2 = Process.monitor(worker2)
ref3 = Process.monitor(worker3)

Each monitor gets a different reference.

We can keep track of them using a map:

%{
  ref1 => "worker-1",
  ref2 => "worker-2",
  ref3 => "worker-3"
}

Think of this map as:

workers whose termination we are still waiting for

ref1 -> worker-1
ref2 -> worker-2
ref3 -> worker-3

Now send different termination messages:

send(worker1, :stop)
send(worker2, :crash)
send(worker3, :stop)

One worker crashes while the other two stop normally.

---

Waiting for All Workers

We can write a small recursive function to collect the monitor notifications:

defp wait_for_workers(monitors) when map_size(monitors) == 0 do
  IO.puts("all workers terminated")
end

defp wait_for_workers(monitors) do
  receive do
    {:DOWN, ref, :process, pid, reason} ->
      case Map.pop(monitors, ref) do
        {nil, monitors} ->
          wait_for_workers(monitors)

        {name, remaining} ->
          IO.puts("#{name} terminated")
          IO.puts("pid: #{inspect(pid)}")
          IO.puts("reason: #{inspect(reason)}")
          IO.puts("")

          wait_for_workers(remaining)
      end
  end
end

The function has a simple purpose:

Wait for each worker to terminate and log what happened.

Conceptually:

3 workers
    |
    | worker terminates
    v
receive :DOWN
    |
    v
log termination
    |
    v
remove monitor
    |
    v
2 workers
    |
    | worker terminates
    v
receive :DOWN
    |
    v
1 worker
    |
    | worker terminates
    v
receive :DOWN
    |
    v
0 workers
    |
    v
finished

The order is not guaranteed.

Even though we send:

send(worker1, :stop)
send(worker2, :crash)
send(worker3, :stop)

the workers execute concurrently.

Any of them may terminate first.

---

The Complete Example

defmodule Worker do
  def start(name) do
    spawn(fn -> loop(name) end)
  end

  defp loop(name) do
    receive do
      {:work, from, value} ->
        IO.puts("#{name} received #{value}")
        send(from, {:done, name, value})
        loop(name)

      :stop ->
        IO.puts("#{name} stopping normally")
        :ok

      :crash ->
        raise "#{name} crashed"
    end
  end
end

defmodule MonitorDemo do
  def normal_exit do
    worker = Worker.start("worker-1")
    ref = Process.monitor(worker)

    send(worker, :stop)

    receive do
      {:DOWN, ^ref, :process, ^worker, reason} ->
        IO.puts("worker-1 terminated")
        IO.puts("reason: #{inspect(reason)}")
    end
  end

  def crash_exit do
    worker = Worker.start("worker-2")
    ref = Process.monitor(worker)

    send(worker, :crash)

    receive do
      {:DOWN, ^ref, :process, ^worker, reason} ->
        IO.puts("worker-2 terminated")
        IO.puts("reason: #{inspect(reason)}")
    end
  end

  def multiple_workers do
    worker1 = Worker.start("worker-1")
    worker2 = Worker.start("worker-2")
    worker3 = Worker.start("worker-3")

    ref1 = Process.monitor(worker1)
    ref2 = Process.monitor(worker2)
    ref3 = Process.monitor(worker3)

    send(worker1, :stop)
    send(worker2, :crash)
    send(worker3, :stop)

    wait_for_workers(%{
      ref1 => "worker-1",
      ref2 => "worker-2",
      ref3 => "worker-3"
    })
  end

  defp wait_for_workers(monitors) when map_size(monitors) == 0 do
    IO.puts("all workers terminated")
  end

  defp wait_for_workers(monitors) do
    receive do
      {:DOWN, ref, :process, pid, reason} ->
        case Map.pop(monitors, ref) do
          {nil, monitors} ->
            wait_for_workers(monitors)

          {name, remaining} ->
            IO.puts("#{name} terminated")
            IO.puts("pid: #{inspect(pid)}")
            IO.puts("reason: #{inspect(reason)}")
            IO.puts("")

            wait_for_workers(remaining)
        end
    end
  end
end

MonitorDemo.normal_exit()
MonitorDemo.crash_exit()
MonitorDemo.multiple_workers()

Run it with:

elixir monitor.exs

---

Monitor vs Polling

We could ask:

Process.alive?(worker)

But that only tells us whether the process was alive at that particular moment.

The process could terminate immediately afterward.

With monitoring:

Process.monitor(worker)

we instead ask the BEAM to notify us about termination.

monitor worker
      |
      v
worker runs independently
      |
      v
worker terminates
      |
      v
BEAM detects termination
      |
      v
{:DOWN, ...}
      |
      v
monitoring process

This fits naturally with the BEAM's message-passing model.

---

What We Built

Without using GenServer or Supervisor, we now have:

Process
   +
Mailbox
   +
Request / Reply
   +
Request References
   +
Process Monitoring

We're gradually building the primitives needed for fault-tolerant process systems.

The important idea from this part is:

Failure detection itself can be represented as message passing.

A worker terminates, and the monitoring process receives a message describing what happened.

---

What's Next?

Monitoring lets us observe another process's failure.

But sometimes processes need a stronger relationship.

What if the failure of one process should affect another process?

That's where process links come in.

In Part 4, we'll explore:

spawn_link/1

and see how failures propagate between linked processes.

---

Series

1. Process Mailbox
2. Correlated Request/Reply
3. Process Monitoring
4. Process Linking
5. Supervisor From Scratch

The examples intentionally use low-level process primitives first so we can understand what OTP abstractions are solving before using them.
Source code of the same can be found at: https://github.com/pckrishnadas88/elixir-distributed-systems-lab