DEV Community: Unnawut Leepaisalsuwanna

GenServer, Registry, DynamicSupervisor. Combined.

Unnawut Leepaisalsuwanna — Fri, 04 Oct 2019 17:34:42 +0000

In omisego/ewallet, we're building/refactoring a TransactionTracker that listens to a huge number of transactions (think money transactions) that also happen to be residing in an external source that is slow to update (Hello, Ethereum).

So we need a way for us to:

Run the trackers concurrently, so to enable massive amount of transaction tracking.
Look up a running tracker, so we can reuse it for different purposes.
Automatically restart a specific tracker that goes wonky, because with external sources, anything can go wrong.

With Elixir and OTP/BEAM behind the scene, we are able to solve the problem by utilizing 3 Elixir core features:

GenServer for building long-running, concurrent tasks.
Registry for looking up those running GenServer's.
DynamicSupervisor for monitoring those arbitrary number of GenServer's, and automatically restart one when it goes wonky.

The problem

If we were to implement our own custom wiring, we would have to do the following:

Start a new tracker process (a GenServer) attached to a DynamicSupervisor.
Register the tracker process with the registry.
Invoke the process by looking up the registry for the process ID.
Make sure the registry handles the process's crash, and remove the process from the registry.
Make sure the registry knows when the process is restarted and registers the new process ID back.
Deregister the process from the registry when it shuts down.

That's a lot of code for the registry, and a lot of code to wire up the GenServer, DynamicSupervisor and Registry together. Since there's a lot of moving parts, our implementation and wiring could be very prone to errors. All of this represents very little business value.

The solution

Because Elixir designed the GenServer, Registry and DynamicSupervisor to work together seamlessly, we are surprised by how few lines of code needed to wire these up together.

defmodule TransactionTracker do
  use GenServer

  @registry TransactionTracker.Registry
  @supervisor TransactionTracker.TrackerSupervisor

  def start(transaction_id) do
    opts = [
      transaction_id: transaction_id,
      name: {:via, Registry, {@registry, transaction_id}}
    ]

    DynamicSupervisor.start_child(@supervisor, {__MODULE__, opts})
  end

  def lookup(transaction_id) do
    case Registry.lookup(@registry, transaction_id) do
      [{pid, _}] -> {:ok, pid}
      [] -> {:error, :not_found}
    end
  end

  def start_link(opts) do
    {name, opts} = Keyword.pop(opts, :name)
    GenServer.start_link(__MODULE__, opts, name: name)
  end

  def init(opts) do
    state = %{
      transaction_id: Keyword.fetch!(opts, :transaction_id),
    }

    {:ok, state}
  end
  #...
end

defmodule TransactionTracker.Application do
  use Application

  def start(_type, _args) do
    children = [
      {Registry, keys: :unique, name: TransactionTracker.Registry},
      {DynamicSupervisor, name: TransactionTracker.TrackerSupervisor, strategy: :one_for_one}
    ]

    Supervisor.start_link(children, name: TransactionTracker.Supervisor, strategy: :one_for_one)
  end
end

The trick lies at name: {:via, Registry, term} described in the Registry docs, which is just one out of many ways that Registry can be used.

With above, we can now do a simple one-liner to start the tracker:

iex> TransactionTracker.start("txn_01dp371w0fnjhf9z2tjebx4vr4")
{:ok, #PID<0.104.0>}

This one-line call would automatically:

Start a new TransactionTracker GenServer for the given transaction ID.
Register the tracker with TransactionTracker.Registry.
Register the tracker with TransactionTracker.TrackerSupervisor.
Restart the tracker when it shuts down abnormally.
Return the correct process ID lookup even after a tracker restart.
Deregister the tracker on expected shutdown.
Allow interactions with the process via the returned pid or by looking up: TransactionTracker.lookup("txn_01dp371w0fnjhf9z2tjebx4vr4")

And this is what happens with an abnormal exit.

iex> TransactionTracker.start("txn_01dp371w0fnjhf9z2tjebx4vr4")
{:ok, #PID<0.157.0>}

iex> {:ok, pid} = TransactionTracker.lookup("txn_01dp371w0fnjhf9z2tjebx4vr4")
{:ok, #PID<0.157.0>}

iex> :ok = GenServer.stop(pid, :its_a_crash)
17:25:49.286 [error] GenServer {TransactionTracker.Registry, "txn_01dp371w0fnjhf9z2tjebx4vr4"} terminating
** (stop) :its_a_crash
Last message: []
State: %{transaction_id: "txn_01dp371w0fnjhf9z2tjebx4vr4"}

iex> {:ok, restarted_pid} = TransactionTracker.lookup("txn_01dp371w0fnjhf9z2tjebx4vr4")
{:ok, #PID<0.162.0>}

iex> :sys.get_state(restarted_pid)
%{transaction_id: "txn_01dp371w0fnjhf9z2tjebx4vr4"}

You would see that the lookup returns the new process automatically, and the process holds the same state as the previous one. Just start the process with your ideal identifier and you'll be able to access the process from anywhere, with guarantee that it'll point you to the correct process even if the process got restarted and the process ID changed.

Isn't it pretty?

Conclusion

By using Elixir's GenServer, Registry and DynamicSupervisor, we're able to reap the following benefits when executing long-running tasks.

A one-liner way to start a long-running process, encapsulating away the registry and supervisor.
The process, when it goes woowoo, gets restarted automatically by the supervisor.
The registry handles a process's shutdown automatically, so no need to worry about deregistering dead processes.
The process can be looked up via the registry with ease, using our own arbitrary identifier, and works across process crashes.

What do you think? Do you have better ways to manage long-running processes? Let me know!

Clean and reusable test helpers with Elixir macros

Unnawut Leepaisalsuwanna — Sat, 28 Sep 2019 18:59:16 +0000

Sometimes, seemingly redundant tests could serve as an assurance that our code works. While it's possible to abstract away the tests to a higher level, it may mean sacrificing readability.

In this post, I'd like to suggest a way to create test helpers that can be reused easily and works seamlessly with ExUnit. While we generally avoid creating macros as the official guide says:

Macros should only be used as a last resort. Remember that explicit is better than implicit. Clear code is better than concise code.

I believe this post presents an exact use case where macros allow us to have explicit, concise and clean tests at the same time.

The problem

To begin, let's say we have two schemas called User and Account. Each of them contain a name field that should not be blank. We could add tests like this:

defmodule UserTest do
  use ExUnit.Case 

  describe "insert/1" do
    #...
    test "fails when given a blank name" do
      {:error, changeset} = User.insert(%{name: nil})
      assert changeset.errors == [{:name, {"can't be blank", [validation: :required]}}]
    end
  end
end

defmodule AccountTest do
  use ExUnit.Case

  describe "insert/1" do
    #...
    test "fails when given a blank name" do
      {:error, changeset} = Account.insert(%{name: nil})
      assert changeset.errors == [{:name, {"can't be blank", [validation: :required]}}]
    end
  end
end

Imagine if you have a dozen of schemas, most of which will require a check for blank fields. How much of your test code will be redundant? And how incomprehensible it would be, as the test code gets larger and larger?

Interim solution: Helper functions

The interim solution we tried was by abstracting away the assertion into a helper function, like below:

defmodule TestHelper do
  import  ExUnit.Assertions

  def not_blank(schema, field) do
    {result, changeset} = schema.insert(%{field => nil})

    assert result == :error
    assert changeset.errors == [{field, {"can't be blank", [validation: :required]}}]
  end
end

defmodule UserTest do
  use ExUnit.Case
  import TestHelper

  describe "insert/1" do
    #...
    test "fails when given a blank name" do
      assert not_blank(User, :name)
    end
  end
end

defmodule AccountTest do
  use ExUnit.Case
  import TestHelper

  describe "insert/1" do
    #...
    test "fails when given a blank name" do
      assert not_blank(Account, :name)
    end
  end
end

While above works great, the problem is that it is still cluttered when you want many assertions in a single test case, or you rather prefer lean test cases by testing one thing at a time.

The real deal: Macros as test helpers

Here's how we use macros to generate clean test cases:

defmodule TestHelper do
  defmacro test_insert_prevent_blank(schema, field) do
    quote do
      test "fails when given a blank :#{unquote(field)}" do
        schema = unquote(schema)
        field = unquote(field)

        {result, changeset} =
          schema
          |> get_factory
          |> params_for(%{field => ""})
          |> schema.insert

        assert result == :error
        assert changeset.errors == [{field, {"can't be blank", [validation: :required]}}]
      end
    end
  end
end

With the macro above, we can now do one-liners like these:

defmodule UserTest do
  use ExUnit.Case
  import TestHelper

  describe "insert/1" do
    #...
    test_insert_prevent_blank(User, :name)
  end
end

defmodule AccountTest do
  use ExUnit.Case
  import TestHelper

  describe "insert/1" do
    #...
    test_insert_prevent_blank(Account, :name)
  end
end

And it is very readable when combined with other similar test helpers:

defmodule UserTest do
  use ExUnit.Case
  import TestHelper

  describe "insert/1" do
    test_insert_generate_uuid(User, :uuid)

    test_insert_prevent_blank(User, :name)
    test_insert_prevent_blank(User, :email)
    test_insert_prevent_duplicate(User, :email)

    test_insert_generate_timestamps(User)

    # Other schema-specific tests...
  end
end

Conclusion

We have been using this test-helpers-as-macros approach in our project at omisego/ewallet with satisfaction. It has worked well so far with the following benefits:

One-liner tests. Helps optimize valuable screen estate for browsing through the tests. Meanwhile the tests still have their explicity, not being hidden away behind some higher-level abstraction.
Not having to worry about human-error applying a change to the test behavior across the codebase. Even the test names are reflected by the macro.
When a test fails, its error pinpoints to the exact assertion. The error messages are very clear, and the test name represents the assertion exactly.
We're not hacking how ExUnit works, relying purely on ExUnit's public API.

If you find this approach interesting, you can find real-world examples of the helper macros at EWalletDB.SchemaCase, and usage at EWalletDB.RoleTest.

What do you think? Do you find any drawback or a better solution? Let me know!