DEV Community: Alex Reichert

Learning Elixir's GenServer with a real-world example

Alex Reichert — Wed, 09 Dec 2020 19:14:22 +0000

I started learning Elixir a few months ago, mostly through hacking on Papercups. I'm ashamed to say most of my Elixir education has been through trial and error, figuring things out as I go along. So this past week I decided to take some time off from Papercups to go a bit deeper into Elixir.

In particular, I was itching to learn more about handling concurrency in Elixir. This, of course, led me to GenServers.

Anyone who's moderately familiar with Elixir has probably at least heard of GenServer. (If you haven't, that's ok too!) Strictly speaking, it's one of those things you can get away with avoiding for a while, and still be reasonably productive while using something like the Phoenix framework.

But it's an incredible useful feature of the language that you can add to your toolbox! There are so many nice things you can do with it. For example, with GenServers, you can:

create a simple cache, buffer, or rate limiter
set up recurring tasks to run every few hours (e.g. check out Jose's answer to "How can I schedule code to run every few hours in Elixir?" on Stack Overflow)
handle async processes much more easily, with built-in functionality for tracing, error reporting, and retry logic

...all without any external dependencies!

What is a GenServer, anyway?

A GenServer is a process, just like any other Elixir process. It can be used to manage state and execute code asynchronously, and includes some handy functionality around tracing and error reporting.

Another way of thinking about it — a GenServer is like an isolated little box, with a few things inside of it:

its internal state (which can be any Elixir data structure, from a simple number to a complex map/struct),
a mailbox to process incoming messages (in the order they're received),
and a callback module, which defines the code to run depending on the message that is received in the mailbox

If that's confusing, hopefully a real-world example will help clarify things! 😉

Finding a good real-world example

While learning about GenServers, I found many of the examples to be a bit contrived... a lot of "key-value" stores, shopping lists, and things like that. I wanted to understand how GenServers are being used "out in the wild", so I turned to one of my favorite Elixir open-source repos: https://github.com/plausible/analytics

A bit of background: Plausible is an open-source analytics platform, with >1000 paying customers. Because they are a real-world production application serving many users, I figured I could learn a lot from diagnosing one of their modules. 🤓

For the sake of this post, I'm going to go through one of the simpler GenServers in their repo: Plausible.Event.WriteBuffer, which can be found at /lib/plausible/event/write_buffer.ex

What does it do?

At a high level, this GenServer allows Plausible to insert large quantities of events into the database in batches, rather than doing an insertion for each individual event. (This seems particularly important for a product like Plausible's, where they are potentially ingesting thousands of events per second!)

At the code level, what this process is doing is essentially:

ingesting events via the insert/1 method
adding them to a buffer in the process's internal state, represented by a list (i.e. [])
"flushing" the buffer every 5 seconds, or if it reaches its capacity of 10,000 events — whichever comes first
- where "flushing" means saving all the events in the buffer to a database (in this case, ClickHouse)

(For the full context, let's take a look at the codebase: we can see that the Plausible.Event.WriteBuffer GenServer is used in the PlausibleWeb.Api.ExternalController.event/2 method, which handles the logic for the POST /api/event API endpoint, which in turn gets called from their JavaScript tracking snippet.)

The code

Before we jump into it, here's the full module, with some minor stylistic tweaks and annotations I've added myself for some additional context:



defmodule Plausible.Event.WriteBuffer do
  use GenServer
  require Logger

  @flush_interval_ms 5_000
  @max_buffer_size 10_000

  # Client APIs

  def start_link(_opts) do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  def insert(event) do
    GenServer.cast(__MODULE__, {:insert, event})

    {:ok, event}
  end

  def flush() do
    GenServer.call(__MODULE__, :flush, :infinity)

    :ok
  end

  # Server (callbacks)

  @impl true
  def init(buffer) do
    Process.flag(:trap_exit, true)
    timer = Process.send_after(self(), :tick, @flush_interval_ms)

    {:ok, %{buffer: buffer, timer: timer}}
  end

  @impl true
  def handle_cast({:insert, event}, %{buffer: buffer, timer: timer} = _state) do
    new_buffer = [event | buffer]

    if length(new_buffer) >= @max_buffer_size do
      Logger.info("Buffer full, flushing to disk")
      Process.cancel_timer(timer)
      do_flush(new_buffer)
      new_timer = Process.send_after(self(), :tick, @flush_interval_ms)

      {:noreply, %{buffer: [], timer: new_timer}}
    else
      {:noreply, %{state | buffer: new_buffer}}
    end
  end

  @impl true
  def handle_info(:tick, %{buffer: buffer} = _state) do
    do_flush(buffer)
    timer = Process.send_after(self(), :tick, @flush_interval_ms)

    {:noreply, %{buffer: [], timer: timer}}
  end

  @impl true
  def handle_call(:flush, _from, %{buffer: buffer, timer: timer} = _state) do
    Process.cancel_timer(timer)
    do_flush(buffer)
    new_timer = Process.send_after(self(), :tick, @flush_interval_ms)

    {:reply, nil, %{buffer: [], timer: new_timer}}
  end

  @impl true
  def terminate(_reason, %{buffer: buffer} = _state) do
    Logger.info("Flushing event buffer before shutdown...")
    do_flush(buffer)
  end

  # Private/utility methods

  defp do_flush(buffer) do
    case buffer do
      [] ->
        nil

      events ->
        Logger.info("Flushing #{length(events)} events")
        events = Enum.map(events, &(Map.from_struct(&1) |> Map.delete(:__meta__)))
        Plausible.ClickhouseRepo.insert_all(Plausible.ClickhouseEvent, events)
    end
  end
end

Note: If you're able to read the code above and know exactly what's going on at every step, you probably don't need to read any further!

Let's start off where the core of the logic actually lives: in the GenServer callbacks.

The GenServer callbacks

The callbacks implemented in this particular module are:

init/1 (required) - invoked when the server is started, and sets the initial state
handle_cast/2 - invoked when GenServer.cast/2 is called, to handle messages asynchronously
handle_call/3 - invoked when GenServer.call/3 is called, to handle messages synchronously
- GenServer.call/3 will block until a reply is received (unless the call times out or nodes are disconnected)
handle_info/2 - invoked to handle all other messages (i.e. outside of those triggered by GenServer.call/3 and GenServer.cast/2, and "system" messages)
- e.g. messages/events handled internally within the GenServer
terminate/2 - invoked when the GenServer is about to exit

Let's go through each one! 🚀

The `init/1` callback

The init/1 callback is invoked when the GenServer is started, and handles setting the initial internal state of the server process.

Here's the method definition from above:



  @impl true
  def init(buffer) do
    Process.flag(:trap_exit, true)
    timer = Process.send_after(self(), :tick, @flush_interval_ms)

    {:ok, %{buffer: buffer, timer: timer}}
  end

The first line of this method (Process.flag(:trap_exit, true)) sets us up to "trap exits". This allows us to handle any "clean-up" tasks before the process terminates, in the terminate/2 callback. (We'll discuss this below!)

Next, we set up a timer to run the :tick event in 5 seconds (@flush_interval_ms == 5_000). As we'll see below, this :tick event also calls itself recursively, so that it effectively runs every 5 seconds while the server is alive. (That explains why it's called "tick"!)

Finally, we store the initial buffer (in this case, an empty list, i.e. []) and the timer in the internal state.

The `handle_cast/2` callback

The handle_cast/2 callback is invoked when GenServer.cast/2 is called, to handle asynchronous requests.

Here's the method definition from above:



  @impl true
  def handle_cast({:insert, event}, %{buffer: buffer, timer: timer} = _state) do
    new_buffer = [event | buffer]

    if length(new_buffer) >= @max_buffer_size do
      Logger.info("Buffer full, flushing to disk")
      Process.cancel_timer(timer)
      do_flush(new_buffer)
      new_timer = Process.send_after(self(), :tick, @flush_interval_ms)

      {:noreply, %{buffer: [], timer: new_timer}}
    else
      {:noreply, %{state | buffer: new_buffer}}
    end
  end

All this is doing is:

Inserting a new event into the internal state's buffer, by prepending it to the list
Checking if the buffer has reached capacity, and if so,
- cancels the current timer to prevent the next automatic flush from happening
- performs the flush operation manually with do_flush/1
- sets up a new timer which will trigger the next automatic flush in 5 seconds
- resets the internal state's buffer to an empty list, and updates the timer as well
If the buffer is not at capacity, it simply updates the internal state's buffer

The reason this is handled in a handle_cast/2 callback rather than a handle_call/3 callback is so that the client can execute this asynchronously, without blocking on the completion of a potential do_flush/1 method invocation. (As we'll see below, the "flushing"/saving of 10,000 events is a potentially expensive operation that could take a few seconds to complete.)

Let's take a quick look at the do_flush/1 method to see what's going on there:



  defp do_flush(buffer) do
    case buffer do
      [] ->
        nil

      events ->
        Logger.info("Flushing #{length(events)} events")
        events = Enum.map(events, &(Map.from_struct(&1) |> Map.delete(:__meta__)))
        Plausible.ClickhouseRepo.insert_all(Plausible.ClickhouseEvent, events)
    end
  end

This method is just standard Elixir. All it's doing is taking the events in the buffer, formatting them into plain maps (rather than structs), and then persisting them to the database (ClickHouse). (If we wanted a slightly more explicit name for this method, we could call it something like save_to_clickhouse/1 instead.)

Note that if we wanted to save some whitespace, this could also be written like:



  defp do_flush([]), do: nil

  defp do_flush(events) do
    Logger.info("Flushing #{length(events)} events")
    events = Enum.map(events, &(Map.from_struct(&1) |> Map.delete(:__meta__)))
    Plausible.ClickhouseRepo.insert_all(Plausible.ClickhouseEvent, events)
  end

Hooray for pattern matching!

The `handle_call/3` callback

The handle_call/3 callback is invoked when GenServer.call/3 is called, to handle synchronous messages. Note that GenServer.call/3 will block until a reply is received (unless the call times out or nodes are disconnected).

Here's the method definition from above:



  @impl true
  def handle_call(:flush, _from, %{buffer: buffer, timer: timer} = _state) do
    Process.cancel_timer(timer)
    do_flush(buffer)
    new_timer = Process.send_after(self(), :tick, @flush_interval_ms)

    {:reply, nil, %{buffer: [], timer: new_timer}}
  end

This code looks quite similar to the logic above, in the handle_cast/2 callback. In this case, we're allow the :flush event to be called manually, which takes whatever is currently in the state's buffer and saves it to the database (in the do_flush/1 method). Once again, the buffer is reset to an empty list, and timer is reset as well.

The `handle_info/2` callback

The handle_info/2 callback is invoked to handle all other messages (i.e. outside of those triggered by GenServer.call/3 and GenServer.cast/2). In our case, we use it to handle messages passed around internally within the GenServer itself.

Here's the method definition from above:



  @impl true
  def handle_info(:tick, %{buffer: buffer} = _state) do
    do_flush(buffer)
    timer = Process.send_after(self(), :tick, @flush_interval_ms)

    {:noreply, %{buffer: [], timer: timer}}
  end

The :tick handles flushing the events in our buffer at a regular interval. (In this case, every 5 seconds.)

It does this by recursively calling itself with Process.send_after(self(), :tick, @flush_interval_ms)

The `terminate/2` callback

This callback is invoked when the server is about to exit, and can handle any clean-up tasks.

Here's the method definition from above:



  @impl true
  def terminate(_reason, %{buffer: buffer} = _state) do
    Logger.info("Flushing event buffer before shutdown...")
    do_flush(buffer)
  end

By trapping exists in the init/1 callback above, we can ensure that before this GenServer process terminates, we flush anything that's left in our buffer. Otherwise, we might end up in a situation where the process is terminated before some of the events in memory don't have a chance to get saved to the database, and would be lost.

The GenServer Client APIs

The APIs exposed to the client are mostly boilerplate, and pretty straightforward:



  def start_link(_opts) do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  def insert(event) do
    GenServer.cast(__MODULE__, {:insert, event})

    {:ok, event}
  end

  def flush() do
    GenServer.call(__MODULE__, :flush, :infinity)

    :ok
  end

Let's break it down!

`Plausible.Event.WriteBuffer.start_link/1`

This is probably the least intuitive of the three methods above — once we understand the code here, the rest should more or less fall into place.



  def start_link(_opts) do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

As you might guess, the start_link/1 method is responsible for, well, starting the GenServer process. All it's doing is calling start_link/3 on GenServer itself, with a few strange arguments:

__MODULE__, which is simply an alias for the module itself (in this case, Plausible.Event.WriteBuffer). This first argument passes the Plausible.Event.WriteBuffer in as the callback module, so that we can use the server callbacks we implemented above when certain messages are passed to the server process via GenServer.call and GenServer.cast.
- Note that calling GenServer.start_link(Plausible.Event.WriteBuffer, [], name: Plausible.Event.WriteBuffer) would result in identical behavior. The main advantage of using __MODULE__ is if we decide to rename the module, we only have to update it in one place!
An empty list (i.e. []) is passed in as the second argument. This is passed to our init/1 callback as the default buffer. We'll take a look at this in more detail below.
The last argument is a keyword list of options. Here, we assign the process a name with name: MODULE (which is the same as name: Plausible.Event.WriteBuffer). This allows us to pass messages to the GenServer using the module name (i.e GenServer.call(MODULE, message, timeout)/GenServer.cast(MODULE, message)). Otherwise, after starting the GenServer, we would have to keep a reference to its process ID (or "PID", which is returned in the start_link/1 method), and then pass it in explicitly. (This is a very common practice.)

(GenServer also has a method called simply start/3, which takes the same arguments as start_link/3 — the only difference is that start_link/3 "links" the GenServer process to the current process, which is useful when we want to start the GenServer as part of a "supervision tree". By starting a process with start_link under a Supervisor, it allows the application to monitor the process for any errors/crashes, and automatically restart the process if necessary.)

In practice, this is where the process is started: https://github.com/plausible/analytics/blob/master/lib/plausible/application.ex

Here's the relevant code below:



defmodule Plausible.Application do
  use Application

  def start(_type, _args) do
    children = [
      # ...
      Plausible.Event.WriteBuffer,
      # ...
    ]

    opts = [strategy: :one_for_one, name: Plausible.Supervisor]
    # ...
    Supervisor.start_link(children, opts)
  end

  # ...
end

GenServer's start_link/3 method is what enables us to start the process under this Supervisor. (When we include Plausible.Event.WriteBuffer amongst the children of the supervisor, its start_link/3 method is invoked automatically when Supervisor.start_link/2 is called.)

Hopefully at this point, we have a pretty good understanding of what's going on in Plausible.Event.WriteBuffer.start_link/1! Now all that's left to cover are the two methods exposed in our client API: insert/1 and flush/0

`Plausible.Event.WriteBuffer.insert/1`

One last time, here's the insert/1 method:



  def insert(event) do
    GenServer.cast(__MODULE__, {:insert, event})

    {:ok, event}
  end

Here we see that all this is doing is taking an event and sending an asynchronous request to our GenServer via the cast/2 method. (Since we're using cast/2 here instead of call/3, we know that this is a non-blocking call which will return immediately, regardless of what happens in the callback.)

The first argument of GenServer.cast/2 takes a process ID (i.e. PID) or a registered server name. In this case, we set the name to __MODULE__ in the 3rd argument of our GenServer.start_link/3 above, which is why we're using that here. (This is a very common practice with GenServers.)

The second argument is the "message" we want to send, often in the form of a tuple or an atom. Here we're sending {:insert, event}, which will be handled in the handle_cast({:insert, event}, ...) callback discussed above.

Since we don't wait for a return value, the insert/1 method just returns an echo of the event that was passed in, in the form of {:ok, event}.

`Plausible.Event.WriteBuffer.flush/0`

We're almost done! Let's take a look at the flush/0 method:



  def flush() do
    GenServer.call(__MODULE__, :flush, :infinity)

    :ok
  end

This is quite similar to what we saw in insert/1. The only differences are:

GenServer.call/3, unlike GenServer.cast/2, blocks the process until the callback completes. So if the :flush event ends up taking 10 seconds to execute, the flush/0 method will not return :ok until 10 seconds have passed.
call/3 takes a third argument, which we see here is :infinity. This represents the maximum amount of time we allow the method to take before a timeout error occurs. By default, this is set to 5_000, or 5 seconds. By passing in :infinity, we're allowing the process to take as long as it needs to complete.

Reviewing it all

Let's take one last look at the full module definition, to double check that we understand everything that's going on. I've added some comments in the code to help out a bit. 😉



defmodule Plausible.Event.WriteBuffer do
  use GenServer
  require Logger

  # Flush/save every 5 seconds
  @flush_interval_ms 5_000

  # Allow a maximum of 10,000 events in our buffer list
  @max_buffer_size 10_000

  ############################################################################
  # Client APIs
  ############################################################################

  @doc """
  Starts a linked GenServer process, passing in the current module
  (`__MODULE__` == `Plausible.Event.WriteBuffer`) as both the callback module
  and the alias/name of the server process for future reference.

  We also pass in an empty list (`[]`) as the initial value of the internal
  state's `buffer` (handled in the `init/1` callback below).
  """
  def start_link(_opts) do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  @doc """
  Sends an `:insert` message asynchronously to the current module's GenServer
  process, which adds the provided `event` to the internal state's `buffer`.
  """
  def insert(event) do
    GenServer.cast(__MODULE__, {:insert, event})

    {:ok, event}
  end

  @doc """
  Sends a (synchronous) `:flush` message to the current module's GenServer
  process, which manually "flushes" all events in the internal state's `buffer`
  to the database.

  Sets the `timeout` option to `:infinity`, allowing this call to take as long
  as it needs to complete.
  """
  def flush() do
    GenServer.call(__MODULE__, :flush, :infinity)

    :ok
  end

  ############################################################################
  # Server callbacks
  ############################################################################

  @doc """
  Sets up the initial state of the GenServer, and traps exits so that we can
  gracefully handle a process termination in our `terminate/2` callback below.
  """
  @impl true
  def init(buffer) do
    Process.flag(:trap_exit, true)
    timer = Process.send_after(self(), :tick, @flush_interval_ms)

    {:ok, %{buffer: buffer, timer: timer}}
  end

  @doc """
  Callback that handles `:insert` messages passed to `GenServer.cast/2`.

  In this callback, we add the new event to the internal state's `buffer`.
  If we reach the maximum capacity of the `buffer`, we "flush" it to the database
  and reset our state. Otherwise, we simply update the internal state with the
  updated `buffer`.
  """
  @impl true
  def handle_cast({:insert, event}, %{buffer: buffer, timer: timer} = _state) do
    new_buffer = [event | buffer]

    if length(new_buffer) >= @max_buffer_size do
      Logger.info("Buffer full, flushing to disk")
      Process.cancel_timer(timer)
      do_flush(new_buffer)
      new_timer = Process.send_after(self(), :tick, @flush_interval_ms)

      {:noreply, %{buffer: [], timer: new_timer}}
    else
      {:noreply, %{state | buffer: new_buffer}}
    end
  end

  @doc """
  Callback that handles internal `:tick` messages.

  In this callback, we automatically flush whatever is in the `buffer`
  every 5 seconds (i.e. `@flush_interval_ms`).
  """
  @impl true
  def handle_info(:tick, %{buffer: buffer} = _state) do
    do_flush(buffer)
    timer = Process.send_after(self(), :tick, @flush_interval_ms)

    {:noreply, %{buffer: [], timer: timer}}
  end

  @doc """
  Callback that handles `:flush` messages passed to `GenServer.call/3`.

  In this callback, we handle manually flushing whatever is currently in
  the internal state's `buffer` to the database.
  """
  @impl true
  def handle_call(:flush, _from, %{buffer: buffer, timer: timer} = _state) do
    Process.cancel_timer(timer)
    do_flush(buffer)
    new_timer = Process.send_after(self(), :tick, @flush_interval_ms)

    {:reply, nil, %{buffer: [], timer: new_timer}}
  end

  @doc """
  Callback that gets invoked when the process is about to shut down.

  In this callback, we make sure to flush whatever is currently in the internal
  state's `buffer` to the database before the process terminates.
  """
  @impl true
  def terminate(_reason, %{buffer: buffer} = _state) do
    Logger.info("Flushing event buffer before shutdown...")
    do_flush(buffer)
  end

  ############################################################################
  # Private/utility methods
  ############################################################################

  @doc """
  Perform a "flush" of the events in the `buffer` to the database. By "flush",
  we simply mean we do a bulk insertion of the events into ClickHouse (the db).
  """
  defp do_flush(buffer) do
    case buffer do
      [] ->
        nil

      events ->
        Logger.info("Flushing #{length(events)} events")
        events = Enum.map(events, &(Map.from_struct(&1) |> Map.delete(:__meta__)))
        Plausible.ClickhouseRepo.insert_all(Plausible.ClickhouseEvent, events)
    end
  end
end

If you still don't understand something or think I missed something, feel free to reach out to me on our public Slack channel. 😬

That's all for now!

If you're interested in contributing to an open source Elixir project, come check out Papercups on Github!

Coming to Elixir from TypeScript

Alex Reichert — Mon, 21 Sep 2020 22:46:33 +0000

I've been working with Elixir for about 2 months so far, and it's been quite fun. Coming from a background in mostly TypeScript/JavaScript and Ruby, I wasn't sure how approachable I would find it.

A lot of articles I've read say that most Ruby developers would feel comfortable getting started with Elixir, but I'm not sure how much I agree with that. Aside from some superficial similarities, Elixir really forces you to think about solving problems in a slightly different way.

Over the course of my career so far, I've dabbled in programming languages unrelated to the jobs I've been paid for, but this was the first time I really learned a language by jumping right in and attempting to build a full-stack application. I'm a little ashamed to say that I've spent relatively little time going through books on Elixir, and have mostly just gone straight to hacking on our product. That being said, a lot of the opinions below come from the perspective of someone who probably hasn't written much high-quality Elixir code in production. 😬

What I like so far

Here are a few of the things that make me excited about working with Elixir. 😊

The community

This is an easy one. One of the first things I did when I started spinning up on Elixir was joining the Elixir Slack group, and it's been one of the most helpful resources for me as a beginner. The community has been nothing but friendly, patient, and supportive. When I was misusing with statements, they showed me how to refactor it. When I was starting to set up authentication, they pointed me to Pow. When I needed to set up workers, they showed me Oban. People have even been nice enough to review some of my shitty code on Github. It's been amazing.

The extensive built-in functionality

It's kind of nice just having so many useful functions built into the language. Want to flatten an array? Boom, List.flatten(). No need to import {flatten} from 'lodash'. Need to group a list of records by a given key? Boom, Enum.group_by(). I could go on and on!

I especially love that lists, maps, and ranges all implement the Enum protocol. For example, if I wanted to map over an object/map in JavaScript and double each value, I'd have to do something like:

const obj = {a: 1, b: 2, c: 3};

const result = Object.keys(obj).reduce((acc, key) => {
  return {...acc, [key]: obj[key] * 2};
}, {});

// {a: 2, b: 4, c: 6}

Whereas in Elixir, I could just do:

map = %{a: 1, b: 2, c: 3}

result = map |> Enum.map(fn {k, v} -> {k, v * 2} end) |> Map.new()

# %{a: 2, b: 4, c: 6}

Edit: Apparently there's an even easier way to handle this using Map.new/2! (Thanks to /u/metis_seeker on Reddit for the tip 😊)

Map.new(map, fn {k, v} -> {k, v * 2} end)

# %{a: 2, b: 4, c: 6}

Lastly, I love that there are methods like String.jaro_distance/2, which calculates the distance/similarity between two strings. I don't currently use it, but I could see how this might be useful for validating email address domains (e.g. foo@gmial.com -> "Did you mean foo@gmail.com?")

Pattern matching

Pattern matching feels like one of more powerful features Elixir offers as language. While it certainly takes some getting used to, I've found that it forces me to write cleaner code. (It's also caused me to write more case statements and much fewer if clauses than I ever have before!)

For example, if I wanted to write a method in Elixir that determines if a user has a given role (e.g. for the sake of restricting access to certain functionality), I might do something like this:

defp has_role?(nil, _roles), do: false

defp has_role?(user, roles) when is_list(roles),
  do: Enum.any?(roles, &has_role?(user, &1))

defp has_role?(%{role: role}, role), do: true

defp has_role?(_user, _role), do: false

(Note the additional use of pattern matching in the 3rd variant of has_role?/2 to check if the user.role in the 1st parameter is the same as the role provided in the 2nd parameter!)

In TypeScript, the (very rough) equivalent of the above might look something like:

const hasRole = (user: User, roleOrRoles: string | Array<string>) => {
  if (!user) {
    return false;
  }

  // This is probably not the most idiomatic TS/JS code :/
  const roles = Array.isArray(roleOrRoles) ? roleOrRoles : [roleOrRoles];

  return roles.some((role) => user.role === role);
};

Still confused? I don't blame you. Here's the Elixir code again, with some annotations:

# If the user is `nil`, return false
defp has_role?(nil, _roles), do: false

# Allow 2nd argument to be list or string; if it is a list, check
# if any of the values match by applying method recursively to each one
defp has_role?(user, roles) when is_list(roles),
  do: Enum.any?(roles, &has_role?(user, &1))

# Use pattern matching to check if the `user.role` matches the `role`
defp has_role?(%{role: role}, role), do: true

# If none of the patterns match above, fall back to return false
defp has_role?(_user, _role), do: false

This approach has taken some getting used to, but it's definitely growing on me. For example, one pattern I've started using to roll out new features (e.g. Slack notifications) is something like this:

def notify(msg), do: notify(msg, slack_enabled?())

# If Slack is not enabled, do nothing
def notify(msg, false), do: {:ok, nil}

# If it _is_ enabled, send the message
def notify(msg, true), do: Slack.post("/chat.postMessage", msg)

Not sure how idiomatic that is, but it's a nice way to avoid if blocks!

Async handling

A lot of JavaScript is conventionally handled asynchronously (non-blocking) by default. This can be a bit tricky for new programmers, but it can be quite powerful once you get the hang of it (e.g. Promise.all is a nice way to execute a bunch of async processes concurrently).

Elixir is handled synchronously (blocking) by default — which makes things much easier, in my opinion — but Elixir also happens to make it incredibly easy to handle processes asynchronously if you would like to.

As a somewhat naive example, when I was setting up our Messages API, I noticed it slowing down as we added more and more notification side effects (e.g. Slack, Webhooks) whenever a message was created. I loved that I could temporarily fix this issue by simply throwing the logic into an async process with a Task:

Task.start(fn -> Papercups.Webhooks.notify(message))

Now, this is definitely not the most ideal way to handle this. (It would probably make more sense to put it on a queue, e.g. with Oban.) But I loved how easy it was to unblock myself.

If we wanted to implement something similar to JavaScript's Promise.all, Elixir gives us something even better: control over timeouts!

tasks = [
  Task.async(fn -> Process.sleep(1000) end), # Sleep 1s
  Task.async(fn -> Process.sleep(4000) end), # Sleep 4s
  Task.async(fn -> Process.sleep(7000) end)  # Sleep 7s, will timeout
]

tasks
|> Task.yield_many(5000) # Set timeout limit to 5s
|> Enum.map(fn {t, res} -> res || Task.shutdown(t, :brutal_kill) end)

This allows us to shutdown any processes that are taking longer than expected. 🔥

The pipe operator

It's almost as if any blog post introducing Elixir is obligated to mention this, so here we are.

Let's just take an example directly from the Papercups codebase. In one of our modules, we do some email validation by checking the MX records of the given domain. Here's how it looks in Elixir:

defp lookup_all_mx_records(domain_name) do
  domain_name
  |> String.to_charlist()
  |> :inet_res.lookup(:in, :mx, [], max_timeout())
  |> normalize_mx_records_to_string()
end

If I wanted to write this in TypeScript, I would probably do something like:

const lookupAllMxRecords = async (domain: string) => {
  const charlist = domain.split('');
  const records = await InetRes.lookup(charlist, opts);
  const normalized = normalizeMxRecords(records);

  return normalized;
};

There's nothing inherently wrong with that, but pipes save us some unhelpful variable declarations, and produce code that is arguably just as readable!

I think the thing people like most about the pipe operator is that it both looks cool AND improves (or at least doesn't detract from) readability. But mostly it just looks cool. 🤓

Since I wasn't able to write anything particulary intelligent about pipes, I'll leave this section with a quote from Saša Juric's "Elixir in Action":

The pipeline operator highlights the power of functional programming. You treat functions as data transformations and then combine them in different ways to gain the desired effect.

Immutability

I can't tell you how many times I've been writing JavaScript and forgotten that calling .reverse() or .sort() on an array actually mutates the original value. (This almost screwed me over in my last technical interview, embarrassingly enough.)

For example:

> const arr = [1, 6, 2, 5, 3, 4];
> arr.sort().reverse()
[ 6, 5, 4, 3, 2, 1 ]
> arr
[ 6, 5, 4, 3, 2, 1 ] // arr was mutated 👎

I love that in Elixir, everything is immutable by default. So if I define a list, and want to reverse or sort it, the original list never changes:

iex(12)> arr = [1, 6, 2, 5, 3, 4]
[1, 6, 2, 5, 3, 4]
iex(13)> arr |> Enum.sort() |> Enum.reverse()
[6, 5, 4, 3, 2, 1]
iex(14)> arr
[1, 6, 2, 5, 3, 4] # nothing has changed 👌

Hooray! This makes the code much more predictable.

Dealing with strings

I love that there are so many ways to format and interpolate strings in Elixir. This might be a bit of a niche use case, but the triple-quote """ approach has been super useful for email text templates, since it removes all the preceding whitespace from each line:

def welcome_email_text(name) do
  """
  Hi #{name}!

  Thanks for signing up for Papercups :)

  Best,
  Alex
  """
end

If I wanted to do this in TypeScript, I'd have to do something like:

const welcomeEmailText = (name: string) => {
  return `
Hi ${name}!

Thanks for signing up for Papercups :)

Best,
Alex
  `.trim();
};

Which just looks... awkward.

What I'm... still getting used to

I almost called this section, "What I dislike so far", but I thought that would be a little unfair. Just because I'm not accustomed to certain ways of thinking doesn't mean I have to hate on it.

So without further ado, here are some of the things I'm still getting used to with Elixir. 😬

Error handling

One of the first things I noticed when I started to dip my toes in Elixir was the prevalence of methods returning {:ok, result}/{:error, reason} tuples. I didn't give it much thought at first, and found myself writing a lot of code that looked like:

{:ok, foo} = Foo.retrieve(foo_id)
{:ok, bar} = Bar.retrieve(bar_id)
{:ok, baz} = Baz.retrieve(baz_id)

...and then got hit with a bunch of MatchErrors.

As you might have guessed (if you've written any Elixir), this led me to start getting a little overly enthusastic about the with statement. Which if you haven't written any Elixir, looks something like this:

with {:ok, foo} <- Foo.retrieve(foo_id),
     {:ok, bar} <- Bar.retrieve(bar_id),
     {:ok, baz} <- Baz.retrieve(baz_id) do
  # Do whatever, as long as all 3 methods above execute without error
else
  error -> handle_error(error)
end

There's nothing particularly wrong with that, but I've also found myself writing some methods that basically just extract the result portion of the {:ok, result} tuple, which feels a little silly:

case Foo.retrieve(foo_id) do
  {:ok, foo} -> foo
  error -> error
end

(It's very possible that the above code is an antipattern, and I'm simply not handling things correctly.)

Anyway, on one hand, I feel like this convention of the language is good because it forces programmers to be more cognizant of error handling. But on the other hand, it definitely takes some getting used to.

Implicit returns (and no `return` keyword)

While pattern matching is great and all, the fact that Elixir does not have the ability to break out of a function early can be a bit frustrating as a beginner.

For example, if I wanted to write a function to compute the total cost of a bill in TypeScript, I might do something like:

const calculateTotalPrice = (bill: Bill) => {
  if (!bill) {
    return 0;
  }

  const {prices = []} = bill;

  // This is a little unnecessary, but illustrates the point of
  // a second reason we may want to return early in a function
  if (prices.length === 0) {
    return 0;
  }

  return prices.reduce((total, price) => total + price, 0);
};

The code above allows me to break early and return 0 under certain circumstances (e.g. when bill is null, or prices is an empty list).

Elixir solves this with pattern matching (as we've discussed in more detail above).

def calculate_total_price(nil), do: 0

def calculate_total_price(%{prices: prices}) when is_list(prices),
  do: Enum.sum(prices)

def calculate_total_price(_bill), do: 0

For someone approaching Elixir as a newbie like myself, this can take some getting used to, because it forces you to take a step back and rethink how you would normally design your functions.

Dialyzer and the development experience

There's not much to say here, other than that Dialyzer can be pretty frustrating to deal with at times. Sometimes it's just slow, and warnings take a few seconds to pop up... this is annoying when I: change some code to fix a warning; the warning goes away for a few seconds; I feel good about myself for having fixed it; and then boom, another warning pops up.

Other times, the warnings and just cryptic or confusing:

(I have no idea what this means...)

Debugging macros

When I was starting off with the Pow library to implement auth, I ran into Elixir macros for the first time. I felt like such an idiot trying to figure out where the pow_password_changeset method was defined, until I finally found this piece of code:

@changeset_methods [:user_id_field_changeset, :password_changeset, :current_password_changeset]

# ...

for method <- @changeset_methods do
  pow_method_name = String.to_atom("pow_#{method}")

  quote do
    @spec unquote(pow_method_name)(Ecto.Schema.t() | Changeset.t(), map()) :: Changeset.t()
    def unquote(pow_method_name)(user_or_changeset, attrs) do
      unquote(__MODULE__).Changeset.unquote(method)(user_or_changeset, attrs, @pow_config)
    end
  end
end

It's pretty cool that Elixir supports macros, but the syntax and the idea of dynamically generating methods is not something I've ever had to deal with. But I'm excited to try it out!

Dealing with JSON

Honestly, I feel like this is true for most languages (other than JavaScript/TypeScript). Since most maps in Elixir use atoms for keys, I've found myself accidentally mixing atom/string keys when I'm unknowing working with a map that has been decoded from JSON.

Unclear trajectory of the language

I honestly have no idea whether Elixir is growing, stagnating, or declining in popularity, but so far things seem much more enjoyable and less painful than I expected.

When we first started building Papercups in Elixir, a few people warned us that the lack of libraries and support would make it much harder to move quickly. While it's clear that the amount of open source libraries is much lower compared to languages like JavaScript, Ruby, Python, and Go, so far this hasn't been a huge issue.

As more well-known companies (e.g. WhatsApp, Discord, Brex) begin using Elixir in production, I'm hoping developer adoption continues to grow. I'm optimistic! 😊

That's all for now!

If you're interested in contributing to an open source Elixir project, come check out Papercups on Github!

Slack notifications with NodeJS in <2 minutes

Alex Reichert — Fri, 12 Jun 2020 03:23:14 +0000

When you’re working on a project, it can be useful to set up a simple notification system for a number of reasons:

You want to be notified whenever a new user signs up 🚀
You want to be alerted when a user encounters an unexpected error 🤔
You want to be reminded every morning to fill out your TODO list

If you’re part of a team, Slack is a great place to notify so that everyone is on the same page!

Getting started

We’re going to use Taro to handle setting up this integration. Taro is a tool we’ve been working on, designed to make developer’s lives easier when it comes to this kind of thing.

To start, create your free account at https://www.gettaro.com/ by clicking the “Get started” button and signing up.

Connecting to Slack

Once you’re in the dashboard, click on the “Integrations” button in the upper-right corner of the page. You should see something like this:

At the bottom of the page, click on “Connect with Slack”, and complete that flow with the Slack channel you’d like to notify.

Once that’s done, grab your API key by clicking “Show” next to the API key input above. You’ll need that in a minute!

Creating a Slack notification in NodeJS

Now that you’ve set up a link to your Slack channel and gotten your API key, you can either set up a new Node project, or use an existing one. For the sake of this guide, let’s create a quick new one.

In a new project directory, run npm init, and keep pressing enter until your package.json file is set up with all the default values. It should look something like this (mine happened to be in a directory called “sandbox”):

Then, create an index.js file, and paste in the code below — remember to replace __YOUR_API_KEY__ with your API key from above 🤓

Run the code

In your terminal, run node index.js — you should receive a notification in Slack! 🎉

That’s it!

By the way — if you’re having trouble with any of these steps, don’t hesitate to leave a comment or reach out at reichertjalex@gmail.com.

Making cron jobs fun again: a tool for simple recurring tasks

Alex Reichert — Tue, 02 Jun 2020 22:20:31 +0000

Recently, a buddy and I built a little tool that makes it super easy to deploy and monitor simple recurring jobs in NodeJS.

Why is this useful?

As developers, every now and then we come up with cool ideas of things to automate. I remember a few years back when I first started learning how to code and was looking for an apartment in San Francisco, I felt like the world’s cleverest hacker when I set up a script to email every new listing on CraigsList in the neighborhoods I was interested in living in. (This had some disastrous consequences, but that’s a story for another time.)

My only problem at the time was that I would have to manually run this script myself every day.

Now, as a more experienced developer, I know enough to know that one approach to this problem is to set up a cron job, or something similar. Recently I found myself wanting to do this again… but every time I want to do it, it takes some effort to remind myself how it all works. At first I thought, maybe I’m just an idiot. But after talking to a few friends, we realized that a bunch of us have the exact same issue.

So we figured, why not build a little tool that makes this process extremely simple? This is what led to the creation of Taro: https://www.gettaro.com/

We'd love to hear the community's thoughts on integrations that would be particularly useful (e.g. alerting with SMS, lightweight storage, etc) as we continue to hack on this!

(P.S. If you're looking for examples of scripts to run with something like this, I put together a little list of examples here!)

DEV Community: Alex Reichert

Learning Elixir's GenServer with a real-world example

What is a GenServer, anyway?

Finding a good real-world example

What does it do?

The code

The GenServer callbacks

The init/1 callback

The handle_cast/2 callback

The handle_call/3 callback

The handle_info/2 callback

The terminate/2 callback

The GenServer Client APIs

Plausible.Event.WriteBuffer.start_link/1

Plausible.Event.WriteBuffer.insert/1

Plausible.Event.WriteBuffer.flush/0

Reviewing it all

That's all for now!

Coming to Elixir from TypeScript

What I like so far

The community

The extensive built-in functionality

Pattern matching

Async handling

The pipe operator

Immutability

Dealing with strings

What I'm... still getting used to

Error handling

Implicit returns (and no return keyword)

Dialyzer and the development experience

Debugging macros

Dealing with JSON

Unclear trajectory of the language

That's all for now!

Slack notifications with NodeJS in <2 minutes

Getting started

Connecting to Slack

Creating a Slack notification in NodeJS

Run the code

Making cron jobs fun again: a tool for simple recurring tasks

The `init/1` callback

The `handle_cast/2` callback

The `handle_call/3` callback

The `handle_info/2` callback

The `terminate/2` callback

`Plausible.Event.WriteBuffer.start_link/1`

`Plausible.Event.WriteBuffer.insert/1`

`Plausible.Event.WriteBuffer.flush/0`

Implicit returns (and no `return` keyword)