DEV Community: Lasse Skindstad Ebert

TIL: Unicode chars in regex

Lasse Skindstad Ebert — Fri, 25 Oct 2019 06:40:37 +0000

Today I needed to match some unicode chars in an Elixir regex.

TL;DR:

Use u modifier and \x{...}, e.g. ~r/\x{1234}/u

Matching a unicode char in a regex

More specifically, I needed to remove all zero width chars from a string.
These are U+200B, U+200C, U+200D and U+FEFF.

Trying to use \u does not work:

iex(1)> ~r/\u200B/
** (Regex.CompileError) PCRE does not support \L, \l, \N{name}, \U, or \u at position 1
    (elixir) lib/regex.ex:209: Regex.compile!/2
    (elixir) expanding macro: Kernel.sigil_r/2
    iex:1: (file)

Looking at the docs, it seems that \x{} is the way to go, but no:

iex(1)> ~r/\x{200B}/
** (Regex.CompileError) character value in \x{} or \o{} is too large at position 7
    (elixir) lib/regex.ex:209: Regex.compile!/2
    (elixir) expanding macro: Kernel.sigil_r/2
    iex:1: (file)

The trick is that we need to apply a unicode (u) modfier to the regex, telling the regex compiler that we're working in Unicode:

iex(1)> ~r/\x{200B}/u
~r/\x{200B}/u
iex(2)> "Hello,\u200BWorld!" |> String.replace(~r/\x{200B}/u, "")    
"Hello,World!"

Yay!

So my final regex could be something like:

~r/\x{200B}|\x{200C}|\x{200D}|\x{FEFF}/u

Interpolation works too.

We can also interpolate strings into a regex, which works the same way and works without the u modifer:

iex(5)> "Hello,\u200BWorld!" |> String.replace(~r/#{"\u200B"}/, "")
"Hello,World!"

Enum.map_reduce/3

Lasse Skindstad Ebert — Tue, 22 Oct 2019 11:52:53 +0000

Today I found a function in Elixir's standard lib that I have often needed.

Introducing: Enum.map_reduce/3

Enum.map_reduce/3 can replace Enum.reduce/3 when the reduce maps each element to another element and we also want to maintain some state or build a result along the way.

This might be a bit cryptic, so let's start with an example.

Have a cache while mapping

Let's pretend I don't know about Enum.map_reduce/3 and I need to map a list of numbers to the maximum of the number and the previous number.

I would do something like this:

input = [5, 2, 8, 9, 1, 2, 7, 2]

{_last, numbers} =
  Enum.reduce(input, {0, []}, fn num, {last, acc} ->
    {num, [max(num, last) | acc]}
  end)

Enum.reverse(numbers)

(Ok, this simple example could also be solved nicely with Enum.chunk_every/2 and Enum.map/2, but for the sake of example let's play with this)

This will give the expected result [5, 5, 8, 9, 9, 2, 7, 7].

Notice that I need to reverse the number list at the end, since I append numbers to the start of the list.

If only there is a way to do this in a smarter way...

Well, with Enum.map_reduce/3 we can do this:

input = [5, 2, 8, 9, 1, 2, 7, 2]

{numbers, _last} =
  Enum.map_reduce(input, 0, fn num, last -> {max(num, last), num} end)

So here we use map_reduce's accumulator to remember what the last number was.

Build a result while mapping

We can also use it to accumulate something we need as a result while we're mapping a list.

Perhaps we want to know the largest number while doing the max-thing above:

{numbers, {max, _last}} =
  Enum.map_reduce(input, {0, 0}, fn num, {max, last} ->
    {max(num, last), {max(num, max), num}}
  end)

Which will give the same numbers as before, but now we also get 9 as the max item.

Enum.flat_map_reduce

In the cases where each iteration might yield zero or many items to the resulting list, we can use Enum.flat_map_reduce/3. This works like Enum.map_reduce/3 with two differences:

Each iteration must yield a list of elements instead of a single element. Notice that we can use an empty list if an iteration should not add something to the resulting list.
It is possible to break early using :halt, almost like with Enum.reduce_while/3

Example: We want to map a list to another list where:

All odd numbers are removed
All even numbers are doubled
We want to sum the original numbers
If we hit 7, we stop

input = [5, 2, 8, 9, 1, 2, 7, 2]

Enum.flat_map_reduce(input, 0, fn num, sum ->
  case num do
    7 -> {:halt, sum}
    n when rem(n, 2) == 0 -> {[n, n], sum + n}
    n -> {[], sum + n}
  end
end)

Which will give {[2, 2, 8, 8, 2, 2], 27}

TIL: Ecto's pin is coercing

Lasse Skindstad Ebert — Mon, 21 Oct 2019 12:45:48 +0000

TL;DR

This:

query |> where([account], account.owner_id == "42")

is not the same as this:

owner_id = "42"
query |> where([account], account.owner_id == ^owner_id)

In the latter case, the value "42" is coerced to the integer 42 (if the owner_id of an account is an integer according to the Ecto schema).

Background

Today I found a bug in some code I wrote a long time ago.

The code was supposed to take a map of atom => String.t and convert some of the values to integers. But it didn't. And my other code dependent on this still ran without errors!

Time to pick up the Sherlock Holmes monocle and find out why this bug never yielded an actual error.

It turns out I later used the value like so in Ecto:

query |> where([account], account.owner_id == ^owner_id)

The account's owner_id is an integer, but I try to compare it with a string, because of the above bug. And it does not fail!?

I opened up an iex and tried this:

iex> Account |> where([account], owner_id: "42") |> Repo.all()
** (Ecto.QueryError) iex:2: value `"42"` cannot be dumped to type :id.
   Or the value is incompatible or it must be interpolated (using ^) so it may be cast
   accordingly in query:

from a0 in Account,
  where: a0.owner_id == "42",
  select: a0

    (elixir) lib/enum.ex:1440: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3 
    (elixir) lib/enum.ex:1440: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
    (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3

This fails. Ah, finally some sanity :)

The error message also suggests that I might need a pin operator to interpolate the value. So I tried this:

iex> Account |> where([account], owner_id: ^"42") |> Repo.all()
[]

And this is why my code did not fail. It was simply ignoring that I had to do with a string and coerced it into a integer.

I thought the pin-operator was only for inserting values of variables, but now I know it also acts as a coercer (or interpolator).

TIL: Understanding dialyzer’s “The pattern can never match the type.”

Lasse Skindstad Ebert — Sun, 20 Oct 2019 19:49:52 +0000

This story was originally posted by me on Medium on 2019-09-03, but moved here since I'm closing my Medium account.

I like dialyzer and typespecs. They are a natural part of my TDD flow:

Write a failing test
Write docs for function
Write typespecs for function
Write the function
See that test is now green
Refactor
See that test is still green

Having written tests, docs and a spec for a function, makes it much more clear what the function should accept as arguments and what it should return. I also tend to write more reusable functions with this flow.

The typespecs will also function as documentation when reading the function.

But dialyzer has a downside. Debugging dialyzer warnings can be a real pain and it is often not clear where to look for a wrong typespec or whatever made dialyzer vomit all over my terminal with less-than-obvious messages.

I spend a good three hours today debugging a dialyzer warning in my Elixir app, so I’m naturally obligated to share my insights with the world :)

The setup

I got an error in a with statement much like this one in function c/0:

defmodule DialyzerTest do
  @spec a() :: :ok | :error
  def a do
    [:ok, :error] |> Enum.random()
  end

  @spec b() :: :ok | :another_error
  def b do
    [:ok, :another_error] |> Enum.random()
  end

  @spec c() :: :ok | :error
  def c do
    with :ok <- a(),
         :ok <- b() do
      :ok
    else
      :error -> :error
    end
  end
end

The dializer warning says:

simple_dialyzer_test.ex:14:pattern_match
The pattern can never match the type.

Pattern:
:error

Type:
:another_error

This is a pretty light-weight warning compared to dialyzer standards. Normally I need to copy-paste the message into an editor to format it and be able to make a little sense out of it.

So apparently :error does not match :another_error, which is obvious, but why complain about it? When I wrote the code, I called b/0 in a way that shouldn’t make it return anything else than :ok, so I would like a runtime exception if anything else is returned from that function.

My mistake was that I considered the above example the same as this one:

defmodule DialyzerTest do
  @spec a() :: :ok | :error | :another_error
  def a do
    [:ok, :error, :another_error] |> Enum.random()
  end

  @spec c() :: :ok | :error
  def c do
    with :ok <- a() do
      :ok
    else
      :error -> :error
    end
  end
end

Notice that we only match on :ok and :error, but not on :another_error. This code does not produce a dialyzer warning.

What the hell is going on?

After a long time of debugging and trying to find a wrong typespec, I realized how dialyzer handles with: For each statement in the with (all the lines with a <- b), we can think of it as “b must either match a or must match at least one clause in else”.

I tried digging deeper in the elixir source code to get a confirmation of this, but with is a special form and implemented in dark magic, so I got no further.

Solution?

Yes, there is an easy solution. If you don’t want the statement in the with to match anything in else, don’t include the statement in the with. You can include it before, after or inside do, whateever fits the use case.

In my small example from the top, I would simply rewrite, so that the call to b/0 was moved inside the do and matched with the expected return value:

@spec c() :: :ok | :error
def c do
  with :ok <- a() do
    :ok = b()
    :ok
  else
    :error -> :error
  end
end

And since I now only have a single statement in the with, I would probably refactor to a case:

@spec c() :: :ok | :error
def c do
  case a() do
    :ok ->
      :ok = b()
      :ok

    :error ->
      :error
  end
end

Again, dialyzer was right. It does not make sense to have a statement inside a with if I’m never going to match a clause in else with it. My code is now easier to read, since it is obvious to the reader that b/0 must return :ok.

Embedded Elixir (without Nerves)

Lasse Skindstad Ebert — Sun, 20 Oct 2019 19:39:29 +0000

This story was originally posted by me on Medium on 2018-08-29, but moved here since I'm closing my Medium account.

I love Nerves. Unfortunately I am in a situation where I can not use it for my current project.

This is a somewhat detailed guide on how to run Elixir on an embedded linux without using Nerves.

My use case

I am a co-founder and a backender of Wise Home, a company located in Denmark. We do smart-home stuff for rental buildings. Our hardware pusher provides a small linux-box, which we call a gateway.

The gateway comes with some software used to communicate with external devices. This is provided as a modified buildroot with the hardware pusher’s custom patches and software. Buildroot is a tool that can build linux images, mostly used for embedded devices.

I can put my extra software on the gateway by modifiying that buildroot configuration, but it would be hard (maybe impossible?) to put their software and patches on my own buildroot config, which is why I can not use Nerves.

How to add Erlang to a buildroot configuration?

The linux image that is the output of running buildroot must contain everything needed to run the gateway. This is because most of the file system is read-only, because the gateway should not be able to get to a state where it can’t just reboot and everything is fine again.

This means that we have to put Erlang on the linux image. The gateway runs an ARM processor, so we must cross-compile Erlang to ARM. The heavy work of this is done by buildroot, so we can just configure it to install Erlang.

The buildroot configuration I’m locked to is outdated and contains an old Erlang config, but luckily I was able to just pull the latest Erlang config into my own buildroot without any trouble.

From here it is just a matter of selecting Erlang when configuring buildroot:

Buildroot will now do the heavy lifting of actually cross-compiling Erlang and add it to my linux image.

If you have never used buildroot before and want to use it, I recommend spending a little time on their guide.

So how is Elixir installed, then?

I thought I needed to also do some cross-compiling for Elixir, but I had two eureka-moments when investigating this.

First of, Elixir does not have any native executables. An Elixir installation is basically a bunch of bash-scripts and some beam-files. The bash-scripts starts Erlang and the beam-files are Elixir compiled into BEAM-code.

Secondly, Elixir (the beam files compiled from Elixir) is already included when making a release with Distillery. It is actually “just” a library sitting there with all the other included libraries:

This means that if we can deploy a Distillery release of our app onto the linux image, we’re golden. So that’s up next.

Deploying a Distillery release to the linux image

The only non-default configuration I made to my Distillery build was to not include ERTS:

release :balrog do
  set version: current_version(:balrog)
  set include_erts: false
  set include_src: false
end

ERTS is the Erlang runtime (Erlang Run-Time System) and that is what we needed to cross compile. When not including ERTS in the release, we must make sure that the Erlang version used to build the release is exactly the same as on the target linux image.

Normally when I “deploy” stuff it means that I build a release which is uploaded to a server. With embedded devices, we need to include it in the linux image before it is booting up for the first time.

With buildroot, we can simply add files to the file system of the linux box. This is called “rootfs overlay”. For convenience, I made a small script that builds the release and unpacks the resulting tar onto my rootfs overlay.

After the next buildroot build, it will be contained in my linux image.

Running the release on a read-only file system

By now, I thought I was done. But no. The Distillery release expects by default the file system to be read-write, but on an embedded linux, most of the file system is read-only.

The first attempt to run the release resulted in an error message saying that the log file folder could not be made.

It turns out that a lot of environment variables can affect how a release is started. I ended up using these four in a winning combo:

PIDFILE: On the gateway there is a monitoring system called monit. It expects each monitored application to write its own process-id to a file. With PIDFILE we tell Distillery where that file should be located.

LC_ALL=en_US.UTF-8: Not really Erlang or Elixir specific, but Elixir complained if Erlang was not run with UTF-8. I added the en_US.UTF-8 locale via buildroot and select to use it with this env var.

RELEASE_MUTABLE_DIR: This is the env var that saved my day. I can tell distillery where all mutable files can be located. On my gateway the /data dir was read-write, so I chose a subdir of that.

HOME: Again, not specific to Erlang, but Erlang tries to save a cookie file in the home dir for some reason. Since the home dir in my case was read-only, I set it to something else when running the release.

All in all, this is the command I ended up with that worked nicely:

HOME=/data/balrog/ \
RELEASE_MUTABLE_DIR=/data/balrog/ \
LC_ALL=en_US.UTF-8 \
PIDFILE=/tmp/balrog.pid \
/balrog/bin/balrog start

Yes, my app is called Balrog. Don’t ask. I might do a post later on naming things, which is of course the hardest thing in computer science.

Isolating code in contexts

Lasse Skindstad Ebert — Sun, 20 Oct 2019 19:17:15 +0000

This story was originally posted by me on Medium on 2017-12-15, but moved here since I'm closing my Medium account.

This blog post is about a design pattern that I know as “The Context Pattern”.

I am a software developer and my key focus is writing maintainable software. This is why I love software patterns. Patterns allow me to reason about my software on a more abstract level and they force me to write maintainable software from day one of a new project.

I use Elixir to illustrate my points, but I think the pattern can be adapted to most programming languages.

History of The Context Pattern

I like to think of The Context Pattern as a specialized form of The Facade Pattern.

The Facade pattern evolved in object oriented languages and was probably first formalized by the Gang Of Four.

The Context Pattern differs from The Facade Pattern in that it always isolates internal (not external) code, hiding away its complexity from the caller. It also isolates source code files in the file system from the rest of the code base.

The Phoenix framework called the pattern name with the release of Phoenix 1.3. Although Phoenix contexts are kind of Phoenix specific, I am using the pattern for non-Phoenix code too.

So what is it, exactly?

Disclaimer: This is my version of The Context Pattern. I’m not sure if the pattern has been formalized for general usage by anyone.

The pattern is simple and yet extremely powerful. You only have to do these steps to follow the pattern:

Find code in your code base that belong together, i.e. has many internal dependencies.
Move all code files into a specific directory in your project directory. Rename namespaces accordingly.
Define a single entry point of all the functionality the code exposes.

The new group of code is then conceptually referred to as “context”.

This gives the immediate benefit of simplicity for the outside world. All the code not in the context only have to worry about the public API of the context, which is all defined in a single entry point.

Changing stuff later inside the context is now also simple. Everything can be changed as long as the public API remains unchanged.

While writing a context or refactoring existing code into a context, the concept of a context makes the developer worry about dependencies and interfaces. Only the needed amount is exposed to outside code.

Example time

Finally some code.

A while ago, I had to implement geolocation lookup functionality in an existing project. For a given city, I needed to know the lattitude and longitude coordinates.

I needed the following parts to complete the feature:

A client that can talk to the Google Maps API
A database model for long-term caching the API lookup
A cache with logic to operate on the database model

I could probably write it all in one Elixir module, but that would be a big mess. It would not be testable and maintainable.

Furthermore, I needed different implementations of the client HTTP layer, since I wanted to swap it out with a mock when testing.

I ended up with the following directory structure:

lib/my_project/geolocation
├── cache.ex
├── city_location.ex
├── geolocation.ex
└── google_maps
    ├── client.ex
    ├── http.ex
    └── http_mock.ex

The only thing that the outside code should call is the MyProject.Geolocation module which is in the file lib/my_project/geolocation/geolocation.ex.

All the other modules are namesspaced under Geolocation, e.g. MyProject.Geolocation.Cache.

The Geolocation module has a single function with this signature:

@type address :: {zip :: String.t, city :: String.t, country :: String.t}
@type geolocation :: {latitude :: number, longitude :: number}

@doc """
Finds the geolocation for the given address
"""
@spec lookup(address) :: {:ok, geolocation} | {:error, any}
def lookup(address) do
  # Logic that calls the inner modules
end

What about OTP and GenServers?

I’m glad you asked :) The Cache in the example is actually a GenServer to synchronize cache lookups and avoid race situations.

So how does the main application know to start the Cache when it is not allowed to peak inside the context?

Elixir 1.5 to the rescue!

Elixir 1.5 introduced streamlined child specs, which allows any module to define a child spec, and that module can be added by itself to a supervisor.

That was a confusing explanation, so here is an example:

Instead of adding worker(MyProject.Geolocation.Cache, []) as a child to my main supervisor, with Elixir 1.5 I just add MyProject.Geolocation. I don’t even need to specify if that will start a worker or a supervisor:

children = [
  # Many children here
  # ...
  MyProject.Geolocation
]

opts = [strategy: :one_for_one, name: MyProject.Supervisor]
Supervisor.start_link(children, opts)

Then in the Geolocation main entry module, I specify a child spec, which is a recipe to start a child under a supervisor:

alias MyProject.Geolocation.Cache

def child_spec([]) do
  %{
    id: __MODULE__,
    start: {Cache, :start_link, []}
  }
end

Nice! Geolocation now controls which parts of the geolocation context is started under the main supervisor.

If I later need to change it into a more complex scenario with a supervisor with multiple workers, I still only have to make changes to the context.

Child specs can also define restart strategy and more. For reference see the Supervisor docs.

What did I gain?

I implemented the geolocation functionality as a context, exposing only a single function (oh yes, and a child spec) to the outside world.

Changing the inner works of the context is simple.

Extracting the context as a standalone application or library is simple.

Calling the functionality of the context is simple.

Completely replacing the context with another context is simple.

Changing the database structure of the db model is simple.

I like simple.

The complete feature was implemented in 329 lines split into 6 files. Only 10 percent, 32 lines, of that is revealed as public interface. Here is the complete main Geolocation module:

defmodule Legolas.Geolocation do
  @moduledoc """
  Public interface for all geolocation related functionality
  """

  alias Legolas.Geolocation.Cache
  alias Legolas.Geolocation.CityLocation
  alias Legolas.Geolocation.GoogleMaps.Client

  @type address :: {zip :: String.t, city :: String.t, country :: String.t}
  @type geolocation :: {latitude :: number, longitude :: number}

  def child_spec([]) do
    %{
      id: __MODULE__,
      start: {Cache, :start_link, []}
    }
  end

  @doc """
  Finds the geolocation for the given city address
  """
  @spec lookup(address) :: {:ok, geolocation} | {:error, any}
  def lookup(address) do
    case Cache.fetch(address, fn -> Client.lookup(address) end) do
      {:ok, %CityLocation{} = location} ->
        {:ok, {location.latitude, location.longitude}}
      {:error, reason} ->
        {:error, reason}
    end
  end
end

And yes, I name the different repos in my project after LOTR characters.

I hope you enjoyed this blog post ;)

Elixir PubSub Hub With Pattern Matching

Lasse Skindstad Ebert — Sun, 20 Oct 2019 18:58:50 +0000

This story was originally posted by me on Medium on 2017-04-15, but moved here since I'm closing my Medium account.

In a previous post, I wrote about some ideas to utilize Elixir pattern matching when subscribing to messages in a pub-sub hub.

The idea evolved to production code, and has now been extracted as an open source Elixir library called Hub.

This post is about the making of the library.

If you just want to see some code and example usage, look at the github page: https://github.com/wise-home/hub

My use case: IoT messages

As many ideas, this idea started when a need arose. I’m building an application that has persistent TCP connections to a lot of small linux boxes, which in turn each has a wireless connections to a lot of battery powered IoT devices.

To get the terms right, the small linux boxes are called gateways and the IoT devices are referred to as just devices.

I have a two way communication with each device via the gateways. Most messages are measurements send on a regular basis from the devices, but the messages can also be administrative messages just to keep the network running or to configure devices.

Flows

Some messages come in flows. That is, the messages form a conversation and only make sense if the previous messages in the flow are known. As an example for this blog post, I’ll look at how I ask a gateway which authorized devices it knows about:

I’ll send a get entry request message to the gateway, asking for all known authorized devices starting from index zero.

The gateway responds with a get entry response, giving me all serial numbers of the devices that can fit in one message with some maximim message size. Let’s say that 10 device serials can fit in one message.

I’ll re-query with another get entry request, this time asking for devices starting at index 10.

This continues until a get an empty get entry response, which means that there are no more authorized devices. I can then perform some logic with the known authorized devices in my end of the application.

This flow is easily recognized as a client-server pattern with a request and a response. However, the raw TCP connection I have does not know anything about server and client. In fact, the roles of the client and server might switch in the next flow.

Until now, I have solved this in a not-so-nice way by almost ignoring that flows exists, and only care about single messages. In the above case, if I need all authorized devices, I’ll send a get entry request. If I then later receive a get entry response, I assume that it was a response to a previous request I send and I’ll “continue” the flow and send another get entry request.

There are multuple problems with this approach:

I cannot register that a timeout happened. If I send a request and do not receive a response within some time frame, I can not register this as being an error. I just fire-and-forget the request.

I cannot handle complex flows where similar messages might mean something different depending on where we currently are in the flow or which flow it is received in.

The network sends a lot of messages by itself, and I can not easily differentiate which messages was a response to something I send, and which messages I can ignore, because the network itself handles them.

The solution

A TCP connection to a gateway is handled by a GenServer. Whenever a messages is received, it is until now handled within that same process.

Now, I have changed this to use a pub-sub hub. Instead of handling the message in the same process, it publishes the message to a pub-sub hub. If no other processes subscribed to this exact message, it is processed by a default message handler, which mostly just logs the message.

Whenever I want to start a flow of messages, I start a process that handles just that flow. The process is started as a Task and is monitored by a TaskSupervisor.

The new process subscribes to a specific kind of message from a specific gateway and has a timeout. The process only exists in the lifespan of the flow.

Hub, first version

To make this possible, I needed to be able to subscribe to a particular message, perhaps with some particular value inside the message. The many different message types and combinations of values didn’t make this possible unless I want to publish each message to each process, which I think could quickly become a performance concern.

Instead I would like to subscribe with an Elixir pattern, just like one can use them in guard clauses and function definitions.

In my previous post, I described how I can save a pattern and use it later. I used this to build a first version if Hub, which was powered by a GenServer with the internal state being all subscriptions.

In my flows, I can often limit a subscription to messages received via a specific gateway, so I use the gateway id as channel and can then subscribe with a pattern inside that channel.

This worked but had some missing parts. I especially missed monitoring of processes, so that dead processes was auto-unsubscribed.

Hub, with Phoenix Presence

Monitoring of processes is what Phoenix Presence is good at. I could easily change my GenServer to a simple module that used Phoenix Presence as the backend.
My application was already using Phoenix and Presence, so I actually had one less running process by this change.

Hub, the standalone version

I wanted to extract Hub as a standalone hex package, but I did not want it to only being able to run inside a Phoenix application.

I started to look at how Phoenix Presence is implemented and found out it is powered by Phoenix PubSub, which is a standalone package. Yay.

I extracted Hub to its own repository and changed it to use Phoenix PubSub instead of Phoenix Presence. This required me to run two processes. A Tracker and a PubSub process. The documentation in Phoenix PubSub is good, so I was up and running in no time.

Hub can now be used by anyone and has a simple interface.

See more on the github page: https://github.com/wise-home/hub

Subscribe to messages with pattern matching

Lasse Skindstad Ebert — Sun, 20 Oct 2019 18:49:00 +0000

This story was originally posted by me on Medium on 2017-04-13, but moved here since I'm closing my Medium account.

Today I needed to build a small and simple publish/subscribe hub. The feature I cared about was to be able to subscribe to messages with Elixir pattern matching.

Before I dive into that, let me explain the difference from a normal publish/subscribe hub.

Often with the publish/subscribe pattern, subscribers subscribe to some topic and publishers publish a term to some topic.

Example:

# Probably started by a supervisor:
{:ok, pid} = PubSub.start_link

# In our process
PubSub.subscribe(:user_created)
receive do
  {:user_created, user} -> # Do something with user
end

# And in some other process:
PubSub.publish(:user_created, user)

This is all fine. As long as we don’t need an infinite number of topics.

Subscribe with pattern matching

In my application, I receive around 100 different types of messages from a lot of IoT devices. I would like different processes to be able to subscribe to incomming messages, but each process may have very different needs in terms of which messages they need to see.

Some processes would like to see all messages originating from a specific device. Some would like all messages of a certain type. Some would like to subscribe to a message with a certain sequence number.

This is not doable with the topic-approach. What I would really like is to use Elixir pattern matching to decide which messages to subscribe to.

Something like this:

# Subscribe to all messages from a specific origin
PubSub.subscribe(%{origin: "1234"})

# Subscribe to all messages of a specific type
PubSub.subscribe(%SomeMessageType{})

# Subscribe to all messages where the signal strength is bad
PubSub.subscribe(%{signal_strength: strength} when strength < 5)

Can this be done in Elixir? YES! It can be done. Or at least in my small prototype, which I will start to use in a real application very soon to test it out.

I started to have a look at Kernel.match/2 (docs here: https://hexdocs.pm/elixir/Kernel.html#match?/2), as it seemed to solve the same problem I was aiming at. Unfortunately, it couldn’t be used directly, but I found a lot of inspiration from the (pretty simple) source code of that macro.

I began to play in iex with quote, Macro.escape and other functions and macros that does something to the AST:

iex(41)> quote do: %{signal_strength: strength} when strength < 5
{:when, [],
 [{:%{}, [], [signal_strength: {:strength, [], Elixir}]},
  {:<, [context: Elixir, import: Kernel], [{:strength, [], Elixir}, 5]}]}
iex(42)> quote do
...(42)> case %{signal_strength: 7} do
...(42)> %{signal_strength: strength} when strength < 5 -> true
...(42)> _ -> false
...(42)> end
...(42)> end
{:case, [],
 [{:%{}, [], [signal_strength: 7]},
  [do: [{:->, [],
     [[{:when, [],
        [{:%{}, [], [signal_strength: {:strength, [], Elixir}]},
         {:<, [context: Elixir, import: Kernel],
          [{:strength, [], Elixir}, 5]}]}], true]},
    {:->, [], [[{:_, [], Elixir}], false]}]]]}

This is almost all I need.

The solution

I came up with this lovely function, which takes a quoted pattern and a value and returns true if the value matches the pattern:

def pattern_match?(quoted_pattern, term) do
  ast = {:case, [], [
    Macro.escape(term), [
      do: [
        {:->, [], [[quoted_pattern], true]},
        {:->, [], [[{:_, [], Elixir}], false]}
      ]
    ]
  ]}

  {result, _} = Code.eval_quoted(ast)
  result
end

And it worked! This nice little function lets me use a saved pattern stored as AST to match a value.

I assembled a simple Hub built on GenServer:

defmodule Hub do
  @moduledoc """
  Publish/subscribe server.
  Subscription is done with a pattern.
  Example:
    {:ok, _pid} = Hub.start_link(name: :hub)
    Hub.subscribe(:hub, %{count: count} when count > 42)
    Hub.publish(:hub, %{count: 45, message: "You rock!"})
  """

  @doc """
  Starts the Hub GenServer process
  """
  @spec start_link(Genserver.options) :: GenServer.on_start
  def start_link(options \\ []) do
    GenServer.start_link(__MODULE__, :ok, options)
  end

  @doc """
  Convenience macro for subscribing without the need to unquote the pattern.
  example:
    Hub.subscribe(:hub, %{count: count} when count > 42)
  """
  defmacro subscribe(hub, pattern, pid \\ nil) do
    quote do
      pid = unquote(pid) || self()
      Hub.subscribe_quoted(unquote(hub), unquote(Macro.escape(pattern)), pid)
    end
  end

  @doc """
  Subscribes the given pid to the quoted pattern
  example:
    Hub.subscribe(:hub, quote do: %{count: count} when count > 42)
  """
  @spec subscribe_quoted(pid, any, pid) :: :ok
  def subscribe_quoted(hub, quoted_pattern, pid \\ self()) do
    GenServer.call(hub, {:subscribe, {pid, quoted_pattern}})
  end

  @doc """
  Publishes the term to all subscribers that matches it
  """
  @spec publish(pid, any) :: :ok
  def publish(hub, term) do
    GenServer.cast(hub, {:publish, term})
  end

  def init(:ok) do
    {:ok, []}
  end

  def handle_call({:subscribe, subscriber}, _from, subscribers) do
    {:reply, :ok, [subscriber | subscribers]}
  end

  def handle_cast({:publish, term}, subscribers) do
    subscribers
    |> Enum.each(fn {pid, pattern} ->
      if pattern_match?(pattern, term) do
        send(pid, term)
      end
    end)
    {:noreply, subscribers}
  end

  defp pattern_match?(pattern, term) do
    ast = {:case, [], [
      Macro.escape(term), [
        do: [
          {:->, [], [[pattern], true]},
          {:->, [], [[{:_, [], Elixir}], false]}
        ]
      ]
    ]}

    {result, _} = Code.eval_quoted(ast)
    result
  end
end

Notice that it is possible to subscribe either using a function or a convenience macro.

This lets me do this:

{:ok, _pid} = Hub.start_link(name: :hub)

# In one process
Hub.subscribe(:hub, %{signal_strength: strength} when strength < 5)

# In another process
Hub.publish(:hub, %MyMessageType{signal_strength: 4})

And that’s it!

If you know of a better or more obvious way to acheive the same thing, please let me know :)

Running observer inside a Vagrant box

Lasse Skindstad Ebert — Sun, 20 Oct 2019 18:40:09 +0000

This story was originally posted by me on Medium on 2016-11-28, but moved here since I'm closing my Medium account.

The :observer

Elixir is shipped with a ton of nice features. One of those is the ability to start the erlang :observer on a running appliciation and see the process tree, inspect indiviual processes and more.

Just to recap, the observer is started from an iex session that is preloaded with the mix application we want to inspect:

$ iex -S mix
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :observer.start

Note that it is not necessary to start iex with -S mix for observer to work.

The observer is a GUI application that shows a lot of useful information about the running application. I use it a lot when developing if a Supervisor or GenServer is not behaving as I would expect.

If you a new to observer, find a post that describes it in more detail, It is time well spent.

Vagrant

I have a habit of running almost every software project inside a Vagrant virtual machine. Doing so allows me to have different versions of the database or other dependencies installed for each project.

I also like that I don’t pollute my host machine with all kinds of software needed for a specific project.

But running my development environment in Vagrant makes it harder to run the observer, since a Vagrant box is headless. I can’t run a GUI inside it.

Connecting to a node inside Vagrant

Luckily Erlang has me covered. I can start a node outside my Vagrant (on my host machine) and connect to a running application inside Vagrant. To do so the two nodes must share a secret, known as a cookie in Erlang terms.

We also need to name each node, so that each of them can be found by name and host.

Start the application inside Vagrant with a name and a cookie:

iex --name my_app@192.168.33.10 --cookie 123 -S mix phoenix.server

My project is a Phoenix project, which is why I start it with mix phoenix.server. The IP address is the one of my Vagrant box as specified in my Vagrantfile:

config.vm.network "private_network", ip: "192.168.33.10"

Then I start a node on my host machine, and from that I can connect to the Vagrant node and start observer:

iex --name debug@127.0.0.1 --cookie 123
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(debug@127.0.0.1)1> Node.connect(:"my_app@192.168.33.10")
true
iex(debug@127.0.0.1)2> :obsever.start

In the observer I can choose another node than the local one from the menu and I now see the application list and supervision trees from the remote node as if I had started observer from that node.

Neat!

But… I soon ran into trouble with a firewall.

Connecting to a node behind a firewall

I later installed a firewall on my Vagrant box. Mostly to test the firewall setup on the production system. After this I could no longer connect from a remote iex. This was kind of expected since that is pretty much the firewall’s job: Only allow the specified ports to connect.

So how do I configure a firewall to allow incoming node connections?

It turns out that connecting to a remote node requires two ports to be open. The port for discovering the node and the port at which the actual connection is opened at.

The discovery port defaults to 4369 but the other port is chosen at random in some huge range. We can limit this port range by specifying some erlang properties when starting iex.

This limits the range to the single port 9000:

iex --name my_app@192.168.33.10 \
    --cookie 123 \
    --erl '-kernel inet_dist_listen_min 9000' \
    --erl '-kernel inet_dist_listen_max 9000' \
    -S mix phoenix.server

Now I can just allow port 4369 and 9000 in my firewall, and I’m good to connect to my firewalled application inside a Vagrant box!

Credits

I learned the erlang flags for limiting the port range by reading this good article about clustering by James Whiteman.

Test driving a Phoenix endpoint, part I

Lasse Skindstad Ebert — Sun, 20 Oct 2019 18:27:13 +0000

This story was originally posted by me on Medium on 2016-10-18, but moved here since I'm closing my Medium account.

I have worked part time professionally with Elixir for around a half year. Starting next month I will work with Elixir full time. And I can’t wait!

The project I’m working on is split into four applications, one of which is a REST api using Phoenix as the main framework.

I am very thrilled about Elixir, ExUnit, Ecto and a lot of other stuff in the Elixir ecosystem. But I have been a little careful not liking Phoenix too much.

Maybe because I burned myself when I jumped into Rails around 5 years ago. Now I think Rails and Active* is everything Ruby should not be. I hope Phoenix will develop differently and it certainly looks good right now.

But some of the example code attached to the Phoenix documentation is a bit alarming. I know. It is only example code and people shouldn’t copy-paste it into their own production code. But it could be exemplary examples.

As an example, take a look at this update action from the Phoenix documentation:

def update(conn, %{"id" => id, "user" => user_params}) do
  user = Repo.get!(User, id)
  changeset = User.changeset(user, user_params)

  case Repo.update(changeset) do
    {:ok, user} ->
      conn
      |> put_flash(:info, "User updated successfully.")
      |> redirect(to: user_path(conn, :show, user))
    {:error, changeset} ->
      render(conn, "edit.html", user: user, changeset: changeset)
  end
end

In my opinion, controllers should be doing web stuff and delegate all non-web stuff to other places in the codebase. Because the non-web stuff tend to grow in complexity and keeping it inside the controller is a mix of concerns.

This example also shows the default behaviour of defining a changeset in the schema module. In the User module in this case.

Changesets should be defined in a specific context. Validations, accepted attributes and so on might change depending on which endpoint, cron-job or other code I’m in.

Why? Because reusability, testability and maintainability.

Test driving

This is the first part of a mini-series of blog posts. I estimate it will consist of 2–4 blog posts. This post is about testing the controller web logic and separating it from the non-web logic.

Later posts will address other issues in testing such as writing a mock for an external service.

Edit: Second post about mocks can be found here.

I will take you through the journey of test-driving a single Phoenix endpoint. The endpoint lives in POST /users and is responsible for inviting a new user to our system.

When we’re done, we have build an endpoint that will create an inactive user and send an email to the user with instructions on how to activate.

When test-driving the code, it usually end up being reusable and maintainable, simply because we write the consumer of the code first.

Let’s get started!

We will create a Phoenix application from scratch. We start by using the phoenix.new mix task to create our application called MyApp. Since we are building a REST api, we opt-out of using HTML and Brunch stuff:

$ mix phoenix.new --no-html --no-brunch my_app

First test

We start by building the create-user-part. Later we will build the send-activation-email-part.

We can go one of two ways. Inside-out or outside-in. Start with the REST endpoint or start with the inner most layer of the application, which might be a database table or something else. Since we don’t know which layers come after the controller yet, we choose outside in: Starting in the controller and work our way in to the inner layers.

The first thing we do before anything else, is writing a test. We will use the first test as a kind of integration test, testing all the layers of the application and the integration between them. Later we will add unit tests to test edge case behaviours of the controller and other pieces of code.

Here is our first test:

defmodule MyApp.UserController.InviteTest do
  use MyApp.ConnCase

  test "inviting a user responds with the new user" do
    conn =
      build_conn
      |> post("/users", email: "alice@example.com")

    body = conn |> response(201) |> Poison.decode!

    assert body["id"] > 0
    assert body["email"] == "alice@example.com"
  end
end

We will run this test. But not yet.

First we should make some assumptions about what is going to fail. If we are wrong in our assumption, we have either written our test wrong or we have learned something new.

Looking at the test, we first post some payload to /users which should be ok. Then we assert that the response has status 201, which will not be true, because the endpoint does not exist yet. Let’s give it a try:

$ mix test
...

  1) test inviting a user responds with the new user (MyApp.UserController.InviteTest)
     test/controllers/user_controller/invite_test.exs:4
     ** (RuntimeError) expected response with status 201, got: 404, with body:
     {"errors":{"detail":"Page not found"}}
     stacktrace:
       (phoenix) lib/phoenix/test/conn_test.ex:362: Phoenix.ConnTest.response/2
       test/controllers/user_controller/invite_test.exs:9: (test)



Finished in 0.08 seconds
4 tests, 1 failure

Randomized with seed 448141

Surely enough. Our tests reveals that we get a 404 when posting our payload. The test result almost tells us what to do next, which is one of the things I like about test driven development. The next thing we should do is to make sure our application understands a POST /users which means we should add something to the router.

We edit the web/router.ex file:

Adding the route

defmodule MyApp.Router do
  use MyApp.Web, :router

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", MyApp do
    pipe_through :api

    resources "/users", UserController, only: [:create]
  end
end

We added the users resource and run the tests again:

$ mix test
Compiling 3 files (.ex)
...

  1) test inviting a user responds with the new user (MyApp.UserController.InviteTest)
     test/controllers/user_controller/invite_test.exs:4
     ** (UndefinedFunctionError) function MyApp.UserController.init/1 is undefined (module MyApp.UserController is not available)
     stacktrace:
       MyApp.UserController.init(:create)
       (my_app) web/router.ex:1: anonymous fn/1 in MyApp.Router.match_route/4
       (my_app) lib/phoenix/router.ex:261: MyApp.Router.dispatch/2
       (my_app) web/router.ex:1: MyApp.Router.do_call/2
       (my_app) lib/my_app/endpoint.ex:1: MyApp.Endpoint.phoenix_pipeline/1
       (my_app) lib/my_app/endpoint.ex:1: MyApp.Endpoint.call/2
       (phoenix) lib/phoenix/test/conn_test.ex:224: Phoenix.ConnTest.dispatch/5
       test/controllers/user_controller/invite_test.exs:7: (test)



Finished in 0.06 seconds
4 tests, 1 failure

Randomized with seed 999258

The test tells us that we need the UserController, which was expected, since we reference it from the router, but did not yet create it.

Adding the controller

defmodule MyApp.UserController do
  use MyApp.Web, :controller

  def create(conn, _params) do
    conn
    |> resp(201, "{}")
  end
end

We just return status 201 and an empty JSON object.

Again, we make assumptions about where the test will fail and run the test again. First, let’s review the test:

body = conn |> response(201) |> Poison.decode!

    assert body["id"] > 0
    assert body["email"] == "alice@example.com"

Take a moment to figure out where it will blow up next before we run the test. Done? Now for the test output:

$ mix test
Compiling 1 file (.ex)
Generated my_app app
...

  1) test inviting a user responds with the new user (MyApp.UserController.InviteTest)
     test/controllers/user_controller/invite_test.exs:4
     Assertion with == failed
     code: body["email"] == "alice@example.com"
     lhs:  nil
     rhs:  "alice@example.com"
     stacktrace:
       test/controllers/user_controller/invite_test.exs:12: (test)



Finished in 0.06 seconds
4 tests, 1 failure

Randomized with seed 773707

Ok, this was suprising! At least to me. My guess was that it would fail on body["id"] > 0 since we don’t return any id. But it fails on the email assertion on the next line. What happened?

In Elixir everything is comparable to everything else. Not every comparison makes sense, but be sure to remember that we can compare any two data types.

Looking in the Elixir documentation, we see this order for comparing non-same types:

number < atom < reference < function < port < pid < tuple < map < list < bitstring

Since nil is an atom and 0 is a number, our assertion passes. Which means our test is wrong. Let’s fix it:

 body = conn |> response(201) |> Poison.decode!

    assert body["id"] |> is_integer
    assert body["id"] > 0
    assert body["email"] == "alice@example.com"

Running the test again now shows that our expectations are correct (the expectation that the assertion is wrong) and we should return an id and an email to satisfy the test.

But what should we return? We have no id and no user row in the database. We don’t even have a User schema. The next step is to actually create a user. The logic of user-creation is something we should delegate to another module, since we want to be able to reuse and unit test it.

We update the controller:

defmodule MyApp.UserController do
  use MyApp.Web, :controller

  def create(conn, params) do
    {:ok, user} = MyApp.Users.Invite.call(params)

    conn
    |> resp(201, "{}")
  end
end

This, of course, makes the test fail because MyApp.Users.Invite does not exist yet. We will leave the controller test for now, while we create the invitation logic. After that we will return here and make it green.

Creating the invitation logic

Our efforts so far has lead us to the clarification that we need a Users.Invite module which will contain the logic related to inviting a user.

We start on a blank page since we’re building a new thing now. So just as before we will start by writing a failing test and take it from there. When this test is green, we will return to the controller test.

defmodule MyApp.Users.InviteTest do
  use MyApp.ModelCase

  alias MyApp.Users.Invite
  alias MyApp.User

  test "it creates a user" do
    {:ok, user} = Invite.call(%{email: "alice@example.com"})

    assert user.id > 0
    assert user.email == "alice@example.com"

    assert (from u in User, select: count(u.id)) |> Repo.one == 1
  end
end

Running the test tells us that we should create MyApp.Users.Invite:

$ mix test test/contexts


  1) test it creates a user (MyApp.Users.InviteTest)
     test/contexts/users/invite_test.exs:6
     ** (UndefinedFunctionError) function MyApp.Users.Invite.call/1 is undefined (module MyApp.Users.Invite is not available)
     stacktrace:
       MyApp.Users.Invite.call(%{email: "alice@example.com"})
       test/contexts/users/invite_test.exs:7: (test)



Finished in 0.04 seconds
1 test, 1 failure

Randomized with seed 973477

So we will create it:

defmodule MyApp.Users.Invite do

  alias MyApp.User
  alias MyApp.Repo

  @create_params [:email]

  def call(%{email: email}) do
    email
    |> build_changeset
    |> create_user
  end

  defp build_changeset(email) do
    params = %{email: email}

    %User{}
    |> Ecto.Changeset.cast(params, @create_params)
    |> Ecto.Changeset.unique_constraint(:email)
  end

  defp create_user(changeset) do
    changeset
    |> Repo.insert
  end
end

This code is pretty simple for now. It will 1) build a changeset using only the email and 2) save the changeset.

Where does this fail when we run the test again? I have an assumption. Let’s see:

$ mix test test/contexts
Compiling 1 file (.ex)

== Compilation error on file lib/my_app/contexts/users/invite.ex ==
** (CompileError) lib/my_app/contexts/users/invite.ex:17: MyApp.User.__struct__/1 is undefined, cannot expand struct MyApp.User
    (stdlib) lists.erl:1354: :lists.mapfoldl/3
    (stdlib) lists.erl:1354: :lists.mapfoldl/3

Right. It fails because we don’t have a user schema yet. So we create the user schema and migration.

Creating the schema and database table

First we simply create the User schema:

defmodule MyApp.User do
  use MyApp.Web, :model

  schema "users" do
    field :email, :string

    timestamps
  end
end

Notice the lack of a changeset function. When using the Phoenix generators, a changeset function is created for us in the schema modules. We don’t need nor want this.

We defined the changeset in the Users.Invite module. This module contains all the logic we need to create and invite a user. Hence the changeset related to creating a user this way is also placed there. In other places we might need another changeset definition for creating a user.

Defining the changeset where we use it allows us to maintain the changeset definition with confidence and without fear of breaking a random part of our application.

We run the test again:

$ mix test test/contexts     


  1) test it creates a user (MyApp.Users.InviteTest)
     test/contexts/users/invite_test.exs:7
     ** (Postgrex.Error) ERROR (undefined_table): relation "users" does not exist
     stacktrace:
       (ecto) lib/ecto/adapters/sql.ex:463: Ecto.Adapters.SQL.struct/6
       (ecto) lib/ecto/repo/schema.ex:397: Ecto.Repo.Schema.apply/4
       (ecto) lib/ecto/repo/schema.ex:193: anonymous fn/11 in Ecto.Repo.Schema.do_insert/4
       test/contexts/users/invite_test.exs:8: (test)



Finished in 0.04 seconds
1 test, 1 failure

Randomized with seed 711320

Just as we thought! We need a database table to fill the user into. We create a migration for this.

$ mix ecto.gen.migration add_users_table

defmodule MyApp.Repo.Migrations.AddUsersTable do
  use Ecto.Migration

  def change do
    create table(:users) do
      add :email, :string

      timestamps
    end

    create index(:users, [:email])
  end
end

$ mix ecto.migrate

We migrate only for the dev environment, since Phoenix by default is set up to migrate the test environment before every test run with these lines in the mix.exs file:

defp aliases do
  [...more aliases here],
  "test": ["ecto.create --quiet", "ecto.migrate", "test"]]
end

It’s time to run the test for Users.Invite again.

$ mix test test/contexts                           
.

Finished in 0.05 seconds
1 test, 0 failures

Randomized with seed 163288

Yay! First green test. We now have a reusable component in our application and it is working. This is a big deal.

We can now revisit our failing controller test and make that green.

Going back the the controller test

Since we started with the test for Users.Invite we only ran that test every time we tested. We now go back to run all tests again, since we only expect one test to fail.

Let’s run all tests and see how the controller test fails:

$ mix test              
...

  1) test inviting a user responds with the new user (MyApp.UserController.InviteTest)
     test/controllers/user_controller/invite_test.exs:4
     ** (FunctionClauseError) no function clause matching in MyApp.Users.Invite.call/1
     stacktrace:
       (my_app) lib/my_app/contexts/users/invite.ex:8: MyApp.Users.Invite.call(%{"email" => "alice@example.com"})
       (my_app) web/controllers/user_controller.ex:5: MyApp.UserController.create/2
       (my_app) web/controllers/user_controller.ex:1: MyApp.UserController.action/2
       (my_app) web/controllers/user_controller.ex:1: MyApp.UserController.phoenix_controller_pipeline/2
       (my_app) lib/my_app/endpoint.ex:1: MyApp.Endpoint.instrument/4
       (my_app) lib/phoenix/router.ex:261: MyApp.Router.dispatch/2
       (my_app) web/router.ex:1: MyApp.Router.do_call/2
       (my_app) lib/my_app/endpoint.ex:1: MyApp.Endpoint.phoenix_pipeline/1
       (my_app) lib/my_app/endpoint.ex:1: MyApp.Endpoint.call/2
       (phoenix) lib/phoenix/test/conn_test.ex:224: Phoenix.ConnTest.dispatch/5
       test/controllers/user_controller/invite_test.exs:7: (test)

.

Finished in 0.1 seconds
5 tests, 1 failure

Randomized with seed 633478

Ok? This was a surprise again. A function clause not matching in Users.Invite. After reviewing the code we see that from the controller we pass in %{"email" => "someemail"} but Users.Invite expects the email key to be an atom.

The actual mistake was that we passed the params directly to the underlaying logic without first extracting the things we need. We will fix this now.

We change

{:ok, user} = MyApp.Users.Invite.call(params)

to:

attrs = %{email: params["email"]}
{:ok, user} = MyApp.Users.Invite.call(attrs)

This will fix the problems. But note that normally we should parse the input params with an Ecto.Schema or similar.

When we only have one parameter this will be fine. If we’re dealing with just a few more parameters this method of manually parsing them will lead to bugs at some point. Parsing the params should be a job of it’s own, but for the sake of this blog post let’s keep it simple as above.

We run the test again:

$ mix test
Compiling 1 file (.ex)
warning: variable user is unused
  web/controllers/user_controller.ex:6

....

  1) test inviting a user responds with the new user (MyApp.UserController.InviteTest)
     test/controllers/user_controller/invite_test.exs:4
     Expected truthy, got false
     code: body["id"] |> is_integer
     stacktrace:
       test/controllers/user_controller/invite_test.exs:11: (test)



Finished in 0.08 seconds
5 tests, 1 failure

Randomized with seed 847631

Yes! It now yells about the missing id again, which was what we hoped would fail. This is because we still return an empty JSON object from the controller even though we create a user. We even get a warning that our user is unused.

Serializing an internal data type to a json object is also a job of it’s own, and many great libraries exists for this single purpose. Again, for the sake of the essence in this blog post, we simply just create a map and serialize it with Poison.

This is the updated controller:

defmodule MyApp.UserController do
  use MyApp.Web, :controller

  def create(conn, params) do
    attrs = %{email: params["email"]}
    {:ok, user} = MyApp.Users.Invite.call(attrs)

    {:ok, body} = %{
      id: user.id,
      email: user.email
    }
    |> Poison.encode

    conn
    |> resp(201, body)
  end
end

We run the tests again:

$ mix test
Compiling 1 file (.ex)
.....

Finished in 0.08 seconds
5 tests, 0 failures

Randomized with seed 174980

Finally! For the first time since we started the blog post, all tests are green. We have now made a REST endpoint that can create a user. Good job!

Edge cases

We have, however, only tested the “normal case” of the application. What should happen if we send an obviously incorrect email address like "Alice and Bob"? What if we try to create a user with an email that exists on another user? What if the email is blank? Not submitted at all?

A good REST API knows how to respond to all kinds of wierd input. A meaningful HTTP status code and a detailed error message can be a life-saver for an api user getting a response back from an invalid request.

We will of course test drive edge cases just like we did with the “normal case”.

We decide that we only want to handle one edge case for the sake of the blog post. In a real production site we should handle all the weird edge case scenarios we can think of.

An invalid email should give 400: {"message": "email is invalid”}.

Check for missing email

Up until now we have gone through the testing very thoroughly. This process is usually pretty quick in practice, but very verbose in writing.

We will now go up one level and dicuss which tests should be written and where certain parts of the logic belong. For this section, I will not post the entire test output for each test run.

When checking for invalid email, the actual check should be in the Users.Invite logic, since we also want to check for invalid emails if we reuse the module somewhere else. But the HTTP status and error message belongs in the controller, since that is both web specific and specific to that controller.

We start by adding the test to the controller:

 test "invalid email gives a 400 response" do
    conn =
      build_conn
      |> post("/users", email: "Alice and Bob")

    body = conn |> response(400) |> Poison.decode!

    assert body["message"] == "email is invalid"
  end

This test fails because the code just creates the user with invalid email.

We realize that the actual check should happen in Users.Invite so we write a similar test for that.

  test "it handles invalid email" do
    {:error, :invalid_email} = Invite.call(%{email: "Alice and Bob"})
  end

Notice how much smaller this test is compared to the controller test. This is a sign that this is the right place to do edge case testing. If we want to test a lot of invalid and valid email addresses, we should add them to this test and just leave the single test in the controller.

Running the test tells us that the logic in Users.Invite accepts "Alice and Bob" as a valid email. We fix this by ensuring that the email has at least one @. This is intentionally simple, since we can’t possibly write a good validator for whatever RFC document describes what a valid email looks like.

As an example "Alice and Bob"@example.com is a valid email address. We just check for the @ because leaving it out is usually a sign that someone misread a form somewhere and entered a name instead of an email address. Our real email validation happens when we send out the activation email. If the user can get the email, the email is valid.

We fix the logic:

  def call(%{email: email}) do
    with :ok <- validate_email(email) do
      email
      |> build_changeset
      |> create_user
    end
  end

  #...

  defp validate_email(email) do
    case String.match?(email, ~r/@/) do
      true -> :ok
      false -> {:error, :invalid_email}
    end
  end

We wrap the user creation in a with :ok <- validate_email(email). This tells Elixir to go on if validate_email returns :ok, but return the result of that function call if it returns another value than :ok. This makes it easy to step out early froma series of function calls.

The new test now passes, but the controller test is still red. Let’s fix that. This is the entire controller after the fix:

defmodule MyApp.UserController do
  use MyApp.Web, :controller

  def create(conn, params) do
    attrs = %{email: params["email"]}

    case MyApp.Users.Invite.call(attrs) do
      {:ok, user} ->
        render_user(conn, user)
      {:error, :invalid_email} ->
        resp(conn, 400, %{message: "email is invalid"} |> Poison.encode!)
    end
  end

  defp render_user(conn, user) do
    {:ok, body} = %{
      id: user.id,
      email: user.email
    }
    |> Poison.encode

    conn
    |> resp(201, body)
  end
end

Notice the case statement. This is a place to add all the kinds of responses we can think of. When we later review the code, it’s easy to see what this controller can return.

Summing up

We have build a highly reusable and testable module for inviting a user. It is so reusable that it already has two consumers: The controller and the unit test.

The controller we have build is only concerned with web-specific stuff like parsing params, returning the right HTTP status code and serializing the response body to JSON. It’s easy to read and easy to maintain.

This blog post is long and it might seem like a long way to get to something very simple.

But in practice, when you sit down and write code with your headphones on, listening to the best of Hans Zimmer, writing this code might take around 5 minutes. Maybe only 3 minutes on a good day.

I don’t test because I’m afraid that the code base will break later on. I test because it yields a better code base. One that is easy to maintain and is highly reusable.

That I can run my tests again and again to ensure that nothing is broken is just a nice bi-product.

Stay tuned for Part II in which we’ll explore how to build a Mailer mock to use in tests.

Edit: Part II is now ready here.

Mocks in Elixir

Lasse Skindstad Ebert — Sun, 20 Oct 2019 18:26:48 +0000

Or “Test driving a Phoenix endpoint, part II”

This story was originally posted by me on Medium on 2016-10-32, but moved here since I'm closing my Medium account.

As the subtitle suggests, this is the second part of a blog post series about Phoenix and TDD. However, this post is not Phoenix-specific and the patterns presented here can be used in any Elixir application.

The first part was about separation of concerns and how to make clean and reusable code within Phoenix. It can be found here.

This part is about testing modules that somehow makes external calls that we would like to avoid while testing.

Mocking external calls

Since I started with Elixir, I have wondered how to mock stuff in tests. And when to mock stuff.

I come from Ruby, and I have developed a habit of injecting doubles for all dependencies of a class when unit testing the class.

Something like this:

class InviteUser
  def initialize(mailer: Mailer.new, user_repo: UserRepo.new)
    @mailer = mailer
    @user_repo = user_repo
  end

  attr_reader :mailer
  attr_reader :user_repo

  def invite(email)
    user = user_repo.create(email)
    mailer.send_activation_mail(user)
    user
  end
end

RSpec.describe InviteUser do
  describe "#invite" do
    it "sends an activation email" do
      mailer = instance_double(Mailer, create: User.new)
      user_repo = instance_double(UserRepo, send_activation_mail: true)

      invite_user = InviteUser.new(mailer: mailer, user_repo: user_repo)

      invite_user.invite("me@example.com")

      expect(mailer).to have_received(:send_activation_mail)
    end
  end
end

This is also possible in Elixir. One could just inject a module into a function, but it does not seem to be the preferred way in the Elixir community.

I needed an answer to two questions:

When do I use a mock?
How do I mock?

There seems to be a very unambiguous trend in the community to the first question: Only use mocks when you would otherwise hit a third party service. The database does not count as a third party service here, since it is clearly possible to use it in tests.

Since we’re building something that can send out emails (see Part I), we would surely like to somehow mock that out. That brings us to the second question: How do we mock?

Different mocking patterns

I have explored the possibilities of mocking in Elixir and boiled it down to four different techniques, which I will briefly go through here.

Spoiler: The last one is my preferred.

Injection with pure Elixir

As explained with the Ruby example above, we could simply inject a module to a function and have a default value for that:

defmodule InviteUser do
  def invite(email, mailer \\ Mailer) do
    user = create(email)
    mailer.send_activation_mail(user)
  end
end

This would work and does not require any hex libs. But it does not feel very Elixirish. One of the things I love about Elixir is the stuff that happens at compile time. This runtime injection takes away that joy.

I also like that everything is very clear in Elixir. You can look at a function and immediately know what it is doing. With runtime injection, anything can happen. We don’t know what kind of mailer is send to the function.

I like this method of mocking in Ruby, but I don’t like it for Elixir.

Injection with a library

I have tried a hex package called pact that offers dependency injection. It is basically the same as the injection with pure Elixir, except it uses a process to keep a store of modules and some nice convenience functions to create mocks of those modules from a test.

But it is still runtime injection, which I don’t think is a good match in Elixir. An example with pact would look something like this:

defmodule InviteUser do
  def invite(email) do
    user = create(email)
    MyApp.Pact.get("mailer").send_activation_mail(user)
  end
end

And you would then be able to mock out the content of MyApp.Pact.get("mailer") at runtime in tests.

This seems like an even more poor solution than the injection with pure Elixir, since it requires a process to be running and more confusion when trying to read what a function actually does.

Mocking with mock

mock is an Elixir library that uses the meck Erlang library for mocking. I have not tried to use it, so what I’m saying here might be a little wrong. Please correct me.

It seems that mock does runtime substitution of module functionality with some dark magic meta programming. It reminds me a lot of the mocking functionality in RSpec from Ruby.

mock allows you to e.g. tell String.reverse/1 to accept a number and return the number times two(?). Here is an example from the readme:

efmodule MyTest do
  use ExUnit.Case, async: false

  import Mock

  test "multiple mocks" do
    with_mocks([
      {HashDict,
       [],
       [get: fn(%{}, "http://example.com") -> "<html></html>" end]},
      {String,
       [],
       [reverse: fn(x) -> 2*x end,
        length: fn(_x) -> :ok end]}
    ]) do
      assert HashDict.get(%{}, "http://example.com") == "<html></html>"
      assert String.reverse(3) == 6
      assert String.length(3) == :ok
    end
  end
end

This is the kind of mocking that José Valim would describe as “mock (verb), not mock (noun)”.

If, for example, we would like to test an API client, we could mock (verb) the underlaying HTTP library to return the expected responses. In my humble opinion, this is just wrong and takes away all the functional part of Elixir.

A better way, José says, is to create a mock (noun) and at compile time choose to use either the real implementation or the mock implementation.

This leads us to the fourth and my preferred way to test external calls in Elixir.

Compile time settings

This section is basically a summary of José’s post.

We define two modules: One which holds the real implementation and one that holds the mock implementation. In the config we define that the test environment uses the mock implementation and all other environments use the real implementation.

We then define a behaviour to make sure both implementations follow the same interface.

That’s it. It’s simple, it’s compile time and it’s pure Elixir.

Why is it good that it is happening at compile time, you might ask. It’s good because the compiled production code will be essentially the same as if we didn’t use a mock in tests.

In the rest of this blog post we will create a mailer mock, TDD style.

Creating the mailer

Before we create a mock, we create a working mailer, which we test in the dev environment via iex. We write no tests for this implementation.

I followed the Phoenix documentation of how to send emails and came up with this mailer:

defmodule MyApp.Mailer do
  use Mailgun.Client, [
    domain: Application.fetch_env!(:my_app, :mailgun_domain),
    key: Application.fetch_env!(:my_app, :mailgun_key)
  ]

  @from "support@myapp.com"

  def send_test_mail(email) do
    {:ok, _} = send_email [
      to: email,
      from: @from,
      subject: "Test!",
      text: "Test from MyApp"
    ]
    :ok
  end
end

We can easily one-time test this from iex -S mix and verify that we indeed get an email in our real mailbox.

Test driving the mailer mock

We are going to keep the mail composing functionaility in MyApp.Mailer and move the Mailgun implementation to MyApp.Mailer.Mailgun. Likewise we will create the mock in MyApp.Mailer.Mock.

But first, we write a test:

defmodule MyApp.MailerTest do
  use ExUnit.Case

  test "it uses mock in tests" do
    MyApp.Mailer.Mock.clear
    MyApp.Mailer.send_test_mail("me@example.com")

    assert MyApp.Mailer.Mock.mails |> length == 1

    [mail] = MyApp.Mailer.Mock.mails

    assert mail.to == "me@example.com"
    assert mail.text == "Test from MyApp"
  end
end

This is mostly a test that the mock works as expected, but it also tests the small amount of mail composing functionality in the send_test_mail function.

We run the test

$ mix test
Compiling 16 files (.ex)

== Compilation error on file lib/my_app/mailer.ex ==
** (ArgumentError) application :my_app is not loaded, or the configuration parameter :mailgun_domain is not set
    (elixir) lib/application.ex:261: Application.fetch_env!/2
    lib/my_app/mailer.ex:3: (module)
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6

And we get a compilation warning? How? The mailer worked fine in development. We would expect the test to fail, since we don’t have a mock yet, not because it can’t compile.

This is because the current Mailer implementation uses Mailgun and fetches some settings from the config at compile time. We haven’t defined the mailgun settings for the test environment, which is why we get a compilation error.

To fix this, we need to refactor the Mailer. Instead of having a Mailgun implementation, it should just delegate the mail sending part (not the mail composing part) to another module specified in the config.

We should then write the Mailgun implementation and the Mock implementaion.

First, let’s rewrite the Mailer, and save the Mailgun parts for later:

defmodule MyApp.Mailer do
  @from "support@myapp.com"
  @mailer_impl Application.fetch_env!(:my_app, :mailer)

  defmodule Behaviour do
    @callback send_mail([key: String.t]) :: :ok
  end

  def send_test_mail(email) do
    :ok = @mailer_impl.send_mail(
      to: email,
      from: @from,
      subject: "Test!",
      text: "Test from MyApp"
    )
  end
end

We keep the composing logic and delegate the sending to a @mailer_impl which we fetch from the config. We also define a behaviour which can be implemented by the different mailer implementations.

Lets write the Mailgun and Mock implementations

defmodule MyApp.Mailer.Mailgun do
  @behaviour MyApp.Mailer.Behaviour

  use Mailgun.Client, [
    domain: Application.fetch_env!(:my_app, :mailgun_domain),
    key: Application.fetch_env!(:my_app, :mailgun_key)
  ]

  def send_mail(mail) do
    {:ok, _} = send_email(mail)
    :ok
  end
end

defmodule MyApp.Mailer.Mock do
  @behaviour MyApp.Mailer.Behaviour

  def start_link do
    Agent.start_link(fn -> [] end, name: __MODULE__)
  end

  def mails do
    Agent.get(__MODULE__, &(&1))
  end

  def clear do
    Agent.update(__MODULE__, fn _mails -> [] end)
  end

  def send_mail(mail) do
    mail = mail |> Enum.into(%{})
    Agent.update(__MODULE__, fn existing_mails -> [mail | existing_mails] end)
    :ok
  end
end

Each implementation just have to implement the send_mail function as defined by the behaviour. The Mailgun mailer is mostly the same as our original Mailer.

The Mock is implemented with an Agent to keep track of which emails are sent.

Before running the tests again, we define the mailer implementation to use in the config.

In config/test.exs we add this:

config :my_app, mailer: MyApp.Mailer.Mock

And in config/config.exs we add this:

config :my_app, mailer: MyApp.Mailer.Mailgun

This sets the Mailgun mailer as the default and overwrites the setting in tests.

Time to run the test again to see it pass:

$ mix test
Compiling 18 files (.ex)

== Compilation error on file lib/my_app/mailer/mailgun.ex ==
** (ArgumentError) application :my_app is not loaded, or the configuration parameter :mailgun_domain is not set
    (elixir) lib/application.ex:261: Application.fetch_env!/2
    lib/my_app/mailer/mailgun.ex:5: (module)
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6

What? We still get the same compilation error about missing mailgun settings!

This is because even though we don’t use the Mailgun mailer in tests, it is still being compiled. We should just add some dummy settings in the test config:

config :my_app, [
  mailgun_domain: "notusedintests",
  mailgun_key: "notusedintests"
]

There! Lets run the tests again. Now it should be all green.

$ mix test
.......

  1) test it uses mock in tests (MyApp.MailerTest)
     test/mailer_test.exs:4
     ** (exit) exited in: GenServer.call(MyApp.Mailer.Mock, {:update, #Function<0.6592501/1 in MyApp.Mailer.Mock.clear/0>}, 5000)
         ** (EXIT) no process
     stacktrace:
       (elixir) lib/gen_server.ex:596: GenServer.call/3
       test/mailer_test.exs:5: (test)



Finished in 0.1 seconds
8 tests, 1 failure

Randomized with seed 356689

Ok, now it compiles at least. Looking at the error message, we see that the Mock agent is not started. This is (doh) because we did not start it. We should add this to our test_helper.exs:

MyApp.Mailer.Mock.start_link

Now it should be green!

$ mix test
........

Finished in 0.09 seconds
8 tests, 0 failures

Randomized with seed 776617

Yay! All green. We have succesfully created a mock (noun) for our mailer.

Afterword

It took me quite some time to figure out when and how to use mocks in Elixir. I think the approach described here is really nice and the most Elixirish of the patterns I have seen.

It is simple
It is pure Elixir. No libs needed.
It is unobtrusive compared to the injection patterns
It treats the mock as any other implementation, which allows for easy substitution of the real implementation.
One could even have a one implementation for production, one for staging, one for dev and one for test.

I hope somebody saves some time by reading this blog post :)

Iterating in Elixir

Lasse Skindstad Ebert — Sun, 20 Oct 2019 06:36:08 +0000

This story was originally posted by me on Medium on 2016-10-07, but moved here since I'm closing my Medium account.

Like many Elixir developers I come from Ruby. Elixir is my first functional language after years of Java, VB.Net, C# and Ruby. And a little bit of Javascript.

One of the first things that stroke me as being different in functional programming, was how I should map, reduce, loop and otherwise iterate data models.

The each

In Ruby we might find something like this:

def concat_messages(items)
  result = ""
  items.each do |item|
    result += item.message if item.ok?
  end
  result
end

The issue of converting this to Elixir, is that everything is immutable in Elixir.

Trying to do the same thing would yield an empty result, since everything that happens inside Enum.each stays inside Enum.each. It’s basically the Las Vegas of iterations. You only bring home the actions with side effects, like writing to the console or getting married.

There are a number of ways to write the same code in Elixir. Let’s try with Enum.filter and Enum.reduce:

def concat_messages(items) do
  items
  |> Enum.filter(&Item.ok?/1)
  |> Enum.reduce("", &(&2 <> Item.message(&1))
end

This looks really nice! It exposes the intention of the code and makes it more readable.

Couldn’t we do the same thing in Ruby? Yes, easily. The only problem is that a lot of the loop-like code with mutable state exists in Ruby code bases. Writing the Elixir code above in Ruby would give us this equivalent:

def concat_messages(items)
  items
    .select(&:ok?)
    .reduce("") { |acc, item| acc + item.message }
end

Be a good Ruby programmer. Don’t mutate state when you don’t need to.

The loop

In many programming languages it is possible to loop and break out of the loop. Something like this:

def fetch_something
  tries = 0
  body = nil
  loop do
    status, body = make_external_http_call
    break if status == 200
    tries += 1
    if tries >= 5
      body = "Service not available" 
      break
    end
  end
  body
end

Trying to get an overview of this code takes a while :( It can break out of the loop two different places, and it mutates state between loop iterations.

How could be better? Let’s force ourselves to immutability with Elixir and rewrite the code:

def fetch_something(tries \\ 0)

def fetch_something(tries) when tries < 5 do
  case make_external_http_call do
    {200, body} -> body
    {_status, _body} -> fetch_something(tries + 1)
  end
end

def fetch_something(tries) when tries >= 5 do
  "Service not available"
end

This was a big improvement in the Ruby code too! Of course, the lack of pattern matching and guard clauses makes it not-as-nice as the Elixir version.

Conclusion

It might be hard at first to rewrite mutable, object-oriented iterations to an immutable functional language. For a long time it felt like I had to think backwards.

But for some reason, immutability forces you to write easy-to-read and concise code, which is clearly a long-term benefit of any software project.

I still write Ruby sometimes, and when I do, I tend to have a more functional code style than before I met Elixir. It helps me reason about my code and I avoid mixing logic and data. I have become a fan of the Ruby keyword module_function which allows you to define a module with pure functions in it, much like in Elixir.

Elixir and immutability is here to stay, and I see a big future for Elixir coming up.

DEV Community: Lasse Skindstad Ebert

TIL: Unicode chars in regex

TL;DR:

Matching a unicode char in a regex

Interpolation works too.

Enum.map_reduce/3

Have a cache while mapping

Build a result while mapping

Enum.flat_map_reduce

TIL: Ecto's pin is coercing

TL;DR

Background

TIL: Understanding dialyzer’s “The pattern can never match the type.”

The setup

Solution?

Embedded Elixir (without Nerves)

My use case

How to add Erlang to a buildroot configuration?

So how is Elixir installed, then?

Deploying a Distillery release to the linux image

Running the release on a read-only file system

Further reading and investigation

Isolating code in contexts

History of The Context Pattern

So what is it, exactly?

Example time

What about OTP and GenServers?

What did I gain?

Elixir PubSub Hub With Pattern Matching

My use case: IoT messages

Flows

The solution

Hub, first version

Hub, with Phoenix Presence

Hub, the standalone version

Subscribe to messages with pattern matching

Subscribe with pattern matching

The solution

Running observer inside a Vagrant box

The :observer

Vagrant

Connecting to a node inside Vagrant

Connecting to a node behind a firewall

Credits

Test driving a Phoenix endpoint, part I

Test driving

Let’s get started!

First test

Adding the route

Adding the controller

Creating the invitation logic

Creating the schema and database table

Going back the the controller test

Edge cases

Check for missing email

Summing up

Mocks in Elixir

Or “Test driving a Phoenix endpoint, part II”

Mocking external calls

Different mocking patterns

Injection with pure Elixir

Injection with a library

Mocking with mock

Compile time settings

Creating the mailer

Test driving the mailer mock

Afterword

Iterating in Elixir

The each

The loop

Conclusion