DEV Community: Miguel Palhas

Alchemy Conf 2025

Miguel Palhas — Thu, 06 Feb 2025 11:44:27 +0000

Alchemy Conf 2025 is happening in Braga, Portugal, from March 31st to April 2nd, 2025! This premier Elixir event includes insightful talks, engaging workshops, and the chance to network with some of the brightest minds in the Elixir community.

🎟️ Tickets at https://alchemyconf.com

It’s a four-day event carefully crafted for Elixir developers and enthusiasts, featuring talks, hands-on workshops, and delightful networking events. The team behind Alchemy Conf are working extremely hard to ensure this will go above and beyond what you’d expect from a conference.

Check out the website for info on talks, workshops, and our beautiful city!

You can also see our recent interview with one of our speakers, Saša Jurić:

Building State Machines in Elixir with Ecto

Miguel Palhas — Thu, 30 Jul 2020 15:46:16 +0000

Among the many useful patterns in computer science, there is the concept of a Finite-state machine (FSM).

It's a great abstraction in many different scenarios, where you want to model a certain process that goes through a predefined set of states, with different behaviors, depending on what state it is in.

In this post, you'll learn how to implement this pattern with Elixir's Ecto and when to use it.

Use Cases

When you model a long-running flow that requires multiple steps, and where each step has different
requirements, a state machine can be a good choice as an abstraction. A few examples:

A user onboarding flow where the user first signs up, then adds some extra required info, confirms his email, then enables 2FA, and only then is allowed into the system
A shopping cart which starts out empty, allows products to be added indefinitely and can proceed to payment/shipping if enough products are added
A task in a project management pipeline. e.g: Tasks start out in the backlog, can be assigned to people, moved to "in progress", and later to "done"

An Example of a Finite-State Machine

For this post, we'll stick with a small example that illustrates the flow of a state machine: A door.

A door can be locked or unlocked. It can also be opened or closed. While unlocked, it can also be opened.

We could model this as a finite-state machine, such as the following:

This FSM has:

3 possible states: Locked, Unlocked, Opened
4 possible transitions or events: Unlock, Open, Close, Lock

It can be inferred from the diagram that it's impossible to transition from Locked to Opened. Or in plain words: you need to
unlock the door first.
The state machine diagram describes the behavior. But how can we go about implementing it?

State Machines as Elixir Processes

Since OTP 19, Erlang provides a :gen_statem module that allows implementing gen_server-like processes that behave as state machines (where the current state influences their behavior). Let's see what that would look like for our door
example:

defmodule Door do
  @behaviour :gen_statem

  def start_link do
    :gen_statem.start_link( __MODULE__,:ok,[] )
  end

  @impl :gen_statem
  def init(_), do: {:ok, :locked, nil}

  @impl :gen_statem
  def callback_mode, do: :handle_event_function

  @impl :gen_statem
  def handle_event({:call, from}, :unlock, :locked, data) do
    {:next_state, :unlocked, data, [{:reply, from, {:ok, :unlocked}}]}
  end

  def handle_event({:call, from}, :lock, :unlocked, data) do
    {:next_state, :locked, data, [{:reply, from, {:ok, :locked}}]}
  end

  def handle_event({:call, from}, :open, :unlocked, data) do
    {:next_state, :opened, data, [{:reply, from, {:ok, :opened}}]}
  end

  def handle_event({:call, from}, :close, :opened, data) do
    {:next_state, :unlocked, data, [{:reply, from, {:ok, :unlocked}}]}
  end

  def handle_event({:call, from}, _event, _content, data) do
    {:keep_state, data, [{:reply, from, {:error, "invalid transition"}}]}
  end
end

This implements a process that starts out in the :locked state. By sending appropriate events, we are able to match
the current state with the transition requested and perform the required transformations. An additional data argument
is kept for any additional state that is needed, but we're not using that in this case.

To use this process, you can call it with the desired transition that you want to execute. If the current state allows that
transition, it will work. Otherwise, an error is returned (due to the last, catch-all match of the code snippet).

{:ok, pid} = Door.start_link()

:gen_statem.call(pid, :unlock)
# {:ok, :unlocked}

:gen_statem.call(pid, :open)
# {:ok, :opened}

:gen_statem.call(pid, :close)
# {:ok, :closed}

:gen_statem.call(pid, :lock)
# {:ok, :locked}

:gen_statem.call(pid, :open)
# {:error, "invalid transition"}

If our state machine is more data-oriented than process-oriented, we may want to go with a different
approach...

State Machines as Ecto Models

There are a couple of Elixir packages that deal with this problem. For this post, I'll be using
fsmx, but other packages such as machinery also provide similar features.

This package allows us to model the same kind of states and transitions within an existing Ecto Model:

defmodule PersistedDoor do
  use Ecto.Schema

  schema "doors" do
    field :state, :string, default: "locked"
    field :terms_and_conditions, :boolean
  end

  use Fsmx.Struct, transitions: %{
    "locked" => "unlocked",
    "unlocked" => ["locked", "opened"],
    "opened" => "unlocked"
  }
end

We can see that Fsmx.Struct receives all possible transitions as an argument. This allows it to check for unwanted
transitions and prevent them from happening. Now, we can transition using a traditional, non-Ecto approach:

door = %PersistedDoor{state: "locked"}

Fsmx.transition(door, "unlocked")
# {:ok, %PersistedDoor{state: "unlocked", color: nil}}

But we can also ask for the same in the form of an Ecto changeset:

# get an existing door from the Database
door = PersistedDoor |> Repo.one()

Fsmx.transition_changeset(door, "unlocked")
|> Repo.update()

This changeset only updates the :state field. But we can extend it to include additional fields, as well as
validations. Let's say that, in order to open a door, we need to accept its terms & conditions:

defmodule PersistedDoor do
  # ...

  def transition(changeset, _from, "opened", params) do
    changeset
    |> cast(params, [:terms_and_conditions])
    |> validate_acceptance(:terms_and_conditions)
  end
end

Fsmx looks for an optional transition_changeset/4 function in your schema and calls it with the previous state as
well as the next one. You can pattern match on those to add specific conditions for each transition.

Dealing With Side Effects

It's one thing to transition the state machine itself and move forward with the state.
But another big benefit of state machines is the ability to deal with particular side effects that come out of each
state.

Let's say, for example, you want to get notified every time someone unlocks your door. You might want to trigger an email
when it happens. But you want these two operations to be a single, atomic operation.

Ecto deals with atomicity via Ecto.Multi which groups multiple operations inside a database transaction. It also has
an Ecto.Multi.run/3 function that allows you to run arbitrary code within that same transaction.

Fsmx in turn, integrates with Ecto.Multi by providing you with a way to run state transitions as part of an
Ecto.Multi, while also providing you an additional callback that is executed in that case:

defmodule PersistedDoor do
  # ...

  def after_transaction_multi(changeset, _from, "unlocked", params) do
    Emails.door_unlocked()
    |> Mailer.deliver_later()
  end
end

Now, you can execute a transition as shown:

door = PersistedDoor |> Repo.one()

Ecto.Multi.new()
|> Fsmx.transition_multi(schema, "transition-id", "unlocked")
|> Repo.transaction()

This transaction will use the same transition_changeset/4 from above to compute the necessary changes to the model
and will include the new callback as an Ecto.Multi.run call. The result is that an email is sent (asynchronously,
using Bamboo, so as not to run within the transaction itself). If the
changeset is invalidated for some reason, the email ends up never being sent, resulting in an atomic execution of both
operations.

Conclusion

Next time you're modeling some kind of stateful behavior, consider looking into a state-machine approach. Regardless of
which flavor you use, the ability to translate a concrete diagram to actual code and the ability to test each state,
transition, and side effect as a separate piece is a huge advantage.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Guest author Miguel is a professional over-engineer at Portuguese-based Subvisual. He works mostly with Elixir, DevOps, and Rust. He likes to build fancy keyboards and playing excessive amounts of online chess.

Best Practices for Background Jobs in Elixir

Miguel Palhas — Thu, 16 Jul 2020 09:51:41 +0000

Erlang & Elixir are ready for asynchronous work right off the bat. Generally speaking, background job systems aren't
needed as much as in other ecosystems but they still have their place for particular use cases.

This post goes through a few best practices I often try to think of in advance when writing background jobs, so that I don't hit some of the pain points that have hurt me multiple times in the past.

If you've ever deployed a new task, only to find out that it has gone rogue with a bug that caused it to misbehave
(e.g.: sending way too many emails, way too quickly), you may have gone through similar bugs as well.

Flavours

Elixir already gives you the ability to schedule asynchronous work pretty easily. Something as simple as this already
covers a lot:

Task.async(fn ->
  # some heavy lifting
end)

You might need something a bit more powerful, either just for convenience (having some tooling & monitoring around
that task), or because you need something like periodic jobs. Again, all this can be achieved with something like
a GenServer:

defmodule PeriodicJob do
  use GenServer

  @period 60_000

  def init do
    Process.send_after(self(), :poll, @period)

    {:ok, :state}
  end

  def handle_info(:poll, state) do
    # some heavy lifting

    Process.send_after(self(), :poll, @period)
  end
end

You can also use a job queuing library such as Quantum. If you come from Ruby land and are used to libraries such as
Sidekiq, you might be more familiar with something like this:

#
# lib/my_app/scheduler.ex
#
defmodule MyApp.Scheduler do
  use Quantum.Scheduler, opt_app: :my_app
end

#
# config/config.exs
#
config :my_app, MyApp.Scheduler,
  jobs: [
    first: [
      # every hour
      schedule: "0 * * * *",
      task: {MyApp.ExampleJob, :run, []}
    ],
    second: [
      # every minute
      schedule: "* * * * *",
      task: {MyApp.AnotherExampleJob, :run, []}
    ]
  ]

Some may argue that since Erlang/OTP already provides the infrastructure for creating these processes, packages such
as Quantum are not necessary. However, the structure created by them can end up being more intuitive, especially if you're
not that familiar with OTP. This might be the case with someone coming from Ruby or other such communities.

How to Structure Background Jobs

Let's now get into a few tips that will help you keep your jobs ready to deal with potential future problems!

Most of them are preventive measures due to the fact that all of these are background processes. They're not responding
to an HTTP request and they happen without any intervention, thus sometimes, debugging can be hard if you don't take some
precautions.

Let's consider a small example that sends confirmation emails to users that haven't received it yet:

defmodule MyApp.ExampleJob do
  def run do
    get_users()
    |> Enum.each(fn user ->
      # send single email to user
    end)
  end

  defp get_users do
    MyApp.User
    |> where(confirmation_email_sent: false)
    |> MyApp.Repo.all()
  end
end

1. Put in a Kill Switch

This is one of those mistakes I'll never make again since it has hurt me so many times.

Let's say you've created a background job, tested, deployed, and configured it to run periodically and send some emails.

It hits production, and you soon notice that something's wrong. The same 100 people are being spammed with emails every
minute. You messed up the geth_next_batch/1 function, and it always goes over the same batch of users.
It's a developer's horror story. You need to fix it (or kill it) quickly, but all that time waiting for a new release to get
online is physically painful.

So, avoid that:

defmodule MyApp.ExampleJob do
  def run do
    return if !enabled?

    # ...
  end

  defp enabled? do
    # check a Redis flag, or a database record, or anything really
  end
end

You can plug in some persistent system that allows you to quickly toggle the job on/off. A good suggestion would be to
use a feature flag package, such as FunWithFlags.

2. Always Batch Your Jobs by Default

It's easy to miss this one on a first draft. You're just trying to quickly get something online.
But in some cases, it may be important to not hurt your performance if you're working on a very resource-intensive job,
or simply, if your list of records to process grows too quickly.

Doing User |> where(confirmation_email_sent: false) |> Repo.all() can be dangerous if there's potential for that to
yield too many results.
You may end up consuming too many resources for something that could be done in smaller batches, keeping your system
a lot more stable:

defmodule MyApp.ExampleJob do
  @batch_size 100

  defp get_users do
    MyApp.User
    |> where(confirmation_email_sent: false)
    |> limit(^@batch_size)
    |> MyApp.Repo.all()
  end
end

Whatever job queue mechanism you plug this worker into, it will end up being called frequently. So you shouldn't hurry
in processing smaller batches one at a time.

3. Avoid Overlaps

This is kind of related to the previous point, but it's a concern that goes beyond performance.

If you program a job to run every minute, and a single execution has the potential to last longer than that, you end up
risking cascading performance problems, or even worse, race conditions, where the first and second executions are both
trying to process the same set of data, and conflict with each other in the process.

This is obviously dependent on what your exact business logic is, but as a general rule, it's best to be defensive here.

If you use a GenServer approach like the one showcased above, this is solved automatically, as instead of scheduling
jobs every minute, you can instead use Process.send_after(self(), :poll, delay) to only schedule the next run after
the current one has finished, avoiding overlap.

When using Quantum, you also have an overlap: true option that you can add to automatically prevent this:

config :my_app, MyApp.Scheduler,
  jobs: [
    first: [
      # every hour
      schedule: "0 * * * *",
      task: {MyApp.ExampleJob, :run, []}
      overlap: false
    ]

This, by the way, might already be reason enough to consider using a package rather than just plain Elixir.

4. Plug in a Manual Mode

If your job is processing a batch of records, it's useful to plug in some public functions that allow you to manually
process specific records. This can serve two purposes:

Better ability to debug the job
Ability to do a few manual runs before enabling the global job (by toggling the feature flag discussed above)

A sample structure could look like this:

defmodule MyApp.ExampleJob do
  import MyApp.Lock

  def run do
    lock("example_job", fn ->
      get_users()
      |> Enum.each(&process_user/1)
    end)
  end

  def run_manually(users) when is_list(users) do
    lock("example_job", fn ->
      users
      |> Enum.each(&process_user/1)
    end)
  end

  def run_manually(user), do: run_manually([user])

  def process_user(%User{} = user) do
    # process a single user
  end
end

In this case, we're creating a run_manually/1 public function that can receive either a single user or a batch of
them and performs the same logic as the automatic job would.

One important detail here is to again avoid a race condition, which in this case, is being done with a custom Lock
module that uses the redis_mutex package to prevent potential issues:

defmodule MyApp.Lock do
  use RedisMutex
  require Logger

  def lock(lock_name, fun) do
    with_lock(lock_name, 60_000) do
      fun.()
    end
  rescue
    _e in RedisMutex.Error ->
      Logger.debug("#{locker_name} another process already running")
  end
end

The lock, which is invoked both on manual runs as well as the regular background job, ensures that you won't cause any
unintentional conflicts if you try to do a manual run at the same time the job is doing the same processing. It also
happens to solve the overlap problem discussed previously in this post.

Conclusion

All these tips come from something that I bumped into in the past, usually related to production bugs or user
complaints. So I hope some of them help you avoid the same mistakes. Let me know if you
have any further thoughts! 👋

Guest author Miguel is a professional over-engineer at Portuguese-based Subvisual. He currently works with UTRUST as well, being the head of infrastructure & blockchain integrations. He builds fancy keyboards and plays excessive amounts of online chess.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Migrating Production Data in Elixir

Miguel Palhas — Wed, 04 Mar 2020 15:41:05 +0000

When requirements change for your product, there arises a need to change not only the codebase but also the existing data that already lives in production.

If you're performing the changes locally, the whole process seems fairly simple. You test your new feature against a sparkling clean database, the test suite is green, and the feature looks great. Then you deploy, and everything goes to hell because you forgot that production was in a slightly different state.

In this post, you'll learn how to handle migrations that may involve systems other than the database itself, while keeping the entire process idempotent and backward-compatible.

Prerequisites

The post mostly requires knowledge of how database migrations work in frameworks such as Ruby on Rails and Phoenix. It also helps to have some knowledge of the constraints of deploying updates to an application that already has existing data in use.

If you've ever dealt with a production system, this is probably nothing new to you.
Other tools, such as Redis, are mentioned, but you don't need to know anything about them, other than the fact that they exist.

Migrating Feature Flags to Redis

Here's an example of a list of feature flags, or settings, for your application:

Feature            | Enabled
2fa_sms            | true
notifications_beta | false
pagination         | true

You now need to migrate these over to Redis, to take advantage of its Pub-Sub and notify all of your services when a flag gets toggled.

You need to somehow get these flags over to the other system, without breaking the behavior in production.

One simple way to do this without having to patch your code to temporarily manage two storage systems simultaneously is to automatically perform this migration right after the deploy.
This is typically where database migrations happen. But this is no common database migration. It’s not even a good idea to do it all at once since it touches a Redis instance, which has nothing to do with your database.
To solve this, you can write a mechanism similar to those migrations, but tailored to your own needs. Imagine the following script:

You need to update your order management to also include the converted price in EUR (because the finance department needs it).

defmodule DataMigrator.Migrations.MigrateFlagsToRedis do
  import Ecto.{Query, Changeset}
  alias App.{Flag, Repo, Redis}

  def run do
    Redis.start_link()

    |> Repo.all(Flag)
    |> Enum.each(&migrate_flag/1)

    :ok
  end

  defp migrate_flag(%Flag{name: name, enabled: enabled}) do
    Redis.put(name, enabled)
  end
end

But how do we guarantee that this runs automatically? And even more important, how do we make this idempotent? We certainly don’t want this running again by accident a few days later when the database flags have long become outdated.

Data Migrations

This is where the concept of migrations comes in handy. Ecto migrations keep a table in PostgreSQL tracking which migrations have already been executed. On every run, the system checks which ones are new and runs only those. We can do something similar ourselves.

First, let’s write a function that looks for new migrations at ./priv/data_migrations for any new files:

defmodule App.DataMigrator do
  alias App.Repo
  alias DataMigrator.DataMigration

  def new_migrations do
    already_migrated = Repo.all(from dm in DataMigration, select: dm.version)

    Application.app_dir(:app, "priv/data_migrations/")
    |> Path.join("*")
    |> Path.wildcard()
    |> Enum.map(&extract_migration_info/1)
    |> Enum.reject(fn
      # reject migrations that have already ran
      {version, _, _} -> Enum.member?(migrated_versions, version)
      _ -> true
    end)
    |> Enum.map(fn {version, _name, file} ->
      # load elixir module in given file
      [{mod, _}] = Code.load_file(file)
      {mod, version}
    end)
  end

  defp extract_migration_info(file) do
    file
    |> Path.basename()
    |> Path.rootname()
    |> Integer.parse()
    |> case do
      {integer, _} ->
        {integer, file}

      _ ->
        nil
    end
end

With this snippet, we can write data migrations into ./priv/data_migration using a name similar to what is done for database migrations, such as 202002111319_migrate_flags_to_redis.exs

The new_migrations/0 function will find those files, filter out the ones that are present in a DataMigration model (meaning migrations that have already run, and return that as [{202002111319, DataMigrator.Migrations.MigrateFlagsToRedis}]

The Migrator

Next, it’s purely a matter of writing a task that runs all new migrations and stores their timestamp in the DataMigration:

defmodule DataMigrator do
  ...

  def run() do
    Application.load(:app)
    Repo.start_link()

    new_migrations()
    |> Enum.map(&execute_data_migration/1)
  end

  def execute_data_migration({version, mod}) do
    case mod.run() do
      :ok ->
        # migration successful, track it
        Repo.insert!(%DataMigration{
          version: version,
          created_at: DateTime.truncate(DateTime.utc_now(), :second)
        })

        :ok

      {:error, error} ->
        # migration failed, most likely a bug that needs fixing
        Logger.error("Data migration failed with error #{inspect(error)}")
        System.halt(1)
    end
  end
end

Finally, using Mix releases, we can easily set this up as a pre-start hook for the next time our application is deployed:

#rel/start/10_data_migrations.sh

$RELEASE_ROOT_DIR/bin/app command Elixir.App.DataMigrator run

By doing this, we’re able to immediately change our code to use Redis instead of PostgreSQL flags, as well as having a guarantee that once our refactor is deployed, all flags will automatically be migrated, without us needing to ensure backward compatibility.

Conclusion

With these suggestions, you should be able to simplify some otherwise painful migrations within your system, particularly those that involve keeping both an old and a new system living in parallel. Instead, you can just automate the migration to the new one, and have this as a hook that gets triggered automatically once you deploy.

Typespecs and Behaviours in Elixir

Miguel Palhas — Fri, 18 Oct 2019 15:17:02 +0000

Today, we will dive into Typespecs and Behaviours. These are two Elixir features that we are ecstatic (pun intended) about. They are great examples of built-in features in Elixir that help get some of the advantages of statically typed code.

Dynamically Typed with Features

Alright, let's set the scene. Elixir is a dynamically typed language. This means that the type of each variable is not checked at compile-time, but rather at run-time. Like most things, this comes with advantages and disadvantages.

The differences between statically and dynamically typed languages are sometimes the cause of heated debate, and there's already a lot of material out there. This post provides a good comparison, and Chris Smith's article is also a great dive into some of the fallacies that come when discussing type systems.

Despite being dynamically-typed, Elixir does a pretty good job of providing some opt-in features to get some of the safety of statically typed languages. This is important because those features often provide important guarantees about your code. This is usually done by performing static analysis on your code and, with the help of the type system, catch mistakes early on.

The two main examples of this are Typespecs and Behaviours.

Typespecs

Typespecs is an opt-in feature of Elixir that lets you annotate your functions to provide hints to the language as to what your function headers should look like. Like this:

defmodule Foo do
  @spec bar(arg :: binary) :: number
  def bar(arg) do
    String.length(arg)
  end
end

The @spec keyword lets you specify what the argument names and types should be, as well as the return type.

This doesn't cause any kind of compilation failure if the types don't match (again, Elixir is dynamically typed, so the types aren't actually enforced at compile-time). But it has two other main benefits:

It allows for other tools to be built, which will perform static analysis on the code, and use these annotations to inform you if something looks wrong. dialyxir is a popular tool for this;
It serves as documentation so that anyone looking at your public API can clearly see what to expect.

Elixir provides a set of basic types that you can use in these specifications. binary, pid and number are some of them (check the official docs for more on this). But it also allows you to compose these basic types into more complex, custom ones, using the @type directive:

defmodule Foo do
  @type result :: {number, binary}

  @spec bar(arg :: binary) :: result
  def bar(arg) do
    {String.length(arg), arg}
  end
end

Behaviours

Alright, now we take it to the next level and discuss behaviours. You can think of Behaviours as a kind of interface specification, like what you usually get in object-oriented languages.

Behaviours allow you to specify a contract for your modules and force them to respond to a specific API. This allows you to decouple features, using adapter patterns and other such programming techniques to piece together your code.

The upper layers of an application don't really need to care if data is persisted into PostgreSQL, MongoDB, or some other database. That's because Ecto provides a common language (API) to interact with adapters for these storage backends.

A behaviour specifies a list of function headers, here called callbacks. Any other Elixir module which claims to implement said behaviour will have to define those callbacks and their implementation. If one is missing, a compiler warning will be issued, letting the programmer know something's wrong.

An example behaviour might look something like this:

defmodule MyApp.Language do
  @callback greet(name :: binary) :: binary
  @callback thank :: binary
end

This behaviour defines two function headers. These are defined just like you would a typespec, except that @callback is used, instead of @spec.

Now we can write some implementations of our language behaviour:

defmodule MyApp.English do
  @behaviour MyApp.Language

  def greet(name), do: "Hello, #{name}"
  def thank, do: "Thank you"
end

defmodule MyApp.Portuguese do
  @behaviour MyApp.Language

  def greet(_name), do: "TODO"
  def thank, do: "Obrigado"
end

defmodule MyApp.Japanese do
  @behaviour MyApp.Language

  def greet, do: "TODO"
  def thank, do: "TODO"
end

This last implementation will throw a warning because we're failing to fulfill the contract. greet should actually take an argument. And even if we don't use it, we still need to expect it. greet/0 and greet/1 would be two different functions in Elixir, and the behaviour expects the latter.

warning: function greet/1 required by behaviour MyApp.Language is not implemented (in module MyApp.Japanese)
  test.ex:20

Note that this is not a compilation failure, just a warning. These annotations are only meant to guide your development and warn you of potential mistakes. It’s up to you to know what to do with them.

But before looking into a real-life example of this, we need to discuss a pattern that is commonly associated with behaviours and interfaces...

The Adapter Pattern

The Adapter pattern is a well-known software development pattern, described in detail by the Gang of Four’s book on the subject. In short, it’s about building public interfaces within your code, such that pieces can be swapped with other functionally-equivalent pieces while keeping everything compatible.

There are two main benefits to this:

It promotes decoupling. By enforcing that modules only talk with other modules via the specified interfaces, it doesn’t matter what the underlying implementation is. As long as that part remains stable, inner refactors of your code can be made with a lot more confidence that compatibility won’t be broken
It makes it easy to switch between multiple options. Ecto, as mentioned above, is a great example of this. While writing queries with it, you don’t really care if your backend is PostgreSQL, MongoDB, or something else. Ecto’s query language remains the same, and each adapter takes care of translating that to its own language.

As you may guess by now, the go-to way of creating adapters in Elixir is by using behaviours.

An Example Project

To demonstrate the usefulness of behaviours, I’ll take advantage of a real project for which I contributed.

fun_with_flags is an awesome Elixir library for dealing with feature flags. It’s also one of the better-named projects out there

Within my projects, I often felt the need to make feature flags known to my unit tests. Perhaps I’m writing tests to a disabled feature that hasn’t gone live yet, and therefore need to enable it in those tests, to trigger the correct code paths. Or perhaps I want to test how the program responds to different flag values (e.g.: rolling releases).

Either way, I want the ability to enable/disable flags in tests. But the two existing adapters pose limitations to this. Spinning a Redis instance for my test suite seems too much. And using PostgreSQL would require setting up Ecto Sandbox, and giving up on async: true completely for any related tests.

The ideal scenario was to have all this run in memory. Which we can, thanks to the adapter pattern that was chosen.

An InMemory Adapter

And here we go. Everything folded together.

The bulk of the work is to create a module that implements the FunWithFlags.Store.Persistent behaviour. All functions listed in the behaviour (worker_spec/0, get/1, put/1, delete/1, all_flags/0 and all_flag_names/1) need to be implemented in our adapter.

defmodule FunWithFlags.Store.Persistent.InMemory do
  @behaviour FunWithFlags.Store.Persistent
  # ...

  def start_link(opts \\ []) do
    GenServer.start_link(__MODULE__, opts, name: opts: __MODULE__)
  end

  def init(_), do: {:ok, []}

  def get(flag_name) do
    GenServer.call(__MODULE__, {:get, flag_name})
  end

  def put(flag_name, gate) do
    GenServer.call(__MODULE__, {:put, flag_name, gate})
  end

  # ...

  def handle_call({:get, flag_name}, _from, state) do
    # ...
    # search for the given flag in the state, and return it's status
  end

  def handle_call({:put, flag_name, gate}, _from, state) do
    # ...
    # insert the given gate into the current state
  end

  # ...
end

This part of the implementation shows how the get/1 and put/1 functions are hooked up. The module is a GenServer to allow it to store and retrieve data without having to persist it to a database.

Note that I avoided displaying the actual implementation of the various handle_call/3 functions because they’re rather bulky and already beside the point of this post. But the good news is, this is actually published as a hex package, and you can check it out on Github too!

Summary

Now we’ve gone all the way into the rabbit hole of this post, from theory to practice. From TypeSpecs, and how behaviours are a cool implementation of Adapter patterns in Elixir to the real-life example. We even got out at the other end on our best behaviour ;-)

Metaprogramming: From C Preprocessing to Elixir Macros

Miguel Palhas — Tue, 16 Jul 2019 11:56:54 +0000

Developers have a love-hate relationship with metaprogramming. On the one hand, it’s a powerful tool to build reusable code. On the other hand, it can quickly become hard to understand and maintain.

I like to think of it as salt. It’s pretty handy on many occasions, but use just a little too much of it, and you’re left with an unenjoyable dish.

Also, large doses of either of them can lead to increased blood pressure. 😅

However, metaprogramming has come a long way since it’s early days. While I still try not to overuse it, it’s become more useful and easy to work with. Let’s see how it evolved.

C/C++

If we go back a few decades, to a time when programming languages were more close to the metal, the C/C++ preprocessor was one of the only options we had to do something close to metaprogramming.

This preprocessor was literally what the name suggests: A parser that would run through C code, and process specific definitions (keywords such as #define and #if), and would output a final version of the C code to the compiler. This final version could change based on some criteria. It would look something like this:

#define FOO 1

#if FOO == 1
#define MSG "Hello, World"
#else
#define MSG "Goodbye, World"
#endif

#include <stdio.h>

int main() {
  printf(MSG);
end

This program would print "Hello, World", always. As you may guess, changing the FOO definition to 0, and re-compiling the program, would instead cause it to print "Goodbye, World" instead.

These preprocessor directives would often be used to create code targeting specific platforms or architectures. For example, you could set different behaviors for your program when compiled to target Windows systems than when targeting Linux systems. The two resulting binaries would have only the code that was relevant to that specific platform, and thus wouldn’t need to perform runtime checks for these conditions. These savings in storage and runtime performance could often be significant.

However, if you have any C experience at all, you know how dangerous it is just in vanilla form. Now add a lot of preprocessing behavior on top of that, and it quickly becomes quite hard to manage. So it wouldn’t be advisable to use it for much more than small configurations, most of the time.

Ruby

With better technology and higher-level scripting languages, also came the possibility of creating more elaborate styles of programming. Particularly in Ruby, metaprogramming proved to be a powerful, yet scary feature.

The way this works in Ruby is based on the idea that code is nothing more than a string of text, interpreted and executed by the Ruby environment.

Since Ruby is interpreted at runtime, there’s no requirement of having the entire codebase compiled upfront. Ruby allows you to dynamically define instance methods on classes.

Also, due to the way Ruby classes and instances are constructed internally, you can even define methods for individual instances rather than the entire class!

PS: Further reading on Ruby Classes here.

class Foo
  def hello1
    puts "Hello from a regular method"
  end

  [:hello2, :hello3].each do |f|
    define_method f do
      puts "Hello from a dynamically-defined #{f} method"
    end
  end
end

foo = Foo.new

foo.define_singleton_method(:hello4) { puts "Hello only from this instance of Foo" }

foo.hello1
foo.hello2
foo.hello3
foo.hello4

Ruby is also pretty lax when it comes to editing existing code, even from the standard library. This is valid Ruby:

array = [1, 2, 3]

# will print out 3
puts array.size

class Array
  def size
    "Hello"
  end
end

# will now print out "Hello"
puts array.size

Don’t to that, though! It will most likely break your program and is a bad practice overall.

Last but not least, Ruby has some powerful ways of handling unexpected function calls, such as the method_missing callback:

array = [1, 2, 3]

class Array
  def method_missing(method, *args)
    puts "#{method} method not found"

    if method == :sise then
      puts "Did you intend to type size instead?"
    end
  end
end

puts array.sise

Overall, these abilities were a big game-changer for me when I first learned about them. It enabled me to think about my codebase in a whole new different way and improve it in the process.

There were some issues, though. You know what they say: with great power comes great responsibility.

Several Ruby libraries used and abused these metaprogramming mechanisms to create their own Domain Specific Languages. In the long run, this overuse would result in similar problems as we had in C++ times: difficulty maintaining and understanding a codebase.

Elixir took, in my opinion, yet another step forward in the right direction here…

Elixir ❤️

Here, metaprogramming is built into the language’s core in a much more powerful way. Whereas Ruby allowed you to define methods dynamically, or event generate a string and evaluate it as code (the old eval method that we all hate), Elixir allows you to mess with the Abstract Syntax Tree (AST) itself.

This is done through the quote keyword:

iex> expr = quote do
  "Hello, " <> "World"
end

Trying out the above code, you’ll find that the string concat operation doesn’t get executed directly. Instead of a final string, you end up with an AST expression that describes your code:

{:<>, [context: Elixir, import: Kernel], ["Hello, ", "World"]}

Those familiar with Polish Notation may quickly identify that this is equivalent to the string concatenation code from above. So by quoting some code, you get an AST description of that code, which you can then use across the rest of your codebase.

You can then start to reason about your code as if it were a data structure (which it is… an AST), and perform operations to transform it:

Let’s modify things a little bit:

iex> expr = quote do
  "Hello, " <> name
end

Now our expression uses a dynamic name instead. However, where does that name come from?
We don’t have that variable defined anywhere, but it is still syntactically correct:

{:<>, [context: Elixir, import: Kernel], ["Hello, ", {:name, [], Elixir}]}

However, it will fail to execute, which we can test by using Code.eval_quoted/3:

iex> Code.eval_quoted(expr)
** (CompileError) nofile:1: undefined function name/0
    (elixir) lib/code.ex:590: Code.eval_quoted/3
    test.ex:5: (file)

Let’s now create a second AST definition:

definition = quote do
  name = "Miguel"
end

This second expression definition defines a variable called name. However, remember, we’re not defining any value, just creating the AST for that operation.

We can combine these two expressions into a single one:

final_code = quote do
  unquote(definition)
  unquote(expr)
end

This ends up having the same result as if we had typed:

name = "Miguel"
"Hello, " <> name

However, notice we never had to abandon the Elixir syntax and rules while doing so. We’re writing Elixir that writes Elixir!

This is heavily used internally within Elixir’s core. Whenever you define a function, or a simple if statement, you’re executing macros that change the code’s AST according to fit your code into them. Speaking of which…

Elixir’s Macros

Much of Elixir’s features are written with macros. Many of the common operators you use can be rewritten with macros. Let’s take, for instance, the unless operator (which already exists in the language’s core) and define it ourselves:

defmodule Foo do
  defmacro custom_unless(condition, do: do_clause, else: else_clause) do quote do
      if !unquote(condition) do
        unquote(do_clause)
      else
        unquote(else_clause)
      end
    end
  end

  defmacro custom_unless(condition, do: do_clause) do
    quote do
      Foo.custom_unless(unquote(condition), do: unquote(do_clause), else: nil)
    end
  end
end

defmodule Bar do
  require Foo

  Foo.custom_unless true, do: IO.puts("not true"), else: IO.puts("true")
end

Our custom_unless macro take in a boolean value. Inside, we check for the opposite of the condition (we run whatever code AST was given on that condition, and invert the resulting boolean). Then we execute the AST given for either the do or the else clause, depending on the result.

However, the fun part about Elixir is that, since even the basic constructs such as if clauses are often built using macros themselves, we can better embed our macros in the language. In other words, after defining our macro, this is also working Elixir code:

defmodule Bar
  # importing instead of requiring allows us to call the macro directly,
  # without the Foo. prefix
  import Foo

  custom_unless true do
    IO.puts("not true")
  else
    IO.puts("true")
  end
end

This works because the interpretation of a multiline if/else block in Elixir is not much more than syntactic sugar for:

if condition do: something, else: something_else

Conclusion

Hopefully, this has been a useful walkthrough of how macros evolved in the past, especially for Elixir developers that may not know the full power of their language, as well as the history.

If you want to get a regular dose of ⚗️Elixir Alchemy, subscribe to get the next episode of Alchemy delivered straight to your inbox.

This post is written by guest author Miguel Palhas. Miguel is a professional over-engineer @subvisual and organizes @rubyconfpt and @MirrorConf.

Routing in Phoenix Umbrella Apps

Miguel Palhas — Tue, 16 Apr 2019 12:37:24 +0000

Umbrella apps are an awesome way to structure Elixir projects.

Behind the curtains, they are a very thin layer that just compiles everything to a single package. Instead of building a single large monolith, you can structure your code with multiple isolated contexts. They all get compiled and run under the same BEAM instance, so they still have access to each other. Meanwhile the conceptual separation ensures you have separate OTP apps for each of your umbrella children. And it allows you to work on each of them with a certain degree of isolation.

Think of this as a poor man’s microservices solution. You don’t need to add a messaging queue or send HTTP requests between each service since they’re all actually running under the same process, but you still get some of the benefits.

If you want to know more about umbrella applications, I suggest the official guide as a starter, as it clearly outlines the advantages and caveats of umbrella apps.

Now let's look at a real life example where I've implemented an umbrella app.

A Real Example

Let’s say I’m building a website for Magic: The Gathering (MTG) cards. Which… well, I am. The idea is to create an interface where users can browse and search a database of cards. There’s also an admin panel where some administrative tasks can be performed.

Clearly, each of these frontend interfaces has different requirements:

The main frontend is public while the admin side only has private access.
The admin panel may even have its own UI requirements. In this case, I’m using ex_admin for convenience. This means, even UI assets are not shared.
They mostly have completely different back-end logic as well. Only a small subset of the queries and operations can be shared between the two.
I may also want to access both of them through different URLs (e.g. use an admin subdomain for the Admin frontend).

The multiple differences between the two make it clear that it would be better for these to be two separate phoenix apps—each with its own setup.

Something like this:

apps/
  client/
  admin/
  shared/

Looks Easy Enough. What’s the Issue?

The problem comes when you try to figure out how to actually implement this. How do you route requests from the admin subdomain to another Phoenix app while routing other requests to the main Phoenix app?

One solution would be to run each of those apps on a different port. But then, you'll either be left accessing admin.mydomain.com:4001, or you’ll need some other middle layer to abstract away that port distinction from your browser. While this may be fine for an admin page that only you will access, it doesn’t work as well for a general solution.

The old school solution is to put a reverse proxy between your clients and your server. nginx does this job pretty well. But in reality, you know all this is a single Elixir application. It seems weird to need a third party server to be able to route requests to different parts of it.

It also doesn’t solve the problem of local development, unless you want to run nginx locally as well, which is less than ideal.

We’re Elixir developers after all, and we’re pretty smart. So let’s do this the Elixir way:

Introducing a Proxy App

The solution I came up with (i.e. read suggestions from similar use cases on Stack Overflow) was to create an additional umbrella child, which will be the main point of contact to the outside world.

This app, which we’ll call proxy, will receive all incoming HTTP requests and forward them to the appropriate Phoenix app, based on a few simples rules. In our simple use case, requests to admin.mydomain.com will be forwarded to the admin app, and all others will be forwarded to the client app.

This is a very simple phoenix app, which you can generate with mix phx.new like all the others. Dependencies will be kept to a minimum here. We only have phoenix & cowboy as external dependencies (to set up our web server), as well as the client and admin apps to which we’ll be forwarding requests:

def deps do
  [
    {:client, in_umbrella: true},
    {:admin, in_umbrella: true},
    {:phoenix, "~> 1.3.2"},
    {:cowboy, "~> 1.0"}
  ]
end

Since this app will be the actual web server, we should disable the server setting in the other two:

# apps/client/config/config.exs
config :client, Client.Web.Endpoint, server: false

# apps/admin/config/config.exs
config :admin, Admin.Web.Endpoint, server: false

# apps/proxy/config/config.exs
config :proxy, Proxy.Endpoint, server: true

This ensures that only the proxy app will be listening to a port. This is not mandatory but it saves you the trouble of having to define different ports for each one (remember: only one listener per port is allowed) and ensures all requests actually go through the proxy app—which is indeed the expected behavior.

Leaving server: true might be useful in development or testing mode, depending on how you want to set up your environment.

Setting up the Endpoint

The entry point of a Phoenix app is the Endpoint module. In this case, we’ve set this to Proxy.Endpoint. Since this app really has no other responsibility, there’s no need to nest it under the Web module, as is common practice in Phoenix.

However, we can strip down most things from the Endpoint module created for us by the Phoenix generator and end up with a very simple module:

defmodule Proxy.Endpoint do
  use Phoenix.Endpoint, otp_app: :proxy

  @base_host_regex ~r|\.?mydomain.*$|
  @subdomains %{
    "admin" => Admin.Web.Endpoint,
    "client" => Client.Web.Endpoint
  }
  @default_host Client.Web.Endpoint

  def init(opts), do: opts

  def call(conn, _) do
    with subdomain <- String.replace(conn.host, @base_host_regex, ""),
         endpoint <- Map.get(@subdomains, subdomain, @default_host) do
      endpoint.call(conn, endpoint.init())
    end
  end
end

Let’s go over this one step at a time:

@base_host_regex ~r|\.?mydomain.*$|

This is used to extract the subdomain part of the host URL of every request. So for admin.mydomain.com we want to get the string admin and for mydomain.com we will end up with an empty string (meaning, we’ll forward this to the default app. More on that later).

Notice that this doesn’t exactly match the .com part. This is a convenience change I made for local development. Matching on mydomain.* allows me to use admin.mydomain.lvh.me when working on my local machine, and still have this whole logic working without making development-specific changes.

If you don’t know what lvh.me is, this article might be helpful (TL;DR: It’s a development service that resolves its name to localhost).

With the above regex in mind, the next part should be easy to understand:

@subdomains %{
    "admin" => Admin.Web.Endpoint,
    "client" => Client.Web.Endpoint
}
@default_host Client.Web.Endpoint

For every subdomain, we want to match a particular Phoenix endpoint belonging to the app that we want to forward the request to. @default_host is what we’ll use if the subdomain is missing (the empty string scenario we talked above).

def call(conn, _) do
    with subdomain <- String.replace(host, @base_host_regex, ""),
         endpoint <- Map.get(@subdomains, subdomain, @default_host) do
      endpoint.call(conn, endpoint.init())
    end
end

When this endpoint—which is actually not much more than an Elixir Plug—is called, we just grab the subdomain from the request host, then find the matching endpoint from our mapping (defaulting to @default_host), and call endpoint.call/2 on it. This is essentially delegating the call down to the appropriate app.

Now client and admin both have to only worry about their corresponding requests and authentication. All logic related to the multiple subdomains & clients we may need is abstracted away in this app.

Want a new client in the same umbrella? Add it here! Want the same endpoint to respond to additional subdomains? Add it here!

Taking the routing even further

By adding a smart router to our umbrella application, we’re now able to serve requests to different subdomains to different apps in our umbrella application. I first implemented this pattern on a pet project of mine, but have since used and improved it on a few production projects as well.

We could take this much further. For example, if you’re migrating an existing service from Ruby to Elixir. You can have this proxy application route all requests made to the Ruby version of your service redirected back to the Ruby application, ensuring backward-compatibility. Or you may want the opposite scenario, where you’re creating a new API service and want to forward matching requests to a different client or even to a different web server altogether.

We can also take the routing complexity to another level. Routing was done here based solely on the subdomain of the request. But depending on your needs, you can create more complex routing rules using HTTP headers or query parameters. All of this can be done while keeping your actual web services completely oblivious to it.

We had a blast 💥thinking about all the possibilities of Umbrellas and Routing. We hope it set your mind on 🔥fire as well. If you want to get a regular dose of ⚗️Elixir Alchemy, subscribe to get the next episode of Alchemy delivered straight to your inbox.

This post is written by guest author Miguel Palhas. Miguel is a professional over-engineer @subvisual and organizes @rubyconfpt and @MirrorConf.

Pouring Protocols in Elixir

Miguel Palhas — Tue, 19 Feb 2019 13:45:20 +0000

In today's Elixir Alchemy, we will stir into the potion of protocols. Elixir has several mechanisms that allow us to write expressive and intuitive code.

Pattern matching, for instance, is a powerful way of dealing with multiple scenarios without having to go into complicated branching. It allows each of our functions to be clear and concise.

What Are Protocols?

In a way, Protocols are similar to pattern matching, but they allow us to write more meaningful and context-specific code based on the datatype we’re dealing with.

Let’s take the example of a content-delivery website. This website has multiple types of content: audio clips, videos, texts, and whatever else you can think of.

Each of these content types obviously has different attributes and metadata, so it makes sense for them to be represented by independent structs:

Translating this into Elixir, you’d have the following structures:

defmodule Content.Audio do
  defstruct [:title, :album, :artist, :duration, :bitrate, :file]
end

defmodule Content.Video do
  defstruct [:title, :cast, :release_date, :duration, :resolution, :file]
end

defmodule Content.Text do
  defstruct [:title, :author, :word_count, :chapter_count, :format, :file]
end

Each of these types has a few different fields, most of them unique to the type. We also have a common :file field which will point to the file keeping the actual data.

Now, let’s say you want to make your content as accessible as possible. You may, for instance, want to allow your hearing-impaired users to view the transcripts of both your audio and video. For that, you’ll use your awesome AudioTranscriber and VideoTranscriber modules which provide transcribe_audio/1 and transcribe_video/1 functions, respectively.

The implementation of those functions uses state-of-the-art machine learning and will be delegated to a future blog post. Let’s just assume they work and roll with it.

Both transcriber modules are split up into separate modules. Aside from having different function names for transcribing content, they might be completely different libraries. To allow us to use both in a transparent manner, we'll implement a protocol named Content.Transcribe that has a unified API that can handle both types of content.

Implementing the Protocol

Using protocols, we can easily define what the act of transcribing something means to each of our data types. This is done by first defining a transcribing protocol:

defprotocol Content.Transcribe do
  def transcribe(content)
end

and then implementing it separately for each of our types:

defimpl Content.Transcribe, for: Content.Video do
  def transcribe(video), do: VideoTranscriber.transcribe_video(video.file)
end

defimpl Content.Transcribe, for: Content.Audio do
  def transcribe(audio), do: AudioTranscriber.transcribe_audio(audio.file)
end

defimpl Content.Transcribe, for: Content.Text do
  def transcribe(text), do: File.read(text.file)
end

We have separately defined implementations of the same function for all 3 content types.

You may note that for text content, the implementation merely reads the corresponding file, as it's already in text format, while for the other two, we call the corresponding machine-learning-magic function on the file.

We’re then able to call transcribe/1 for all the data types we have an implementation for:

iex> %Content.Video{...} |> Content.Transcribe.transcribe()
{:ok, "We're no strangers to love\nYou know the rules and so do I..."}

iex> %Content.Audio{...} |> Content.Transcribe.transcribe()
{:ok, "Imagine there's no heaven\nIt's easy if you try..."}

iex> %Content.Text{...} |> Content.Transcribe.transcribe()
{:ok, "in a hole in the ground there lived a hobbit..."}

Fallback Implementations

Now, let’s say we add a new type of media to our platform: games (we’re kidding! We are a very ambitious hypothetical startup, and admittedly, success may be getting into our heads).

What happens when we try to transcribe the newly-added content?

iex> %Content.Game{...} |> Content.Transcribe.transcribe()
** (Protocol.UndefinedError) protocol Content.Transcribe is not implemented for %Content.Game{...}. This protocol is implemented for: Content.Audio, Content.Text, Content.Video

Whoops! We’ve hit an error. Which makes sense. We didn’t provide any transcription implementation for this type.

But it doesn’t really make sense to do so, does it? Games are supposed to be interactive experiences, and there simply may be no way to make them accessible to everyone.

So we could just provide an implementation that always fails:

defimpl Content.Transcribe, for: Content.Game do
  def transcribe(game), do: {:error, "not supported"}
end

But this doesn’t seem very scalable, does it? If we keep adding new content types, we'll end up having to duplicate this for every single type that we cannot transcribe.

Instead, we can simply add a fallback implementation for any type we don’t specify. This is done precisely by providing an implementation for the Any type, and then stating in our protocol that we want to fall back to it when necessary.

defimpl Content.Transcribe, for: Any do
  def transcribe(_), do: {:error, "not supported"}
end

defprotocol Content.Transcribe do
  @fallback_to_any true
  def transcribe(content)
end

The implementation for Any can usually be used by asking Elixir to automatically derive implementations from it (you can read more about this in the official Elixir Getting Started guide).

But by adding @fallback_to_any true to our protocol, we’re stating that whenever a specific implementation is not found, the Any implementation should be used. This allows us to fail gracefully for any unsupported data type:

iex> %Content.Game{...} |> Content.Transcribe.transcribe()
{:error, "not supported"}

iex> %{key: :value} |> Content.Transcribe.transcribe()
{:error, "not supported"}

Failed Gracefully

Can we close off any better than with a graceful fail? We'll leave you now that we've experimented with protocols and we gracefully haven't broken any alembic today.

If you love experimenting with code, make sure you don't miss an episode of Elixir Alchemy!

This post is written by guest author Miguel Palhas. Miguel is a professional over-engineer @subvisual and organizes @rubyconfpt and @MirrorConf.

Understanding Elixir’s GenStages: Querying the Blockchain

Miguel Palhas — Tue, 13 Nov 2018 14:22:40 +0000

In this edition of Elixir Alchemy, we'll dive into Elixir's GenStage module. Along the way, we'll explain backpressure and we'll write a Genstage to query the blockchain. Let's start by discussing how using a GenStage can solve buffering problems.

What Is a GenStage?

Imagine you’re consuming data from an external source. That source could be anything “streamable” - such as reading a file line-by-line, a table in a database, or even a sequence of requests to a 3rd party API.

In such scenarios, where you need to stream data into your system, and probably do some processing on each data point, it’s common to use a buffer to read in a few items, process the whole batch, and then fetch a new set into the buffer. I remember, from the time I was learning C/C++, that this would be a common, although arguably naive way to do things.

With that approach, you may run into one of two problems: the buffer can get too small, or the buffer van get too large.

Buffer Too Small
This happens if you read too few items at a time. Since you’re switching back and forth between reading and processing items, there will be a performance cost from the task switching. In the example of reading a file, your hardware or Operating System may be reading more data than what you’re actually requesting, resulting in sub-optimal performance, in addition to having to fetch the same part of the file later on.
Buffer Too Large
In this case, you request too much from your data source. You may end up either creating a bottleneck (e.g. having to wait for your hard drive to read everything you requested), or not being able to process all the data in an efficient manner. If you’ve ever heard of a buffer overflow (a common performance and security concern), this is it. You’re reading more than what your system can keep up with, resulting in all kinds of problems, from performance to actual failures.

The Solution: Backpressure

The term backpressure refers to the behavior of a system that builds up input, then halts the receiving of new data once the buffer is full, resuming it once again when the system is ready to handle it.

This is the core idea behind Elixir’s GenStage.

GenStage

GenStage is an abstraction built on top of GenServer to provide a simple way to create a Producer/Consumer architecture, while automatically managing the concept of backpressure.

In a GenStage, you create a pipeline of multiple Producers & Consumers. Producers generate data points, or read them from a source, and then pass them down to the pipeline. They can then be sent through one or more Consumers that will do whatever processing you need done.

The idea of backpressure is applied in the way items are created in a Producer. When the pipeline is ready to receive new items, the handle_demand/2 function of the Producer is called, requesting a specific amount of items.

The amount requested is decided internally (although you can specify a maximum value), and the function is called whenever there is room for them in the pipeline. If items take too long to process, Producers end up being idle for a while, thus relieving some pressure from the system.

Use Case

As an example of what a GenStage can be useful for, let’s consider reading chunks of data from an external data source. In this case, we’ll use the Ethereum blockchain, since it fits this concept nicely.

A blockchain is composed of a series of blocks, each one containing multiple transactions. If we want to process the entire blockchain (for example, to look up all transactions involving a given address, or to listen to it continuously when integrating with your application), a GenStage is a perfect fit.

In this context, each block can be considered as a single data item. Let’s see how this can be achieved.

Querying the Blockchain

We’re going to use Infura’s public HTTP API to interact with the Ethereum blockchain. Let’s start by building a wrapper to its interface. I’ll be using the Tesla library for this (this is just a personal preference, feel free to choose your own).

defmodule EthSync.Infura do
  use Tesla

  plug(Tesla.Middleware.BaseUrl, "https://ropsten.infura.io/")

  # encode/decode body as json
  # Infura doesn't set the "content-type" header to "application/json"
  # so we need to tell Tesla that we want text/plain requests to be decoded as well
  plug(Tesla.Middleware.JSON, decode_content_types: ["text/plain; charset=utf-8"])

  @doc "Get an entire block"
  def get_block(number) do
    case call(:eth_getBlockByNumber, [to_hex(number), true]) do
      {:ok, nil} ->
        {:error, :block_not_found}

      error ->
        error
    end
  end

  @doc "Sends a JSON-RPC call to the server"
  defp call(method, params \\ []) do
    case post("", %{jsonrpc: "2.0", id: "call_id", method: method, params: params}) do
      {:ok, %Tesla.Env{status: 200, body: %{"result" => result}}} ->
        {:ok, result}

      {:error, _} = error ->
        error
    end
  end

  @doc "Converts integer values to hex strings"
  def to_hex(decimal), do: "0x" <> Integer.to_string(decimal, 16)
end

We’ll only need a single endpoint for this: getting a block’s data, given its index on the chain. The block number must be given in hexadecimal format, so we also need a helper method to handle the conversion.

We can verify that this is working via iex:

iex(1)> EthSync.Infura.get_block(1)
{:ok,
 %{
   "number" => "0x1",
   "transactions" => [],
   # ...
 }
}

iex(2)> EthSync.Infura.get_block(1_000_000_000_000)
{:error, :block_not_found}

Building the Producer

Our Producer will be a process with the responsibility of fetching Ethereum blocks.

defmodule EthSync.Producer2 do
  alias EthSync.Infura
  use GenStage

  def init(_) do
    {:producer, 1}
  end

  def handle_demand(demand, next_block) when demand > 0 do
    IO.puts("Demanding #{demand}")

    blocks =
      next_block..(next_block - 1 + demand)
      |> Enum.map(fn n ->
        IO.puts("Fetching block #{n}")
        Infura.get_block(n)
      end)

    {:noreply, blocks, next_block + length(blocks)}
  end
end

Building the Consumer

The Consumer will receive lists of blocks and then process them. In the example, we’ll use :timer.sleep/1 to simulate processing time since we’re not doing any actual work. Keep in mind that the list of blocks received is not necessarily the same as what was sent in the Producer. Items can be buffered according to the GenStage’s internal rules. It may also happen that you have multiple Consumers and items get split between them.

defmodule EthSync.Consumer do
  alias EthSync.Infura
  use GenStage

  def init(_) do
    {:consumer, nil}
  end

  def handle_events(blocks, _from, state) do
    blocks
    |> Enum.each(fn
      {:ok, %{"number" => n}} ->
        IO.puts("Received block #{n}")
        :timer.sleep(1_000)
    end)

    {:noreply, [], state}
  end
end

Wiring It All Up

To start the pipeline, we need to start the processes for our Producer & Consumer, and then link them together, so that items produced by the former get sent out to the latter:

iex> {:ok, producer} = GenStage.start_link(Producer2, [])
{:ok, #PID<0.160.0>}

iex> {:ok, consumer} = GenStage.start_link(Consumer2, [])
{:ok, #PID<0.162.0>}

iex> GenStage.sync_subscribe(consumer, to: producer, max_demand: 3)
{:ok, #Reference<0.2486793675.579338241.116277>}
Demanding 3
Received block 0x1
Received block 0x2
Received block 0x3
Demanding 1
Received block 0x4
Received block 0x5
Demanding 1

Notice that even though we start the Producer at the beginning, it only started fetching blocks once we wired the Consumer to it. That’s because there was no demand until that point. Additionally, even though we specify max_demand: 3, that’s not necessarily the amount requested at all times. Since we only have a single Consumer, and it takes 1 second to process each block, the GenStage is smart enough not to overflow it with too many blocks. It adjusts the number of events as needed.

Consumed the coolness?

With the Producer, Consumer and having wired them together we've created a basic GenServer example. We love how GenStages provides an elegant way to create a producer/consumer architecture that automatically manages Backpressure.

This post is written by guest author Miguel Palhas. Miguel is a professional over-engineer @subvisual and organizes @rubyconfpt and @MirrorConf.

If you'd love to read more Elixir Alchemy, be sure to subscribe to the mailing list!

Lists vs Tuples in Elixir

Miguel Palhas — Tue, 21 Aug 2018 12:22:44 +0000

Welcome to another edition of Elixir Alchemy. Today, we'll explore the use of Lists and Tuples in Elixir. We'll take a look at how each of these data structures is used and see when it's appropriate to use either one over the other.

Lists and Tuples in Elixir

Lists and Tuples share some similarities that can make it difficult to discern when each data structure is suitable for use. They can both hold several values of any type. Visually, they even look similar, the only difference is in the braces that each uses:

iex> list = [1, 2, true, 3]
iex> tuple = {1, 2, true, 3}

However, they have fundamental differences that are important when deciding which one to use to store your data.

Lists are meant to hold variable-length data, and tuples are meant for single data points that have a fixed number of elements.

Conceptually speaking, you can think of this as storing a sequence of numbers (a list of numbers) versus storing several numbers corresponding to a certain value, for example, three numbers that correspond to a 3D point in space (i.e. a single value made up of 3 numbers).

Lists

Lists are represented internally as linked lists. They have the following characteristics:

They support adding/removing elements
Memory usage scales with the list size. The more elements the list has, the more memory it requires
Fetching elements may sometimes be slow

Picture the list above ([1, 2, true, 3]) laid out in your computer’s RAM. It is distributed arbitrarily, in no particular order, with each element holding a reference to the next one:

In order to reach the 3rd element, the system needs to know where the list begins (which is pretty much all it knows), and then traverse from there. This means that to get to the 100th element, all 99 elements before it have to be traversed.

To add a new element, a new memory slot is reserved anywhere where free memory is available, and then a reference to it is added to the previous last element.

Granted, there are optimizations that can be done by your compiler to optimize all of this. But the overall principle still holds.

Tuples

Tuples behave like static data structures, except that they don’t have explicit names for each element. Aside from that, all their other characteristics are pretty much the same:

They are meant to hold a fixed number of elements
As a result of the previous point, memory usage is also static, and allocated upfront.
The data structure does not support addition/removal of elements. Any change requires recreating an entirely new version of the structure.

In memory, this looks a lot different from lists:

You can see that memory usage is a lot more predictable (it's 100% predictable, actually).

All that's needed to access an element is:

A reference to where the tuple’s memory begins
The index of the element you’re accessing

Coming from other programming languages, you might recognize this as being very similar to accessing a static array. And you’d be right. Tuples are not much more than abstractions to statically-sized arrays. The difference comes in the semantic meaning given to both.

Arrays tend to hold data we can append or remove from. And while old C-style arrays would often have a fixed size, more recent languages easily provide ways for us to push, pop, shift and unshift elements. The way memory is handled to support this varies from language to language.

Tuples, on the other hand, are not meant to change. They provide one piece of information, not a list of pieces.

Examples of items that would fit in a tuple:

Name & Email Pairs

You can see this represented in most email clients when you see "Miguel Palhas <miguel@example.com>". In Elixir, this could be {"Miguel Palhas", "miguel@example.com"}. There’s actually an Elixir library that does this.

2D Points

Such as { 1, 2 }. This could also be represented as a map %{ x: 1, y: 2 }. Whichever you choose to use is mostly a matter of personal choice, and of the specific use-case.

Key-value Pairs

Any mapping between a key and a value can be a two-element tuple:

a = { :name, "Miguel Palhas" }
b = { :email, "miguel@example.com" }

Which leads us to an interesting point: Keyword lists. In Elixir, you may already have come across something like this:

list1 = [ name: "Miguel Palhas", email: "miguel@example.com" ]

As it turns out, this is just syntactic sugar for a list, where each value is a 2-element tuple. It is equivalent to:

    list2 = [
      { :name, "Miguel Palhas"},
      { :email, "miguel@example.com" }
    ]

You can verify this yourself by fetching a single element from this list:

    iex> Enum.at(list1, 0)
    {:name, "Miguel Palhas"}

Looking at how this would look like in memory, we’d see something like this:

Can you Add to Tuples?

Yes, you can. In Elixir, you have access to functions such as Tuple.append/1, Tuple.insert_at/1 or Tuple.delete_at/1.

Looking at what they do, you may be tempted to use them as you would with lists, adding and removing elements at-will.

However, these functions are rarely used, since what they actually do is create an entirely new tuple with the updated result somewhere else in memory. As you can probably see, this doesn't scale well. Constantly re-creating your tuple to add a few elements quickly gets out of hand.

With a list, adding/removing elements is a cheap operation that doesn’t need to mess with existing memory, which is why they’re more suited for dynamic collections.

Summary

We touched on the differences in data structure between Lists and Tuples. And got to the cool stuff about how this works out in memory. Tuples are cheaper to get data out of, Lists are cheaper to add to.

We’d love to know what you thought of this article, or if you have any questions. We’re always on the lookout for topics to investigate and explain, so if there’s anything about Elixir you’d like to read about, don’t hesitate to leave a comment.

This post is written by guest author Miguel Palhas. Miguel is a professional over-engineer @subvisual and organizes @rubyconfpt and @MirrorConf.

Parsing Numbers in Elixir

Miguel Palhas — Tue, 24 Jul 2018 12:03:02 +0000

Like any modern programming language, Elixir has built-in tools for doing basic tasks, such as parsing numbers from strings. Although they're built-in and ready to use, it's udeful to understand the underlying algorithms.

In this post we'll first explain how to convert strings to integers in Elixir. This will be quick and useful. After that we'll go straight down the rabbit hole and explain the underlying algorithms. This is the Alchemy we love. It may help you implement a parser for something similar, but it will definitely satisfy your curiosity about the mathematical ideas behind parsing numbers in Elixir.

The Quick (and Boring) Built-in

In Elixir, you can convert strings to floating point numbers using Float.parse/1:

iex> Float.parse("1.2")
{1.2, ""}

This returns a tuple, where the first element is the parsed number, and the second is whatever was left of your input string once a non-numeric character was found. This is useful if you’re unsure whether your input contains additional data:

iex> Float.parse("3 stroopwafels")
{3.0, " stroopwafels"}

iex> Float.parse("stroopwafels? 3, please")
:error # This fails because the number needs to be at the beginning of the string

If you’re sure your input is a well-formatted floating point number with no additional characters, you can use a more direct approach:

iex> String.parse_float("1.2")
1.2

iex> String.parse_float("3 stroopwafels")
** (ArgumentError) argument error
    :erlang.binary_to_float("3 stroopwafels")

To satisfy our technical curiosity, let's dive in and see how this works internally. We won't implement everything that's required for reliably parsing floats and integers, but we'll learn enough to understand the fundamentals.

Down the Rabbit Hole

One way of thinking about number parsing is by decomposing a number into multiple components:

1234 = 1000 + 200 + 30 + 4

Using this knowledge, we can use a divide-and-conquer strategy to parse the number, by parsing each of its digits individually. Elixir’s pattern matching and recursive capabilities also fit in nicely here.

Parsing a Single Digit

Let’s start with a single digit integer for demonstration purposes.

defmodule Parser do
  def ascii_to_digit(ascii) when ascii >= 48 and ascii < 58 do
    ascii - 48
  end
  def ascii_to_digit(_), do: raise ArgumentError
end

The ascii_to_digit/1 function expects the ASCII code of a single digit and returns the corresponding integer. This should only work for actual numeric characters, which are in the 48 to 57 range of the ASCII table. Any other value will cause an exception to be raised.

Knowing the fact that numerical digits are declared sequentially in the ASCII table, we can simply subtract 48 to get the actual numeric value. This function will be a useful helper in the next section.

Parsing an Entire Integer

Now let’s add a function to handle an entire string containing an integer:

defmodule Parser do
  def parse_int(str) do
    str
    |> String.reverse()
    |> do_parse_int(0, [])
  end

  def do_parse_int(<<char :: utf8>> <> rest, index, cache) do
    new_part = ascii_to_digit(char) * round(:math.pow(10, index))

    do_parse_int(
      rest,
      index + 1,
      [new_part | cache]
    )
  end
  def do_parse_int("", _, cache) do
    cache
    |> Enum.reduce(0, &Kernel.+/2)
  end

  # ...
end

Here, the do_parse_int/3 function traverses the string, using two auxiliary arguments: a counter that increments with every new digit, giving us our current index in the traversal, and an array where we keep intermediary values.

Also, notice that we’re first reversing the string. This is because Elixir’s pattern matching only allows us to match the beginning of a string, not the end. We want to start from the least significant digit, which is at the right end of a number. So we first reverse the string, then make the traversal from left-to-right.

For each, digit, we’re multiplying it with 10^index. This means that for the string "1234" we end up with the following array:

[1000, 200, 30, 4]

All that’s left is to sum up all the elements, which is done once we match an empty string.

Note: An optimized version of this could run the Enum.reduce call on the original characters of a string, summing the digits right away instead of keeping a temporary list. We didn't do this here so that we could split the responsibilities a bit better and leave the code more readable.

Parsing Floats

To implement a parse_float/1 function, all that’s left is to handle decimals. Fortunately, we can reuse our existing parse_int/1 function, along with a couple of fancy tricks to make everything work:

defmodule Parser do
  @float_regex ~r/^(?<int>\d+)(\.(?<dec>\d+))?$/

  def parse_float(str) do
    %{"int" => int_str, "dec" => decimal_str} = Regex.named_captures(@float_regex, str)

    decimal_length = String.length(decimal_str)

    parse_int(int_str) + parse_int(decimal_str) * :math.pow(10, -decimal_length)
  end
end

We define a @float_regex module variable, which holds a regular expression capable of capturing both the left and right side of a floating point number. The decimal separator, and it’s subsequent digits, are optional, so this regex will match "123" just as well as it matches "123.456".

Explaining the details of this regex is out of the scope of this article, but feel free to play around with it in your Elixir console.

When we run the regex, against our input, say "123.456", we end up with the following map:

%{
  "int" => "123"
  "dec" => "456"
}

We can now see where parse_int/1 comes in handy. It can be used on both parts to get 123 and 456, respectively. But how can we combine them to have the desired 123.456 as a result?

Again, math comes to our rescue. Multiplying the decimal part by 10^-3, where 3 is the length, gives us 0.456, which we can them add to the integer part to get the final result.

Our end result can parse integers and floats from strings.

defmodule Parser do
  def parse_int(str) do
    str
    |> String.reverse()
    |> do_parse_int(0, [])
  end

  def do_parse_int(<<char::utf8>> <> rest, index, cache) do
    new_part = ascii_to_digit(char) * round(:math.pow(10, index))

    do_parse_int(
      rest,
      index + 1,
      [new_part | cache]
    )
  end
  def do_parse_int("", _, cache) do
    cache
    |> Enum.reduce(0, &Kernel.+/2)
  end

  @float_regex ~r/^(?<int>\d+)(\.(?<dec>\d+))?$/

  def parse_float(str) do
    %{"int" => int_str, "dec" => decimal_str} = Regex.named_captures(@float_regex, str)

    decimal_length = String.length(decimal_str)

    parse_int(int_str) + parse_int(decimal_str) * :math.pow(10, -decimal_length)
  end

  def ascii_to_digit(ascii) when ascii >= 48 and ascii < 58 do
    ascii - 48
  end
  def ascii_to_digit(_), do: raise(ArgumentError)
end

Unhandled Cases

This was a somewhat summarized demonstration of how a low-level number parser could work in Elixir. However, it does not cover every possible scenario one might want. Some things that weren't covered are:

Support for negative numbers
Support for scientific notation (e.g. 1.23e7)
More graceful error handling. If you’re building a parser for your own use-case, then error handling should also depend on what the use case is, as well as its conditions. Hence, it was not covered here.
Handling more numerical systems. Did you notice we’re using :math.pow(10,x) in a few places? Making that 10 configurable should allow us to support binary, octal or hexadecimal strings.

We’d love to know what you thought of this article, or if you have any questions. We’re always on the lookout for topics to investigate and explain, so if there’s anything in Elixir you’d like to read about, don’t hesitate to leave a comment.