DEV Community: Jiahao

Migrating from Hugo Astro

Jiahao — Wed, 16 Apr 2025 00:54:39 +0000

Migrated my blog from Hugo → Astro to centralize my blog posts, writings for other companies, notes about CS and SWE, recommendations, and short takes on tech, internships, and life into the same place!

Blog 👉 https://blog.woojiahao.com

Subscribe to the RSS feed for updates!
👉 https://blog.woojiahao.com/rss.xml

Short about migrating 👉 https://blog.woojiahao.com/short/the-great-migration/

Pitfalls of Metaprogramming in Elixir

Jiahao — Tue, 23 Nov 2021 16:05:48 +0000

Welcome back to this final part of our four-part series on metaprogramming in Elixir.

Previously, we explored the various applications of macros.

In this part, we'll delve into common pitfalls that you might encounter when metaprogramming in Elixir.

Common Perils of Macros

According to the official documentation:

Macros should only be used as a last resort. Remember that explicit is better than implicit. Clear code is better than
concise code.

While it may be tempting to use metaprogramming for everything, it may not always be the best option.

The Applications of Macros part of this series outlines the majority of use cases of macros.

However, you should only use macros with great caution.

We'll be looking at these three common pitfalls to avoid when using macros:

Injecting unnecessary functions
Over-injecting behavior
Replacing regular functions

Let's kick off by looking at what happens if you inject unnecessary functions into modules with macros.

Note: These points are inspired by Metaprogramming Elixir.

1. Injecting Unnecessary Functions with Macros

While macros can be used to inject functions into a caller, there are times where this is unnecessary.

Let's look at an example:

defmodule CalculatorTransformer do
  defmacro __using__(_) do
    quote do
      def add(a, b), do: a + b
      def subtract(a, b), do: a - b
      def multiply(a, b), do: a * b
      def divide(a, b), do: a / b
    end
  end
end

defmodule Hospital do
  use CalculatorTransformer

  def calculate_cost(suite, procedure) do
    add(suite * 20, multiply(procedure, 5))
  end
end

We inject various calculator functions into Hospital to eliminate the need for a module identifier.

However, the use of add and multiply now seem like they appear from thin air.

The code loses its semantic meaning and becomes harder to understand for first-time readers.

To preserve the semantic meaning of the code, define CalculatorTransformer as a regular module. This module can be imported into Hospital to eliminate module identifiers:

defmodule CalculatorTransformer do
  def add(a, b), do: a + b
  def subtract(a, b), do: a - b
  def multiply(a, b), do: a * b
  def divide(a, b), do: a / b
end

defmodule Hospital do
  import CalculatorTransformer

  def calculate_cost(suite, procedure) do
    add(suite * 20, multiply(procedure, 5))
  end
end

As well as creating unnecessary functions, macros can also over-inject behavior into a module. Let's explore what this means.

2. Macros Over-injecting Behavior

As Elixir injects macros into the callsite, behavior can be over-injected.

Let's go back to the example of BaseWrapper:

What if we left the parsing logic for post? within the __using__ macro?

defmacro __using__(opts) do
  quote location: :keep, bind_quoted: [opts: opts] do
    # ...

    def post?(url, body) do
      case post(url, body) do
        {:ok, %HTTPoison.Response{status_code: code, body: body}} when code in 200..299 ->
          {:ok, body}

        {:ok, %HTTPoison.Response{body: body}} ->
          IO.inspect(body)
          error = body |> Map.get("error", body |> Map.get("errors", ""))
          {:error, error}

        {:error, %HTTPoison.Error{reason: reason}} ->
          IO.inspect("reason #{reason}")
          {:error, reason}
      end
    end
  end
end

There are two issues with this approach:

By testing post?, you test the inheritor rather than BaseWrapper.

As there are multiple inheritors of BaseWrapper and the entire behavior of post? is injected into the inheritor, we have to test every inheritor individually.

This ensures that any inheritor-specific behavior does not modify the behavior of post?.

Failure to do so can lead to lower test coverage.
Ambiguous error reporting.

Any run-time errors raised by post? will be logged under the inheritor, not BaseWrapper.

Therefore, leaving the entire behavior in post? can create confusion.

The original implementation of BaseWrapper moves the bulk of the parsing behavior into the wrapper instead. This implementation is much neater, semantically more meaningful, and readable.

This minimizes the two issues mentioned above, as:

When you test the core behavior of post?, only BaseWrapper.parse_post is tested — not every single inheritor.
Any errors from parsing will be logged under BaseWrapper.

Note: location: :keep works in a similar fashion.

While we've used wrappers in our example of over-injecting behavior, this can equally apply to regular macros.

A rule of thumb is to minimize the amount of behavior in a macro.
Once the necessary information/computations that require a macro have been accessed/performed, you should move the remaining behavior out of the macro.

The final pitfall we'll examine is the use of macros when regular functions suffice.

3. Macros Used in Place of Regular Functions

As powerful as they are, you don't always need macros. In some cases, you can replace a macro's behavior with a regular function.

Let's say that behavior that does not require compile-time information (or a macro to perform computation) is placed in a macro, for example:

defmodule Foo do
  defmacro double(x) do
    quote do
      doubled = unquote(x) * 2
      doubled
    end
  end
end

defmodule Baz do
  require Foo

  def execute do
    Foo.double(3)
  end
end

iex(1)> Baz.execute
6

Here, double could have easily been substituted for a regular function.

Its behavior does not require compile-time information nor a macro for computation. It will be injected into Baz and evaluated when execute is called, just like a regular function.

defmodule Foo do
  def double(x), do: x * 2
end

defmodule Baz do
  def execute, do: Foo.double(3)
end

iex(1)> Baz.execute
6

As you can see, defining double as a macro does not pose any benefits over a regular function.

Metaprogramming in Elixir: Further Reading

We have finally come to the end of this investigation into metaprogramming in Elixir!

Remember: with great power comes great responsibility. Misusing metaprogramming can come back to bite you, so tread lightly.

While this series has aimed to explain metaprogramming and its intricacies concisely, it is by no means the "bible" on this topic.

There are many wonderful resources you can use to learn more about metaprogramming in Elixir! Here are just a few:

Written guides

Books

Metaprogramming Elixir*

Talks

ElixirConf 2017 - Don't Write Macros But Do Learn How They Work by Jesse Anderson

(* - highly recommended)

Thanks for reading, and see you next time!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Jia Hao Woo is a developer from the little red dot — Singapore! He loves to tinker with various technologies and has been using Elixir and Go for about a year. Follow his programming journey on his blog and Twitter.

How to Use Macros in Elixir

Jiahao — Tue, 02 Nov 2021 12:22:39 +0000

Welcome back to this third part of our four-part series on metaprogramming in Elixir.

Previously, we established the underlying behavior of macros.

Now we will dive into the various applications of macros in Elixir using open-source Elixir libraries.

Let's go!

Using Macro Wrappers to Extend Elixir Module Behavior

Macros can extend module behavior by adding new functions through wrappers that behave similarly to inheritance in object-oriented programming (OOP).

To extend a module, we use use — this is syntactic sugar that performs the following for us:

defmodule Foo do
  defmacro __using__(opts) do
    quote do
      # Extend another module’s behavior here
    end
  end
end

defmodule Bar do
  use Foo
  # equivalent to calling these two expressions
  # require Foo
  # Foo.__using__([])
end

The __using__ callback macro extends the behavior of Bar. It is injected into the callsite and expanded.

We will be using the following terminology:

Wrapper — a module that extends other modules like Foo.
Inheritor — a module that uses wrappers like Bar, inheriting behavior.

Let’s design a wrapper.

Say we are building a website that relies on several APIs to work. Assuming we use an HTTP client like
HTTPoison and each API request has common configurations like
request/response data type, we can handle the API requests in three ways:

Set up each API request separately.
Compose API requests by chaining functions that configure each request.
Build a wrapper that automatically configures each API request per request.

While each of these approaches has its merits, we will focus on the last approach.

HTTPoison offers a wrapper for building basic API wrappers through
HTTPoison.Base. It handles aspects of an API request like setting up the endpoint and parsing the request/response body.

We will build another wrapper on top of this that will enable inheritors to achieve our business requirements:

Parse all request/response bodies to/from JSON.
Generate an API URL, given a base URL and an endpoint for the base URL.
Configure the request headers to accept JSON and set the content type to JSON.
Parse and handle response bodies.

The wrapper should produce inheritors like:

defmodule DogWrapper do
  use BaseWrapper, base_url: “https://api.dogs.com”

  def upload(name, breed) do
    case post?(“upload”, %{“name” => name, “breed” => breed}) do
      {:ok, response} -> {:ok, response[“link”]}
      {:error, error} -> {:error, error}
    end
  end
end

defmodule CatWrapper do
  use BaseWrapper, base_url: “https://api.cats.com”

  def upload(name, breed, talkative) do
    case post?(“upload”, %{“name” => name, “breed” => breed, “talkative” => talkative}) do
      {:ok, response} -> {:ok, response[“url”]}
      {:error, error} -> {:error, error}
    end
  end
end

defmodule AnimalLovers do
  def upload(:dog, name, breed) do
    case DogWrapper.upload(name, breed) do
      {:ok, link} -> redirect(link)
      {:error, error} -> redirect_error(error)
    end
  end

  def upload(:cat, name, breed, talkative) do
    case CatWrapper.upload(name, breed, talkative) do
      {:ok, link} -> redirect(link)
      {:error, error} -> redirect_error(error)
    end
  end
end

The inheritors do not contain any cumbersome API request logic. Instead, they focus on extracting data from the responses and executing the business logic.

Additionally, the wrapper integrates a base URL into the API requests, so inheritors provide only the endpoint.

Now we know what we want to accomplish, let’s build the wrapper.

First, we define our wrapper, calling it BaseWrapper. This module will define the __using__ callback.

opts is a keyword list that will hold the base_url of the inheritor, and it is bound to the quote.

location option for quote ensures that any errors will directly point to the line in the BaseWrapper.

Any behavior defined in quote will be injected into the inheritor.

defmodule BaseWrapper do
  defmacro __using__(opts) do
    quote location: :keep, bind_quoted: [opts: opts] do
      # behavior goes here
    end
  end
end

Then, we extend our inheritor to use the basic behavior from HTTPoison.Base.

# quote
use HTTPoison.Base

Now, we want to retrieve the base URL from opts. We do not need to unquote opts as it was bound.

# quote
base_url = Keyword.get(opts, :base_url)

With that, we can begin configuring our wrapper using HTTPoison.Base.

HTTPoison.Base defines several callbacks used to process requests and responses.

We will override these callbacks to implement the behavior we want.

# quote
# Modify request headers to include necessary information about the API
def process_request_headers(headers) do
  h = [
    {“Accept”, “application/json”},
    {“Content-Type”, “application/json”}
  ]

  headers ++ h
end

# Generate the URL for the endpoint using a base URL received from the inheritor
def process_url(endpoint), do: URI.merge(URI.parse(unquote(base_url)), endpoint) |> to_string()

# Automatically encode request body to JSON
def process_request_body(body), do: Jason.encode!(body)

# Automatically decode request body from JSON
def process_response_body(body), do: Jason.decode!(body)

Finally, we want to parse responses from POST requests based on the HTTP status code of the response.

# quote
def post?(endpoint, body) do
  # post is available as we have used HTTPoison.Base which will perform the necessary imports for us
  # post receives only the endpoint of the request as process_url prepends the base URL already
  response = post(endpoint, body)
  # We defer the parsing of the response to a function within the BaseWrapper module.
  # Reasons for doing so will be discussed later
  BaseWrapper.parse_post(response)
end

# BaseWrapper
def parse_post({:ok, %HTTPoison.Response{status_code: code, body: body}})
    when code in 200..299 do
  {:ok, body}
end

def parse_post({:ok, %HTTPoison.Response{body: body}}) do
  IO.inspect(body)
  error = body |> Map.get(“error”, body |> Map.get(“errors”, “”))
  {:error, error}
end

def parse_post({:error, %HTTPoison.Error{reason: reason}}) do
  IO.inspect(“reason #{reason}”)
  {:error, reason}
end

Putting all this together, we have a module that looks like this:

defmodule BaseWrapper do
  defmacro __using__(opts) do
    quote location: :keep, bind_quoted: [opts: opts] do
      use HTTPoison.Base

      base_url = Keyword.get(opts, :base_url)

      def process_request_headers(headers) do
        h = [
          {“Accept”, “application/json”},
          {“Content-Type”, “application/json”}
        ]

        headers ++ h
      end
      def process_url(endpoint), do: URI.merge(URI.parse(unquote(base_url)), endpoint) |> to_string()
      def process_request_body(body), do: Jason.encode!(body)
      def process_response_body(body), do: Jason.decode!(body)

      # Function injected at compile-time to parse and act on responses accordingly
      def post?(url, body) do
        response = post(url, body)
        BaseWrapper.parse_post(response)
      end
    end
  end

  def parse_post({:ok, %HTTPoison.Response{status_code: code, body: body}})
      when code in 200..299 do
    {:ok, body}
  end

  def parse_post({:ok, %HTTPoison.Response{body: body}}) do
    IO.inspect(body)
    error = body |> Map.get(“error”, body |> Map.get(“errors”, “”))
    {:error, error}
  end

  def parse_post({:error, %HTTPoison.Error{reason: reason}}) do
    IO.inspect(“reason #{reason}”)
    {:error, reason}
  end
end

We've used macro wrappers to extend module behavior. Now, let's focus our attention on performing batch imports using wrappers.

Batch Imports with Macro Wrappers

We can use wrappers to perform batch imports in inheritors.

We can define a group of imports/aliases/requires that will be available during compile-time without performing the imports manually in the inheritor.

This is useful when several inheritors require the same set of imports and is best illustrated in Hound — a browser automation testing library:

# lib/hound/helpers.ex
defmacro __using__([]) do
  quote do
    import Hound
    import Hound.Helpers.Cookie
    import Hound.Helpers.Dialog
    import Hound.Helpers.Element
    import Hound.Helpers.Navigation
    import Hound.Helpers.Orientation
    import Hound.Helpers.Page
    import Hound.Helpers.Screenshot
    import Hound.Helpers.SavePage
    import Hound.Helpers.ScriptExecution
    import Hound.Helpers.Session
    import Hound.Helpers.Window
    import Hound.Helpers.Log
    import Hound.Helpers.Mouse
    import Hound.Matchers
    import unquote(__MODULE__)
  end
end

Hound overrides __using__ to import a host of helper modules available to the inheritor.

The inheritors are always going to be test suites written by other developers. It is convenient and easy to maintain imports through use Hound.Helpers.

Next up, wrappers can also override and dynamically generate functions. Let's see how that works.

Macro Wrappers Define Functions to Override

Like a child class overrides a method in its parent class in OOP, wrappers can define some base behavior for functions and allow inheritors to override this behavior.

This is achieved using defoverridable.

Phoenix uses this to great effect by allowing inheritors of Phoenix.Controller.Pipeline to override functions like init and call to include context-specific behavior.

However, if a custom behavior isn't necessary — i.e. the function is not overwritten — the base behavior will be used instead:

# lib/phoenix/controller/pipeline.ex
defmacro __using__(opts) do
  quote bind_quoted: [opts: opts] do
    # ...

    @doc false
    def init(opts), do: opts

    @doc false
    def call(conn, action) when is_atom(action) do
      conn
      |> merge_private(
        phoenix_controller: __MODULE__,
        phoenix_action: action
      )
      |> phoenix_controller_pipeline(action)
    end

    @doc false
    def action(%Plug.Conn{private: %{phoenix_action: action}} = conn, _options) do
      apply(__MODULE__, action, [conn, conn.params])
    end

    defoverridable init: 1, call: 2, action: 2
  end
end

Macro Wrappers Dynamically Generate Functions

You can also use unquote fragments to dynamically define an arbitrary number of functions. These functions are injected into a module during compile-time. This is especially useful when working with dynamic data or data that comes from an API.

This article will focus on the former, as Metaprogramming Elixir includes a wonderful example of dynamically defining functions for API responses under the code/hub/.

Let's say we are building a script that reads a list of groceries and generates functions for each category. These functions will operate on each unique category and the items within it.

For simplicity, each function will be the category's name and print the contents of the category.

The grocery list will use the following format — <category>:<comma separated list>:

// groceries.txt
dinner:carrots,potatoes,curry,chicken cubes
supplies:paper,pencils
school:folder,binder

We can read this file in an Elixir script:

# groceries.exs
defmodule Groceries do
  @filename "groceries.txt"
  for line <- File.stream!(Path.join([__DIR__, @filename]), [], :line) do
    # line represents each line of the groceries list
  end
end

We've extracted the lines of groceries, now we can parse each line:

# for
[category, rest] = line |> String.split(":") |> Enum.map(&String.trim(&1))
groceries = rest |> String.split(",") |> Enum.map(&String.trim(&1))

Then we can define our dynamic functions.

We use unquote as an argument for def. def is also a macro, so expands the unquote along with anything in its body during compilation. This is known as an unquote fragment.

# for
def unquote(String.to_atom(category))() do
  readable_list = Enum.join(unquote(groceries), " and ")
  IO.puts("Groceries in #{unquote(category)} include #{readable_list}")
end

Putting it all together, we get the following script:

# groceries.exs
defmodule Groceries do
  @filename "groceries.txt"
  for line <- File.stream!(Path.join([__DIR__, @filename]), [], :line) do
    [category, rest] = line |> String.split(":") |> Enum.map(&String.trim(&1))
    groceries = rest |> String.split(",") |> Enum.map(&String.trim(&1))

    def unquote(String.to_atom(category))() do
      readable_list = Enum.join(unquote(groceries), " and ")
      IO.puts("Groceries in #{unquote(category)} include #{readable_list}")
    end
  end
end

iex(1)> Groceries.dinner
Groceries in dinner include carrots and potatoes and curry and chicken cubes
:ok
iex(2)> Groceries.school
Groceries in school include folder and binder
:ok

Now let's turn our attention to implementing domain-specific languages (DSLs) using macros.

Implement Domain-Specific Languages using Macros

Domain-Specific Languages (DSLs) are "computer language(s) specialized to a particular application domain."

DSLs can be built and used in Elixir to solve business requirements.

As the official DSL tutorial points out:

You don’t need macros in order to have a DSL: every data structure and every function you define in your module is part of your Domain-specific language.

However, we will cover a simple implementation of a DSL using macros to demonstrate how that would look.

The simplest example of a DSL is from ExUnit, a built-in testing framework (taken from the ExUnit documentation):

defmodule AssertionTest do
  use ExUnit.Case, async: true

  test "the truth" do
    assert true
  end
end

test is a macro that registers a new test case.

These test cases are collated using accumulating module attributes. Then, when the test suite executes, all of the declared test cases are executed.

You can find a simplified implementation and explanation of this DSL in the official tutorial.

Absinthe — an implementation of the GraphQL specification — has a DSL for defining schemas.

# lib/absinthe/schema/notation.ex
defmacro object(identifier, attrs, do: block) do
  {attrs, block} =
    case Keyword.pop(attrs, :meta) do
      {nil, attrs} ->
        {attrs, block}

      {meta, attrs} ->
        meta_ast =
          quote do
            meta unquote(meta)
          end

        block = [meta_ast, block]
        {attrs, block}
    end

  __CALLER__
  |> recordable!(:object, @placement[:object])
  |> record!(
    Schema.ObjectTypeDefinition,
    identifier,
    attrs |> Keyword.update(:description, nil, &wrap_in_unquote/1),
    block
  )
end

object is a macro that generates the GraphQL schema.

We define it as a macro because we're accessing compile-time information through __CALLER__, and we're attempting to work with the module's metadata. Defining it as a macro loads the schemas during compile-time.

Even the routing functions in Phoenix are a DSL.

Alchemist Camp's 'Creating a DSL for our router' video series explains how to implement a routing system similar to Phoenix's.

It's time to move on to Abstract Syntax Trees (ASTs): how they can be traversed and when to use prewalk vs. postwalk macros.

Traversing Abstract Syntax Trees (ASTs)

We introduced ASTs in part one of this series. You can traverse an existing AST to extract information about — or modify — the structure of the AST to:

Improve performance
Simplify the AST
Perform computations based on the structure of the AST

As the modification of ASTs is a niche process and context-dependent, we will focus on how traversal is performed rather than the specific application of traversal.

Going Back to the Roots of ASTs

Before we can understand the traversal of ASTs, we have to get to grips with the core data structure of ASTs.

Source code is parsed into trees — data structures that house hierarchical tree data. They begin with a root node, followed by a set of sub-trees — children nodes.

Each node includes a reference to its children. Each childless node is known as a leaf.

The figure below illustrates the basic anatomy of a tree:

Order of Traversal in ASTs

Now that we know the underlying data structure of an AST, we can tackle the problem of traversing an AST/tree.

Elixir uses depth-first traversal in either pre-order or post-order.

Depth-first traversal follows a node down to its children recursively until reaching a leaf. The sibling of the node is moved to next, and the recursion repeats.

(Source: Wikimedia Commons)

If we use our example, the nodes are visited in the following order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7.

Pre-order and post-order traversal refer to how depth-first traversal occurs:

Pre-order traversal starts from the root node and traverses depth-first through the AST until the right-most leaf is reached.

In our tree, the nodes would be visited in the following order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7.

Post-order traversal starts from the left-most leaf of each sub-tree and traverses in depth-first fashion until we reach the root node.

In our tree, the nodes would be visited in the following order: 2 -> 4 -> 5 -> 7 -> 6 -> 3 -> 1. The traversal happens upwards towards the root node.

Regardless of the traversal order, the traversal occurs recursively until the "end condition" — i.e., right-most leaf for pre-order and root node for post-order.

In Elixir, functions represent each type of traversal:

Macro.prewalk — performs depth-first, pre-order traversal
Macro.postwalk — performs depth-first, post-order traversal
Macro.traverse — performs depth-first traversal and both pre and post-order traversal on the AST

By traversing the tree in a given order, we can extract information about the AST and use it to modify the AST.

We define functions that will be executed recursively on every node during traversal. These functions are similar to map.

The input is the currently visited node, while the output is a modified/untouched node.

Prewalk vs. Postwalk Macros

You might ask: "What is the benefit of using pre-order traversal over post-order traversal, and vice versa?"

In most scenarios, there is no difference between the two as both will traverse the AST.

However, you'll have a preference for postwalk or prewalk if the operation is order-sensitive — i.e., if you require the traversal to start at the root node or the left-most leaf first.

This may happen when the operation aims to match the first node (in order) against a given condition and only perform the operation on that node. We would rather have the traversal find the node quickly to operate as soon as possible.

In this case, prewalk is preferred over postwalk if the node appears at the beginning of the AST.

Macro.prewalk(ast, fn
  {:match, [], args} -> foo(args)
  otherwise -> otherwise
)

Another consideration is unintended infinite recursion. Take this example:

Macro.prewalk({:foo, [], [:bar]}, fn
  {:foo, [], _} -> {:foo, [], [{:foo, [], [:bar]}]}
  otherwise -> otherwise
end)

In this prewalk, the function matches any node that calls the foo function and replaces it with a recursive call to foo, with foo as the argument.

This will produce an infinite AST that, when converted to Elixir code, will look something like foo(foo(foo(...))).

prewalk is recursively executed on an AST — usually until it reaches the right-most leaf, indicating the AST's end. However, as the AST expands infinitely in the above example, there will never be a right-most leaf, hence infinite recursion.

Using postwalk instead avoids this issue as we simply replace the node once and move on upwards to the root node where the recursion will stop.

Macro.postwalk({:foo, [], [:bar]}, fn
  {:foo, [], _} -> {:foo, [], [{:foo, [], [:bar]}]}
  otherwise -> otherwise
end)

{:foo, [], [{:foo, [], [:bar]}]}

So, while the choice of prewalk and postwalk might not matter when an operation is not order-sensitive, postwalk is preferred over prewalk to avoid infinite recursion.

traverse combines prewalk and postwalk, performing both together. This is useful when we want to traverse the AST in both orders.

Ecto uses prewalk to count the number of interpolations within a given expression.

In this case, there isn't a specific reason to choose prewalk over postwalk.

# lib/ecto/query/builder.ex
def bump_interpolations(expr, params) do
  len = length(params)

  Macro.prewalk(expr, fn
    # The following expression matches a pinned variable which is what Ecto relies on for
    # interpolation
    {:^, meta, [counter]} when is_integer(counter) -> {:^, meta, [len + counter]}
    other -> other
  end)
end

Ecto also uses postwalk to expand dynamic expressions.

# lib/ecto/query/builder/dynamic.ex
defp expand(query, %{fun: fun}, {binding, params, subqueries, count}) do
  {dynamic_expr, dynamic_params, dynamic_subqueries} = fun.(query)

  Macro.postwalk(dynamic_expr, {binding, params, subqueries, count}, fn
    {:^, meta, [ix]}, {binding, params, subqueries, count} ->
      case Enum.fetch!(dynamic_params, ix) do
        {%Ecto.Query.DynamicExpr{binding: new_binding} = dynamic, _} ->
          binding = if length(new_binding) > length(binding), do: new_binding, else: binding
          expand(query, dynamic, {binding, params, subqueries, count})

        param ->
          {{:^, meta, [count]}, {binding, [param | params], subqueries, count + 1}}
      end

    {:subquery, i}, {binding, params, subqueries, count} ->
      subquery = Enum.fetch!(dynamic_subqueries, i)
      ix = length(subqueries)
      {{:subquery, ix}, {binding, [{:subquery, ix} | params], [subquery | subqueries], count + 1}}

    expr, acc ->
      {expr, acc}
  end)
end

Another way you can change module behavior using macros is through compile-time hooks — let's take a quick look at those.

Compile-time Hooks with Macros

Compile-time hooks allow the compilation behavior of a module to be modified. Callbacks accompany these hooks.

The two notable hooks to discuss are @before_compile and @after_compile. They are useful when we want to perform computation right before — or right after — module compilation.

For now, let's look at a basic set of examples for each hook.

With @before_compile, as the callback (defmacro __before_compile__(env)) is called right before compilation, the callback must be declared in a separate module from where the hook references it.

If the callback is declared in the same module, the macro will not compile in time.

defmodule Foo do
  defmacro __before_compile__(env) do
    IO.inspect(env)
    nil
  end
end

defmodule Bar do
  @before_compile Foo
end

An exception to this behavior is when Bar is an inheritor of Foo:

defmodule Foo do
  defmacro __using__(_) do
    quote do
      @before_compile unquote(__MODULE__)
    end
  end

  defmacro __before_compile__(env) do
    IO.inspect(env)
    nil
  end
end

defmodule Bar do
  use Foo
end

In this example, as @before_compile is injected into Bar, its callback is defined in Foo (a different module). Since use calls require, it ensures that Foo is compiled before Bar. This means Foo.__before_compile__/1 is always available to Bar.

With @after_compile, there isn't a need to declare the callback (defmacro __after_compile__(env, bytecode)) in another module. This is because the module housing the callback is already compiled, so the callback is available.

defmodule Foo do
  @after_compile __MODULE__

  def __after_compile__(env, _bytecode) do
    IO.inspect(env)
    nil
  end
end

Another hook that is worth mentioning briefly is @on_definition, which invokes its callback whenever a function/macro is defined in the current module.

ExUnit uses @before_compile in a test suite to inject a final function — __ex_unit__ — to execute the test suites after they have been collated.

This function must be injected right before compilation when all the test suites are collated.

You may also wish to store a list of data across macro invocation, such as when ExUnit collates test cases and invokes them all at once.

This can be achieved using module attributes. Let's see that in action.

Module Attributes as Temporary Storage

When we set a module attribute to accumulate, any invocation of the module attribute will add the given value to the list, rather than overriding it:

defmodule Foo do
  Module.register_attribute(__MODULE__, :names, accumulate: true)
  @names "John"
  @names "Peter"

  def print, do: IO.inspect(@names)
end

iex(1)> Foo.print
["Peter", "John"]

We may also want to accumulate values through a function call.

This happens when the parent module first compiles, and then a secondary module updates the module attribute via a macro, like so:

defmodule Foo do
  defmacro add(name) do
    quote do
      @names unquote(name)
      :ok
    end
  end
end

defmodule Bar do
  require Foo
  import Foo

  Module.register_attribute(__MODULE__, :names, accumulate: true)

  add "john"
  add "henry"

  IO.inspect(@names) # This prints ["henry", "john"] right after the module is done compiling
end

Foo must compile first. The attribute has to be registered under Bar before it can be used in a given module.

The exception to this rule is use:

defmodule Foo do
  defmacro __using__(_) do
    quote do
      import Foo
      Module.register_attribute(__MODULE__, :names, accumulate: true)
    end
  end

  defmacro add(name) do
    quote do
      @names unquote(name)
      :ok
    end
  end
end

defmodule Bar do
  use Foo

  add "john"
  add "henry"

  IO.inspect(@names) # This prints ["henry", "john"] right after the module is done compiling
end

This is how ExUnit accumulates test cases and uses @before_compile to inject a "run all test cases in test suite" function right before compilation, similar to something like this:

defmodule Calculator do
  def add(a, b), do: a + b
  def subtract(a, b), do: a - b
end

defmodule TestCase do
  defmacro __using__(_) do
    quote do
      import TestCase
      Module.register_attribute(__MODULE__, :tests, accumulate: true)
      @before_compile unquote(__MODULE__)
    end
  end

  defmacro __before_compile__(_env) do
    # Inject a run function into the test case after all tests have been accumulated
    quote do
      def run do
        Enum.each @tests, fn test_name ->
          result = apply(__MODULE__, test_name, [])
          state = if result, do: "pass", else: "fail"
          IO.puts "#{test_name} => #{state}"
        end
      end
    end
  end

  defmacro test(description, do: body) do
    test_name = String.to_atom(description)
    quote do
      @tests unquote(test_name)
      def unquote(test_name)(), do: unquote(body)
    end
  end
end

defmodule CalculatorTest do
  use TestCase
  import Calculator

  test "add 1, 2 should return 3" do
    add(1, 2) == 3
  end

  test "subtract 5, 2 should not return 4" do
    subtract(5, 2) == 4
  end
end

CalculatorTest.run
"add 1, 2 should return 3" => pass
"subtract 5, 2 should not return 4" => fail

Finally, we'll touch on deferring computation using macros.

Deferring Computation with Macros

Macros inject behavior into the callsite as-is and can be used to avoid immediate evaluation of an expression.

For instance:

defmodule Foo do
  def if?(condition, do: block, else: else_block) do
    case condition do
      true -> block
      false -> else_block
    end
  end
end

Foo.if? true do
  IO.puts("Truth")
else
  IO.puts("False")
end

"Truth"
"False"

Here, we tried implementing the new if using a regular function. However, the result is not what we expect — rather than only evaluating and printing "Truth", both "Truth" and "False" are printed.

This is because of the nature of a regular function: each block is evaluated immediately, so the case will not work.

If we use a macro instead, the macro has to be expanded first, generating an AST of the case first and evaluating the case accordingly. During this time, the condition is evaluated, before matching against the case, and, finally, the appropriate block is evaluated.

Note: This example is borrowed from Metaprogramming Elixir.

Macros in Elixir: A Powerful Tool, If Used Wisely

You can apply macros to many scenarios to extend an application's behavior in ways that normal code cannot.

However, macros are a double-edged sword — when misused, they can create confusion and muddy code's readability and semantic meaning.

In the final part of this metaprogramming series, we will delve into the common pitfalls you might encounter when working with macros in Elixir.

Thanks for reading, and see you next time!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Under the Hood of Macros in Elixir

Jiahao — Tue, 12 Oct 2021 11:52:21 +0000

Welcome back to part two of this series on metaprogramming in Elixir. In part one, we introduced metaprogramming and gave a brief overview of macros.

In this part, we will explore the inner workings and behaviors of macros in more depth.

As discussed in the previous post, macros are compile-time constructs in Elixir. So, before diving into how macros work, it is important to understand where macros lie within Elixir's compilation process.

Stages of Elixir's Compilation Process

We can boil down the Elixir compilation process to the following basic stages:

(Note: the actual compilation process of an Elixir program is more intricate than above.)

The compilation process can be broken down into the following phases:

Parsing — The Elixir source code (program) is parsed into an AST, which we will call the initial AST.
Expansion — The initial AST is scanned and macro calls are identified. Macros are executed. Their output (AST) is injected and expanded into the callsite. Expansion occurs recursively, and a final AST is generated.
Bytecode generation phase — After the final AST is generated, the compiler performs an additional set of operations that eventually generate and execute BEAM VM bytecode.

As you can see, macros sit at the expansion phase, right before code is converted into bytecode. Therefore, a good knowledge of the expansion phase helps us understand how macros work.

Expansion Phase

Let's start by first examining the expansion phase on a general level.

The compiler will expand macros (as per Macro.expand) to become part of the program's pre-generated AST. Macro expansion occurs recursively, meaning that Elixir will continue expanding a macro until it reaches its most fundamental AST form.

As macros expand right before bytecode is generated, they can modify a program's behavior during compile-time.

If we dig a little deeper, we will find that the compiler first injects the output AST of a macro at its callsite. Then the AST is expanded recursively.

We can observe this behavior as follows:

defmodule Foo do
  defmacro foo do
    quote do
      IO.inspect("hello")
    end
  end
end

iex(1)> require Foo
iex(2)> ast = quote do: Foo.foo
{{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true],
 []}
iex(3)> ast |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423391, alias: false], [:IO]},
    :inspect
  ]}, [], ["hello"]}

ast represents the initial AST generated by the compiler before expansion. It holds a reference to the macro Foo.foo but it is not expanded as the macro has not been evaluated yet.

When we call Macro.expand on the given AST, the compiler begins by injecting the behavior of the macro into the callsite. We can expand the AST one step at a time using Macro.expand_once.

Contexts in Macros

Now that we understand the basics of the expansion phase, we can investigate the parts of a macro.

Macros contain two contexts — a macro context and a caller context:

defmacro foo do
  # This is the macro's context, this is executed when the macro is called

  # This is the return value of the macro (AST)
  quote do
    # This is the caller's context, this is executed when the callsite is called
  end
end

As you can see, the macro's context is any expression declared before the quote.

The caller's context is the behavior declared in the quote. The quote generated AST is the macro's output and is injected into and expanded at the callsite.

The behavior defined under the caller's context 'belongs' to the caller, not the module where the macro is defined. The following example, taken from Metaprogramming Elixir, illustrates this:

defmodule Mod do
  defmacro definfo do
    IO.puts "definfo :: Macro's context #{__MODULE__}"

    quote do
      IO.puts "definfo :: Caller's context #{__MODULE__}"

      def friendly_info do
        IO.puts "friend_info :: Module name #{__MODULE__}"
      end
    end
  end
end

defmodule MyModule do
  require Mod
  Mod.definfo
end

iex(1)> c "context.exs"
definfo :: Macro's context Elixir.Mod
definfo :: Caller's context Elixir.MyModule
[Mod, MyModule]
iex(2)> MyModule.friendly_info
friend_info :: Module name Elixir.MyModule
:ok

As you can see, the module that executes the macro's context is Mod. But the module that executes the caller's context is MyModule — the callsite where the macro is injected and expanded.

Similarly, when we declare friendly_info, we inject this function into the callsite of the macro, which is MyModule. So the function now 'belongs' to MyModule.

But why does there need to be two different contexts? What exactly makes the macro context and caller context different from one another?

Order of Evaluation in Macro and Caller Contexts

The key difference between the macro context and the caller context is that behavior is evaluated at different times.

Let's look at an example:

defmodule Foo do
  defmacro foo do
    IO.puts("macro")
    quote do
      IO.puts("caller")
    end
  end
end

iex(1)> require Foo
iex(2)> ast = quote do: Foo.foo
{{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true],
 []}

iex(3)> ast |> Macro.expand(__ENV__)
macro
{{:., [],
  [{:__aliases__, [counter: -576460752303423293, alias: false], [:IO]}, :puts]},
 [], ["caller"]}

When we expand the AST of a macro call, the macro context is evaluated and the caller context is injected and expanded into the callsite.

However, we can go a level deeper.

Let's look again at the first component of the expansion phase: Macros are executed. Their output (AST) is injected and expanded into the callsite.

For a compiler to know what AST needs to be injected into the callsite, it has to retrieve the output of the macro during compilation (when the expansion phase occurs). The macro call is parsed as an AST during the parsing phase. The compiler identifies and executes these macro call ASTs prior to the expansion phase.

If we think of macros as regular functions, the macro context is the function body and the caller context is the result of the function. During compilation, a macro is executed and evaluates the macro context. Then the quote is evaluated, returning the results of the function. The caller context is injected and expanded into the callsite of the macro.

The macro context is evaluated during compile-time and treated as a regular function body. It executes within and is 'owned' by its containing module.

The caller context is injected into the callsite, so it is 'owned' by the caller. It is evaluated whenever the callsite is evaluated.

The example above showcases what happens when a macro is invoked at the module level. But what if we attempt to invoke the macro in a function? When do the macro and caller contexts evaluate?

Well, we can use another example here:

defmodule Foo do
  defmacro foo do
    IO.puts("macro")

    quote do
      IO.puts("caller")
    end
  end
end

defmodule Bar do
  require Foo

  def execute do
    IO.puts("execute")
    Foo.foo
  end
end

macro
iex(1)> Bar.execute
execute
caller

The caller context evaluates when the function is called (as we established earlier).

However, something interesting happens with our macro context — it evaluates when the module compiles. This evaluation only happens once when Bar compiles, as evidenced by the lack of "macro" in our output when we call Bar.execute. Why is this the case?

Well, we only need to evaluate the macro once to retrieve its output (caller context) and inject it into the callsite (which is a function in this case). The caller context behavior evaluates every time the function is called.

This difference in the order and time of evaluation helps guide us on when to use the macro and the caller contexts.

We use the macro context when we want the behavior to be evaluated during compile-time. This is regardless of when the caller context is evaluated or where the macro is called in the code.

We use the caller context when we want to invoke behavior injected into the callsite at evaluation.

Now that we have a better grasp of the Elixir compilation process, macros, and the order of evaluation, we can revisit unquote.

Revisiting `unquote`

In part one of this series, we established that unquote evaluates a given expression and injects the result (as an
AST) into the AST built from quote. This is only a piece of the puzzle.

Let's dig deeper to understand the behavior of unquote during compilation and the necessity of using it.

While the rest of the quote body is evaluated at the same time as the callsite, unquote is evaluated (immediately) during compile-time — when the macro is evaluated. unquote aims to evaluate and inject the result of a given expression. This expression might contain information that is only available during the macro evaluation, including variables that are initialized during this process. unquote must be evaluated during compile-time along with the macro, so that the AST of the result injects into the quote that we build.

But why do we need to unquote the expression to inject it into the AST? To answer this, let's compare the expanded AST of a macro using unquote against one that does not:

defmodule Foo do
  defmacro foo(x) do
    quote do
      IO.inspect(x)
    end
  end
end

defmodule Bar do
  defmacro bar(y) do
    quote do
      IO.inspect(unquote(y))
    end
  end
end

iex(1)> require Foo
iex(2)> require Bar
iex(3)> ast_foo = quote do: Foo.foo(1 + 2 * 3)
iex(4)> ast_bar = quote do: Bar.bar(1 + 2 * 3)
iex(5)> ast_foo |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423448, alias: false], [:IO]},
    :inspect
  ]}, [], [{:x, [counter: -576460752303423448], Foo}]}

iex(6)> ast_bar |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423384, alias: false], [:IO]},
    :inspect
  ]}, [],
 [
   {:+, [context: Elixir, import: Kernel],
    [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]}
 ]}

Observe that the expanded Foo.foo AST is vastly different from the Bar.bar AST even though they are both given the same variable. This is because Elixir is quite literal with variable references. If a variable is referenced without unquote, an AST of that variable reference injects into the AST.

Using unquote ensures that the underlying AST of the variable's value injects into the quote body.

Now you may ask: What is the difference in variable scoping between the evaluation of macros and the execution of the callsite? Why does it matter?

The scoping of variables in macros can be a confusing subject, so let's demystify it.

Variable Scoping in Macros

Now that we understand how macros are evaluated and expanded, we can look at the scoping of variables in macros, and when to use the options unquote and bind_quoted in quote.

Due to function clause scoping, the arguments of a variable are initialized and 'in scope' during the macro evaluation of a function.

Similarly, variables declared and assigned within a function body remain in scope until the function ceases. The same behavior applies to macros.

When the macro context is evaluated, its arguments and any initialized variables are 'in scope.' This is why unquote can evaluate variable references declared as arguments of the macro or any variables initialized in the macro context.

Any evaluation of variable initialization in the caller context will initialize these variables within the callsite during execution.

To understand this difference better, let's look at a few examples:

defmacro foo do
  quote do
    x = 1 + 1
    def bar, do: IO.inspect(unquote(x))
  end
end

In this first example, unquote will not work. The variable x has not yet been initialized, but should have been initialized during the execution of the callsite. The immediate evaluation of unquote runs too early, so we cannot reference our variable x when we need to. When unquote evaluates during compile-time, it attempts to evaluate the variable reference expression of x and finds that it is not in scope.

How can we fix this? By disabling unquoting. This means disabling the immediate evaluation of unquote. We only want unquote to evaluate when our caller context evaluates. This ensures that unquote can properly reference a variable in scope (x) as variable initialization would have occurred
during the evaluation of the callsite.

defmacro foo do
  quote unquote: false do
    x = 1 + 1
    def bar, do: IO.inspect(unquote(x))
  end
end

This example highlights the impact of scoping in macros. If we attempt to access a variable that is available during the evaluation of the macro context, unquote as-is is perfect for us.

However, suppose we try to access a variable that is only available during the evaluation of the callsite. In that case, we must disable the immediate unquoting behavior to initialize variables in scope before unquote attempts to reference them.

Let's apply this understanding to two other examples.

defmacro foo(opts) do
  quote bind_quoted: [opts: opts] do
    x = Keyword.get(opts, :x)
    def bar, do: IO.inspect(unquote(x))
  end
end

In this example, we have initialized the variable x from a keyword list. As the keyword list is initialized during compile-time (along with the evaluation of the macro context), we first have to bind it to the caller context to:

Generate an initialization of the variable during the evaluation of the callsite, and
Disable unquoting behavior.

We have to bind opts to the caller context, as the variable is no longer in scope during the evaluation of the callsite.

Finally, we have:

defmacro foo(x) do
  quote do
    def bar, do: IO.inspect(unquote(x))
  end
end

In this last example, x remains a variable in scope during the evaluation of the macro context — i.e. when the macro is called. The immediate evaluation of unquote works in our favor. It renders unquote(x) valid, as x is in scope when unquote is evaluated.

Macro Hygiene in Elixir

While we are on the topic of scopes in macros, let's discuss macro hygiene.

According to tutorialspoint.dev:

Hygienic macros are macros whose expansion is guaranteed not to cause the accidental capture of identifiers.

This means that if we inject and expand a macro into the callsite, we need not worry about the macro's variables (defined in the caller context) conflicting with the caller's variables.

Elixir ensures this by maintaining a distinction between a caller variable and macro variable. You can explore this further using an example from the official tutorial.

A macro variable is declared within the body of quote, while a caller variable is declared within the callsite of the macro.

defmodule Foo do
  defmacro change do
    quote do: a = 13
  end
end

defmodule Bar do
  require Foo

  def go do
    a = 1
    Foo.change
    a
  end
end

Bar.go
# => 1

In this example, a referenced in change is not the same variable a in the scope of go. Even when we attempt to change the value of a, the value of a in the scope of go remains untouched.

However, there may be a time where you have to reference a variable from the caller's scope in the macro's scope.
Elixir provides the macro var! to bridge the gap between these two scopes:

defmodule Foo do
  defmacro change do
    quote do: var!(a) = 13
  end
end

defmodule Bar do
  require Foo

  def go do
    a = 1
    Foo.change
    a
  end
end

Bar.go
# => 13

This distinction ensures no unintended changes to a variable due to changes within a macro (whose source code we may not have access to).

You can apply the same hygiene to aliases and imports.

Understanding `require`

In the code examples shown in this section, we have always used require <module> before we invoke the macros within the module. Why is that?

This is the perfect segue into how the compiler resolves modules containing macros — particularly the order in which modules are compiled.

In Elixir, modules compile in parallel, and usually — for regular modules — the compiler is smart enough to compile dependencies of functions in the proper order of use. The parallel compilation process pauses the compilation of a file until the dependency is resolved. This behavior is replicated when handling modules that contain macro invocations.

However, as macros must be available during compile-time, the module these macros belong to must be compiled beforehand.
Here's where require comes into the picture. require explicitly informs the compiler to compile and load the module containing the macro first.

We can use an example to illustrate this behavior:

defmodule Foo do
  IO.puts("Compiling Foo")

  defmacro foo do
    IO.puts("Foo.foo := macro")
    quote do
      IO.puts("Foo.foo := caller")
    end
  end
end

defmodule Bar do
  IO.puts("Compiling Bar")

  require Foo

  IO.puts("Bar := before Foo.foo")
  Foo.foo()
end

Compiling Foo
Foo.foo := macro
Compiling Bar
Bar := before Foo.foo
Foo.foo := caller

(Note that this is just an approximation of the actual compilation process, but it aims to paint a clearer picture of how require works.)

Let's try to understand why the outputs are in this order.

Firstly, Bar tries to compile. The compiler scans and finds a require for Foo before evaluating any module-level expressions within the module (such as IO.puts). So it pauses the compilation of Bar and compiles the Foo module first. As Foo is compiled, module-level code — like IO.puts — is evaluated, and the compiler prints the first line of the output.

Once Foo is compiled, the compiler returns to Bar to resume compilation. Bar is parsed, macro calls are executed, and the macro context is evaluated. Even though Foo.foo is called after IO.puts("Bar := before Foo.foo"), the evaluation of the macro call takes precedence over the evaluation of module-level code.

During expansion, Foo.foo's caller context is injected and expanded into the callsite in Bar. It then behaves just like a regular module-level function call, printing the last three output lines in order of declaration.

In essence, require instructs the compiler on the order of compilation that each module should go through if there are macro dependencies. This ensures that the macros are available during compile-time.

Escaping Macros in Elixir

Before explaining what Macro.escape does, let's look at an example:

iex(1)> x = {1, 2, 3}
iex(2)> quote do: unquote(x)
{1, 2, 3}
iex(3)> ast = quote do: IO.inspect(unquote(x))
{{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [{1, 2, 3}]}
iex(4)> Code.eval_quoted(ast)
** (CompileError) nofile: invalid quoted expression: {1, 2, 3}

Please make sure your quoted expressions are made of valid AST nodes. If you would like to introduce a value into the
AST, such as a four-element tuple or a map, make sure to call Macro.escape/1 before
    (stdlib 3.15) lists.erl:1358: :lists.mapfoldl/3
    (elixir 1.11.3) lib/code.ex:706: Code.eval_quoted/3

That is a strange error. Based on our understanding of unquote and macros, the code should work as intended, but it doesn't. Why is that?

Well, the answer is found on iex(2). When we attempt to unquote x, we return not an AST, but a tuple — the same one initially assigned to x. The error then points to the fact that the tuple is an invalid quoted expression.

When we unquote(x) as-is, we inject a raw tuple into the AST, which cannot be evaluated as it is not a valid AST and throws an error.

So, how do we fix it?

We need to convert the raw tuple referenced by x into a valid AST. This can be achieved by escaping this value using Macro.escape. Let's understand what Macro.escape does:

iex(1)> a = {1, 2, 3}
{1, 2, 3}
iex(2)> Macro.escape(a)
{:{}, [], [1, 2, 3]}
iex(3)> quote do: unquote(a)
{1, 2, 3}
iex(4)> quote do: unquote(Macro.escape(a))
{:{}, [], [1, 2, 3]}

In iex(2), we see that Macro.escape(a) returns an AST of the tuple, not the raw tuple — and this is exactly what we are looking for. By combining Macro.escape's behavior with unquote, we can inject the AST of the tuple into the quote as seen in iex(4).

Let's test this:

iex(1)> x = {1, 2, 3}
{1, 2, 3}
iex(2)> quote do: unquote(Macro.escape(x))
{:{}, [], [1, 2, 3]}
iex(3)> ast = quote do: IO.inspect(unquote(Macro.escape(x)))
{{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [],
 [{:{}, [], [1, 2, 3]}]}
iex(4)> Code.eval_quoted(ast)
{1, 2, 3}
{{1, 2, 3}, []}

As you can see, the code works just as intended because we escape the tuple.

Often when working with data structures like tuples and dictionaries, you may find that the injected data from unquote does not inject a valid AST. In these cases, you should use Macro.escape before unquote.

Guard Clauses and Pattern Matching

Finally, it's worth mentioning that, much like regular functions defined through def, macros can use guard clauses and pattern matching:

defmacro foo(x) when x == 4 do
  IO.inspect("macro with x being 4")
end

defmacro foo(_) do
  IO.inspect("macro with any other value")
end

Up Next: Applications of Macros in Elixir

Congratulations, you've made it to the end of this part! You should now have a better grasp of how macros work internally.

In part three, we will look at the many applications of macros in Elixir.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

An Introduction to Metaprogramming in Elixir

Jiahao — Tue, 14 Sep 2021 12:28:26 +0000

In this world, there are many mysteries — but few are as elusive as metaprogramming in Elixir.

In this four-part series, we'll start by looking at core concepts and then explore how metaprogramming operates in Elixir specifically.

Let's develop an understanding of metaprogramming and uncover some Elixir metaprogramming secrets!

Introducing Metaprogramming in Elixir

According to Harald Sondergaard, metaprogramming is:

a programming technique in which computer programs have the ability to treat other programs as their data; meaning that
a program can be designed to read, generate, analyze, or transform other programs, and even modify itself while
running.

In essence, metaprogramming — much like metadata — revolves around "a set of programs that describe and give information about other programs" (adapted from the Oxford dictionary definition of metadata).

Types of Metaprogramming

There are two types of metaprogramming that prescribe varying degrees of control over a given program:

Introspection

Introspection refers to a program revealing metadata about other programs or itself.

This definition broadly covers the first part of the definition of metaprogramming: "a program can be designed to read...[and] analyze...other programs". The program has access to information about itself or other programs.

Reflection

Reflection refers to a program modifying other programs or itself.

If a program can modify other programs or itself, it — by definition — has access to the metadata of the program, revealing information like names of functions.

By looking at the two types of metaprogramming, we can conclude that reflection encompasses introspection, or introspection is a subset of reflection:

Why Do Metaprogramming?

We will look at how to use metaprogramming specifically in Elixir. But first, let's cover some general concepts and benefits of metaprogramming.

According to Wikipedia, metaprogramming can be used to (but not limited to) achieve the following:

Move computations from run-time to compile-time
Generate code using compile-time computations
Enable self-modifying code

Observe that among these use cases (as well as across metaprogramming), the terms "run-time" and "compile-time" are used frequently. Let's see what they mean.

Defining Run-time and Compile-time

We can broadly classify metaprogramming into two categories: compile-time and run-time metaprogramming.

But what exactly are run-time and compile-time? They both refer to stages of a program's life cycle.

Compile-time is the stage at which source code converts to binary code or intermediate binary code for a machine or virtual machine to execute. Run-time refers to when code executes.

The program life cycle includes the following steps:

Source: https://en.wikipedia.org/wiki/Program_lifecycle_phase

Note that this is not a complete representation of the entire program life cycle, just a simplified one.

Compilation "sets the program in stone" by converting it into binary code. Metaprogramming exposes this process to allow developers to "move computation from run-time to compile-time" or "generate code using compile-time computations". This essentially allows the modification of the source code before/during compile-time, meaning that the generated binary code is slightly different.

Self-modifying code is rather unique. In essence, it performs reflection during run-time. The life cycle of a self-modifying program looks a little different:

Note: You can substitute the "binary" for any intermediate language generated by the compiler, such as JVM bytecode or, in Elixir's case, BEAM VM bytecode.

Interesting Uses of Metaprogramming

The general definition of metaprogramming also encompasses the tools used in a program's life cycle.

For instance, a language compiler is a metaprogramming application designed to receive another program as input and generate binary as output.

We can narrow down broad metaprogramming use cases to the following, more specific, applications in a programming language context:

Code generation

By generating code dynamically during compile-time, it's available during run-time. When the nature of the code that's generated is not fixed, this can prove especially useful. For instance, you can use code generation to design domain-specific languages (DSLs) or generate functions based on input such as files or APIs.

Code instrumentation

Code instrumentation refers to the measure of a program's
performance, error diagnosis, and logging of trace information.

Metaprogramming enables this through dynamic program analysis — software analysis performed by running software through a real or virtual processor.

Code instrumentation enables features like code coverage, memory error detection, fault localization, and concurrency errors.

Behavioral changes

This refers to changing the behavior of a program through metaprogramming. Behavioral changes can include feature toggling, where a given feature is toggled on/off through a flag that is read during compile-time/run-time.

This article series is about metaprogramming within Elixir, so our key focus will be on code generation.

Metaprogramming in Elixir: The Basics

Elixir applies a style of metaprogramming known as macro system metaprogramming (also used in other languages like Rust and Lisp).

In Elixir, metaprogramming allows developers to leverage existing features to build new features that suit their individual business requirements.

The foundation of metaprogramming in Elixir is macros.

Defining Macros

According to the official documentation:

Macros are compile-time constructs that are invoked with Elixir's AST as input and a superset of Elixir's AST as
output.

There are two critical components to this definition. Let's break them down:

Compile-time constructs - evaluated and available during compile-time
Elixir's AST - Abstract Syntax Trees (ASTs) are tree representations of the abstract syntax structure of the source code

We use the representations of the source code as building blocks for compile-time constructs. Since the compiler reasons with the source code through ASTs, we effectively "speak" the compiler's language to build constructs that it can directly reason with.

In Elixir, ASTs are tuples, so we reason with the compiler in a manner that is familiar to us. We do not need to deviate from Elixir's syntax to begin writing macros - lowering our barrier to entry of learning macros. On top of that, we do not even need to write ASTs ourselves. There are constructs in Elixir to handle all of that heavy lifting for us.

The above definition also mentions how a macro receives an AST as input and returns a superset of AST as output. So, you can think of a macro as a regular function with inputs, behavior, and an output. The overall goal is to use a given AST to generate a new AST for the compiler to use.

There is more to come on the compilation process of Elixir programs in part two of this series.

Starting Small with Macros

Now that we understand macros, let's dip our toes into the water and implement a basic macro.

We'll start with a very basic comparison of a macro to a regular function.

The Elixir documentation inspires this code example:

defmodule Foo do
  defmacro macro_inspect(value) do
    IO.inspect(value)
    value
  end

  def func_inspect(value) do
    IO.inspect(value)
    value
  end
end

To define a macro, we use defmacro and declare the parameters just as we would a regular function.

Running the macro in IEX yields the following results:

iex(1)> import Foo
iex(2)> macro_inspect(1 + 2)
{:+, [context: Elixir, import: Kernel], [1, 2]}
3
iex(3)> func_inspect(1 + 2)
3
3

Observe that rather than printing the result of 1 + 2, the macro prints a tuple instead (the AST as input that we defined earlier).

When a macro is first declared, the arguments of that macro are automatically converted into AST so that you don't need to parse the arguments manually. The arguments will not be evaluated beforehand.

However, when the value of the macro is returned, it yields the result of 1 + 2. The macro should return an AST as output (and it is). However, this AST as output is compiled and executed once the macro is called. The expression 1 + 2 is evaluated first, then returned.

Once we understand the basic syntax and declaration of a macro, we can explore the structure of the AST.

AST Structure

As mentioned earlier, the AST is the representation of the source code as a syntax tree. In the example above, we inspect the AST of the expression 1 + 2.

We can break down the AST structure into three components:

Atom — representing the name of the operation
Metadata of the expression
Arguments of the operation

{
  :+,                                 # operation name,
  [context: Elixir, import: Kernel],  # metadata,
  [1, 2]                              # operation arguments
}

While you must understand what comprises an AST, we rarely need to read/write raw ASTs.

Elixir makes it ridiculously easy to interface with macros, so we hardly even need to think about the structure of the AST that we are working on — everything is handled for us.

Interacting with ASTs

As mentioned earlier, ASTs represent the source code and are the input and output of macros. They are the cornerstone of macros. We need to interact with the AST representations of expressions freely, without getting bogged down by reading and writing the ASTs ourselves.

This is where quote and unquote come into the picture.

To generate the AST representation of an expression or body, we use quote:

quote do
  1 + 2 * 3
end

{:+, [context: Elixir, import: Kernel],
 [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]}

When we use quote, we build an AST. While the example above is relatively simple, we will soon discover that quote can be used to build much more complex ASTs.

What if we have a value we want to use in our quote, such as the arguments? We attempt to introduce an external (outside of quote) variable into quote, by using unquote.

unquote evaluates its argument, which is an expression, and injects the result (as an AST) into the AST being built. As everything in Elixir is an expression, we evaluate expressions to inject the results.

For instance, if unquote receives a variable, we will evaluate that expression as the underlying expression referenced by the variable and inject that.

If unquote receives a full expression like 1 + 2 * 3, we will evaluate that to 7 and inject that. unquote expects that the result of the expression is a valid AST.

In part two of this series, we'll discuss the consequences of having an invalid AST and delve into macros more deeply.

Do you recall that macros automatically convert arguments into their AST forms? We will leverage that behavior:

defmodule Foo do
  defmacro foo(exp) do
    quote do
      doubled = unquote(exp) * 2
      doubled
    end
  end
end

Foo.foo(1 + 2 * 3)
14

As you can see, we have built a macro called foo which receives an expression as an argument. Then, we begin to build an AST for the macro in quote. We use unquote(exp) to inject the value of the exp argument into the AST.

You might ask yourself: How do I know that the expression is injected and not evaluated right away?

Well, we can use a handy tool to inspect the AST of the macro and understand how it works under the hood:

iex(1)> require Foo
iex(2)> ast = quote do: Foo.foo(1 + 2 * 3)
iex(3)> ast |> Macro.expand(__ENV__)
{:__block__, [],
 [
   {:=, [],
    [
      {:doubled, [counter: -576460752303423358], Foo},
      {:*, [context: Foo, import: Kernel],
       [
         {:+, [context: Elixir, import: Kernel],
          [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]},
         2
       ]}
    ]},
   {:doubled, [counter: -576460752303423358], Foo}
 ]}

First, we generate the AST of the macro call and assign it to a variable.

Then, with our ast variable, we will use Macro.expand to expand the AST to its fullest form.

We'll look at macro expansion next time. For now, think of it as peeling back the layers of an AST to its most fundamental components.

As you can see, the expanded form of the Foo.foo call contains the AST of 1 + 2 * 3. This proves that unquote only injected the AST of the expression into the quote AST, but didn't evaluate it. The evaluation is performed later on (we will get into this in part two as well).

Note: Macro.expand will only attempt to perform expansion on the root node of the AST.

You can find more information about Macro.expandin the docs.

`quote` Options in Elixir

Now that we understand the fundamentals of macros, we can start to look at our quote options.

While there are several options with quote, we will focus on the three most frequently used and introduce the concepts behind each option.

unquote

Toggles the unquoting behavior in quote. By disabling it, any unquote call is converted to an AST of the macro call (as with any other macro/function call).

This defers the evaluation of unquote to a later point. I'll explain why you'd want to do so in the next part of this series.

For now, let's look at the following example:

iex(1)> a = [foo: 1, bar: 1]
iex(2)> ast = quote do: unquote(a)
  [foo: 1, bar: 1]
iex(3)> ast = quote unquote: false, do: unquote(a)
  {:unquote, [], [{:a, [], Elixir}]}

When we leave the unquoting behavior enabled (iex(2)), unquote(a) will evaluate a as an expression. This returns the keyword list, which is then injected into the quote AST — and the result is as expected.

However, when we disable the unquoting behavior (iex(3)), unquote(a) is converted into another AST expression, which is injected into the quote AST as-is.

bind_quoted

Disables unquoting behavior in the quote and binds given variables in the body of quote.

Binding moves the variable initialization into the body of quote.

We can observe this behavior using Macro.to_string:

iex(1)> a = [foo: 1, bar: 2]
iex(2)> ast = quote bind_quoted: [a: a], do: IO.inspect(a)
iex(3)> ast |> Macro.to_string |> IO.puts
(
  a = [foo: 1, bar: 2]
  IO.inspect(a)
  :ok
)
:ok

As you can see, bind_quoted adds a "copy" of a into the body of quote by assigning it in the body of quote.

In a macro, this is equivalent to binding the variable to the caller context, as the variable is initialized during the evaluation of the callsite.

Note: Contexts will be discussed in greater detail next time.

  defmodule Foo do
    defmacro foo(x) do
      quote bind_quoted: [x: x] do
        IO.inspect(x)
        end
    end
end

location

This option controls whether run-time errors from a macro are reported from the caller or inside the quote.

By setting this option to :keep, error messages report specific lines in the macro that cause the error, rather than the line of the callsite.

You can see a code example in the docs.

Build a Simple Macro in Elixir

We should now be able to build a simple macro that mimics the behavior of an if statement.

Recall that an if statement is comprised of the following components:

if (condition) do
  # body
else
  # body
end

We can replicate this structure using our own macro:

defmodule NewIf do
  defmacro if?(condition, do: block, else: other) do
    quote do
      cond do
        unquote(condition) == true -> unquote(block)
        unquote(condition) == false -> unquote(other)
      end
    end
  end
end

iex(1)> require NewIf
iex(2)> NewIf.if? 4 == 5, do: :yes, else: :no
:no

This macro can receive three arguments:

condition - predicate to evaluate if? statement against
do - block to execute when condition is true
else - block to execute when condition is false

In Elixir, such blocks can be declared as arguments if they follow the following syntax: <formal name>: <variable name>. The formal name is the name used when you call the macro. The variable name is the name used in the macro when you're attempting to reference the block.

After receiving these three arguments, we start by building an AST using quote.

Using a cond statement, we determine which body if? should execute. We use unquote to inject the values of condition, block, and other into the AST we are building.

In doing so, when the macro is evaluated, the condition is evaluated to be true/false, and, based on that result, we will either execute block or other.

We wrap up this behavior into an AST returned by quote (which is the return value of the macro).

Next Up: Macros in Detail

Now we have a good grasp on the foundations of metaprogramming in general and specifically in Elixir.

Join me for the next part of this series, where we'll look into the intricacies behind macros and how everything works.

Until next time!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Jia Hao Woo is a developer from the little red dot - Singapore! He loves to tinker with various technologies and has been using Elixir and Go for about a year. Follow his programming journey at his blog and on Twitter.

Open-source Deep Dive: Broadway (Part 1) - Message queues, concurrency in Elixir, and Broadway architecture

Jiahao — Mon, 12 Apr 2021 00:00:00 +0000

This open-source deep dive has been split into two parts! The first part covers the prerequisite knowledge that would be good to know when trying to understand the inner workings of Broadway. The second part is an in-depth analysis of the implementation of various features of Broadway.

This is the first part of the deep dive and the following topics will be covered:

A brief introduction to what Broadway is
Message queues
Concurrency in Elixir
Producer/consumer model & GenStage
Architecture of a Broadway pipeline
Construction of producer & processor components

If you wish to jump right into the meat of Broadway, you can find the second part here!.

Act 1, Scene 1

You have just received your latest feature to work on and it is to build a system that receives transaction information from a message queue, maps the customer code in this transaction information to the customer's information, and stores this collective information in a separate database to be queried for customer transaction analysis. Your boss has developed an obsession with Elixir recently and is now pushing for every project to use it. Gasp.

You start researching for libraries that can do exactly that and stumble upon Broadway.

...build [concurrent] and [multi-stage] [data ingestion] and [data processing] [pipelines]...

Oh boy... that — that is a mouthful... Let's break it down, shall we?

concurrent - having two or more computations in progress at the same time; in progress meaning that they do not have to be executed at the same time (definition here)
multi-stage - successive operating stages (definition here)
data ingestion - process of moving data from one source to a destination for further storage and analysis (definition here)
data processing - conversion of data into a usable and desirable form (definition here)
pipelines - series of data processing elements (definition here)

In essence, Broadway builds systems that behave like factory assembly lines. Raw materials (data) is fed into the assembly line (Broadway pipeline) which is then pieced together to create the end product or other components used in the final product. The factory has multiple identical assembly lines running so raw material can be fed into any of these lines to be worked on.

For your use case, the flow of data will look something like this:

So how does Broadway achieve all of this?

Lights! Camera! Action!

Before understanding the internals of Broadway, we should establish some basic knowledge of the technologies we will be using so that we won't be headless chickens running into this.

Broadway revolve around the following concepts:

message queues
concurrency in Elixir

What are message queues?

Note! While Broadway can integrate with many types of data sources, the core examples given in the project focus on message queues as the primary data source.

Message queues are like containers that hold sequences of work objects — called messages — that are to be consumed and processed. It aids with building asynchronous modular and concurrent systems.

Messages are created and delivered to these queues by producers and taken from these queues for processing by consumers. These messages can vary from something as simple as plain information to more complex structures like requests or — in our case — transaction information.

Message queues are useful for decentralising the communication mechanism of large systems by acting as a medium for exchanging events between systems which allows for systems to be easily scaled and distributed.

This is a reduced explanation of what a message queue is and what it is capable of. For more information about message queues, the Amazon documentation and this blog post by CloudAMQP are good places to start.

Concurrency in Elixir

Broadway relies heavily on concurrency in Elixir. The topology (architecture) of a pipeline is built on top of processes and many of the features are achieved using the robust concurrency model of Elixir. So what exactly is the concurrency model in Elixir?

Elixir employs the actor concurrency model. In this model, actors are defined as self-isolated units of processing. In Elixir, these actors are called processes and they are managed by the Erlang VM. Elixir code is run in each process and a default/main process is akin to that of the main thread in other concurrency models.

Each process communicates via asynchronous message passing. Think of a process as a mailbox of sorts; it has a "bin" to receive incoming messages and it possess an "address" for other processes to identify it by.

The unique aspect of this model is the lack of shared mutable state that other concurrency models rely on. Rather, state is exclusive to each process.

In order for the state of a process to be altered, the owner process must make the alteration either on request or internally due to certain changes.

The topic of concurrency in Elixir is vast and Elixir provides many other features surrounding its concurrency model such as GenServer. This section is a short preview of what the actor concurrency model and concurrency in Elixir is all about. For more information, you can refer to this thesis paper and the Wikipedia article talking about the actor concurrency model and the official documentation and this tutorial on OTP in Elixir for more examples of concurrency in Elixir.

Cue the producer/consumer model

Using the actor concurrency model as a foundation, another concurrency pattern can be modelled in Elixir — the producer/consumer model.

This model aims to allow for decoupled data production and consumption by setting up two separate processes to handle each task — effectively creating a logical separation of concerns.

However, the producer/consumer model faces a critical issue — what happens if the producer generates excessive messages for the consumer? The consumer will be overwhelmed and will eventually fail trying to keep up with processing that many messages. This is where back pressure comes into play.

Back pressure is a control mechanism for how much a producer should emit based on consumer demand, consumer message buffering, or limited sampling

Back pressure avoids the problem of overloading the consumer with messages by applying one of or a combination of the three methods mentioned above (more information in the link here).

The next frontier (of concurrency): GenStage

Seeing the value of having a standard implementation for the producer/consumer model, the Elixir team decided to develop exactly that.

GenStage is a specification for exchanging events between producers and consumers with back pressure between Elixir processes

Producers emit events to consumers for processing. The events can be of any structure.

The control mechanism used is a demand system. Consumers inform producers of how many events they can handle (demand) and producers emits no more than the demanded amount. This ensures that the consumers are capable of handling the events emitted.

Producer-consumers behave like both producers and consumers. They are used to perform transformations on events emitted by the producer before they are emitted to the consumer.

Similar to GenServer, stages in GenStage exchange events through callbacks.

When a demand is handled — i.e. producer emits events and demanding consumer handles these events — another demand is made, creating a cycle where both stages are always working - ideally.

GenStage is a powerful tool in an Elixir developer's arsenal. More information can be found in the official announcement where a little bit of history of how GenStage came to be was discussed and in a talk by José Valim — creator of Elixir.

With a better grasp of the overarching concepts used in Broadway, we can finally discuss what Broadway is all about and how it does what it does!

Pipeline architecture

It is at this juncture where it would be important to clarify the term "producer". In both message queues and GenStage, a producer is a creator of messages or events. However, in Broadway, a producer is both a consumer of messages and an emitter of events.

For the rest of the article, the following definitions for the following terminology will be used:

producer — producer of events in Broadway
message — message in a message queue or any other data source
event — GenStage events

When messages are consumed by the producer, they will be transformed into events with a fixed structure defined by Broadway.

Each component is a separate process and they are dynamically generated as different topologies (architectures) can be designed. The order of initialisation for a typical pipeline looks something like this:

The producers and processors are both created using interesting conventions that is will be explored now. Other components will be discussed later on as they tie into other features Broadway has.

How it's made: Producers

Producers are built using a pattern similar to the strategy pattern but modified to integrate with the concurrency system in Elixir.

Different data sources require different methods of establishing connections and receiving messages. Thus, we break up the producer process into two modules — ProducerStage defines the behavior for enforcing the rate limit while a dynamically loaded module defines the behavior for establishing a connection to the data source and receiving messages.

ProducerStage assumes that the dynamically loaded module contains the typical GenStage callbacks like handle_call and handle_demand and uses them for things like rate limiting.

The ProducerStage behaves as the context while the dynamic module behaves as the strategy. The dynamic module adopts the Producer module — which defines two callbacks for managing the overall producer life-cycle.

To load the module dynamically, the module name is passed to ProducerStage as an argument. To keep the producer as a single process, we call the init function of the module directly when initialising the ProducerStage. This way, the module will initialise under the newly spawned process for ProducerStage rather than spawning an entirely new process.

@impl true
def init({args, index}) do
  {module, arg} = args[:module]
    # ...
  state = %{
    module: module,
    module_state: nil,
    transformer: transformer,
    consumers: [],
    rate_limiting: rate_limiting_state
  }

    # Calling the init function of the dynamically loaded module
  case module.init(arg) do
        # ...
  end
end

When start_link is called, a new process is spawned first before the init function is called under the new process.

This is done as certain message queue providers like RabbitMQ attach active listeners to the calling process so spawning a separate process for this would mean having to manage two separate processes for a producer.

How it's made: Processors

Processors are created using a concept similar to inheritance in object-oriented programming. This idea comes from the need to standardise the subscription logic of producer-consumers and consumers.

When a processor is started using start_link, a process of the Subscriber module is started with the current processor module passed as a argument.

def start_link(args, stage_options) do
  Broadway.Topology.Subscriber.start_link(
    __MODULE__,
    args[:producers],
    args,
    Keyword.take(args[:processor_config], [:min_demand, :max_demand]),
    stage_options
  )
end

The current module is initialised in the Subscriber process through init.

@impl true
def init({module, names, options, subscription_options}) do
  {type, state, init_options} = module.init(options)

    # ...
end

Other producer-consumers and consumers like batcher and batch processors also use this pattern to create their respective GenStage stages.

A separation of concern is achieved using this pattern. The processor is responsible for event handling while the subscriber handles the subscription logic.

That's a basic rundown of the concepts underpinning Broadway. While it may not be a complete and intensive explanation of everything, hopefully it is able to provide some clarity. In the next part, we will be exploring how features in Broadway have been implemented!

Hop on over to the second part here!

Open-source Deep Dive is a series where I pick apart open-source projects to explain the underlying concepts that power these projects and share my findings about the project!

Open-source Deep Dive: Broadway (Part 2) - Inner workings of Broadway

Jiahao — Mon, 12 Apr 2021 00:00:00 +0000

This is the second part of the deep dive and the following topics will be covered:

Rate limiting
Batching messages
Telemetry
Creating a built-in testing support for pipelines
Achieving graceful shutdowns
Other interesting bits of code

If you want a refresher on the concepts behind Broadway (like message queues and concurrency in Elixir) or to better understand Broadway's pipeline architecture from a bird's eye view, you can find the first part here!

What's the scoop?

Now that we have explored the overall architecture of a Broadway pipeline, we can look at how certain features in Broadway are implemented.

Rate limiting

Rate limiting refers to the act of limiting the amount of data that can be requested or processed in a given period of time

Rate limiting is applied across producers within a single pipeline to control the number of events emitted within a given period of time.

This is especially useful when the hardware of the machine running the pipeline is not able to keep up with processing large numbers of events demanded at a time — possibly due to a poorly configured pipeline.

Some producers do not leverage the rate limiting feature of Broadway. For instance, the RabbitMQ producer creates an active listener, which means that event emission is not inhibited by the rate limiter. Instead, events are emitted the moment messages are published to the message queue (unless otherwise configured).

But for the producers that do leverage the rate limiting — such as the Amazon SQS producer — rate limiting is applied in two instances:

When consumers make demands to the producer-consumer or producer

If the producer can still emit events, any demand made by the consumer will be handled by the producer. We take into account the rate limit threshold. If there are too many events to emit, the excess messages are stored in a message buffer that will have to be cleared later on.

Each message that can be emitted will be transformed into the standard event structure that Broadway uses.

If the producer can no longer emit messages, any demand made is stored in a demand buffer that is cleared later on.
When the rate limit is being reset after the given interval

After the given interval, the rate limit threshold can be reset. However, we may have accumulated demands and messages in their respective buffers. We may find that the threshold has not been met before we reset it. Thus, we can use this remaining threshold to clear any lingering demands and messages stored in their respective buffers.

Once we have cleared as many messages as our remaining threshold allows, we will reset the threshold and schedule for another reset. These resets are scheduled at fixed intervals.

The rate limiting threshold is maintained as an atomic (discussed later on). This atomic array is generated by the RateLimiter process. This module handles all behavior surrounding working with the rate limit threshold. ProducerStage handles the actual logic of managing the demands of consumers.

When the producer cannot emit any more events, i.e. the threshold has been reached, an internal state is set to :closed to avoid future demands from being handled.

Batching

Batching groups events based on given properties and sends them to designated "sub-pipelines" or batch processors to be handled. For instance, we might design a pipeline that stores events with even numbers in an S3 bucket and ones with odd numbers on Google Drive.

The Batcher process is assigned unique names for identification and events that are emitted from the producer must be tagged to a batcher. Failure to do so will result in a runtime error. This only applies if batching is enabled.

In order for the producer to send the appropriate events to the respective batcher, a PartitionDispatcher is used. Essentially, it defines the behavior of how events are emitted to consumers. A PartitionDispatcher dispatches events to certain consumers based on a given criteria (defined as a hash function). In this case, the criterion is the name of the batcher from the given event. This means that when we assign a batcher to the event, it will be dispatched to only that batcher. More information about dispatchers in GenStage can be found in the official documentation.

Even within the batcher, further grouping can be made based on a batch key assigned to the event. This may be used to ensure that certain events are processed together. Internally, the batcher will accumulate events before emitting them. However, as it cannot sit around accumulating events forever, a batch is emitted at regular intervals regardless of how many events are stored in it.

The BatchProcessor process handles a single batch at a time. It is similar to a regular processor, except it works on a batch of events. The handle_batch callback is used here.

Telemetry

Telemetry is used in Broadway to benchmark certain operations that occur such as the duration that a handle_message callback takes.

Broadway relies on the telemetry library. Within the code, events are emitted when these operations occur and key measurements such as duration are tracked. Handlers/listeners of these events can be setup to respond to these events.

Telemetry is not an Elixir-only feature. It is commonly used to perform application monitoring. OpenTelemetry is a really interesting framework that offers powerful application monitoring through telemetry.

Built-in testing

To test the pipeline, we should focus on ensuring that the data processing aspect of the pipeline works as intended. However, as we rely on external services for input, it would be hard to coordinate a test suite to work with a live data source as we may not be able to replicate the data source or publish data to the data source at will due to access limitations. Thus, Broadway has designed a testing utility that allows us to test the pipeline's data processing capacity without relying on the data source.

Broadway provides a placeholder producer module. This producer does not rely on any data sources. Instead, messages are emitted directly into the pipeline.

The producer module should be tested separately if there is core behavior that cannot be tested along with the pipeline.

This form of unit testing ensures that we reduce potential points of failure in our test suite if any of the aforementioned problems with using the original data source should surface.

Graceful shutdowns

Broadway boasts about having graceful shutdowns. This is a rather interesting concept to explore as it relies heavily on the concurrency system of Elixir.

Essentially, the pipeline can only exist in two states — when all components are online and when all components are shutting down. There is no point in time where a single component will shutdown on its own without being restarted. This is because of the way that the supervisor of each component declares restart strategies ensuring that should a child process encounters any errors, it will be restarted without a hitch. This way, the only time where our components can shut down is when we shut down our main process or pipeline supervisor process. When either process is terminated, we want to properly handle all remaining events in the pipeline before shutting off every component.

This is achieved through a mix of concurrency features. But before we can explain how it works, a simple introduction of exit signals and process termination is due.

Exit signals and process termination

Processes can be linked to one another. When either process receives an exit signal — which can occur when the process is terminated forcibly or when it receives an exit signal propagated from its parent — it will propagate the exit signal to the linked process and that process will terminate as well.

However, these exit signals can be trapped instead. When this occurs, rather than terminating the process that receives the propagated exit signal, the exit signal is sent as a message, allowing the receiving process to handle the exit as though it was just another message.

def handle_info({:EXIT, from, reason}, state) do
    # ...
end

When a process is terminated, an optional terminate/2 callback can be declared to perform any cleanup before the process is actually terminated. This is useful if we have any lingering operations that should be completed before we terminate the process.

Supervisors can start a list of child processes and is responsible for managing the restart strategy of each child. The interaction between a supervisor and terminate is rather interesting. When a child is terminated, it is restarted accordingly. When a supervisor terminates, all of its children will also be terminated. If a child process traps exits, the terminate callback is called. If not, it will simply terminate immediately without calling the callback. More information about how supervisor interact with shutdowns can be found in the official documentation.

Back to our regularly scheduled deep dive...

With a basic understanding of exit trapping and process termination, we can actually understand how graceful shutdowns in Broadway works.

When the main process or the pipeline supervisor process is terminated, the main process — which traps exit signals — will invoke its terminate callback which will inform the Terminator process to begin trapping exits and terminate our pipeline supervisor. As this Terminator process is a child of the pipeline supervisor, it will invoke its implementation of terminate.

@impl true
def terminate(reason, %{name: name, supervisor_pid: supervisor_pid, terminator: terminator}) do
  Broadway.Topology.Terminator.trap_exit(terminator)
  ref = Process.monitor(supervisor_pid)
  Process.exit(supervisor_pid, reason_to_signal(reason))

  receive do
    {:DOWN, ^ref, _, _, _} -> :persistent_term.erase(name)
  end

  :ok
end

The exit signal propagates to the other components through their supervisors terminating and they will also invoke their terminate callback if they trap exits such as producers disconnecting from the data source.

The Terminator process is responsible for ensuring that all events still within the pipeline are processed before terminating the pipeline entirely.

It does so in three phases:

Notify that the processors do not resubscribe to producers through a state flag
Drain the producers of any events remaining by emitting the events through the pipeline
Wait for the batch processors (which will be the very last component in the pipeline) to terminate before terminating the supervisor

@impl true
def terminate(_, state) do
  for name <- state.first, pid = Process.whereis(name) do
    send(pid, :will_terminate)
  end

  for name <- state.producers, pid = Process.whereis(name) do
    Broadway.Topology.ProducerStage.drain(pid)
  end

  for name <- state.last, pid = Process.whereis(name) do
    ref = Process.monitor(pid)

    receive do
      {:done, ^pid} -> :ok
      {:DOWN, ^ref, _, _, _} -> :ok
    end
  end

  :ok
end

Interestingly, as the producer may be waiting to drain events, we may not want to cancel all of its consumers immediately. Thus, we rely on GenStage#async_info to queue the message to cancel all consumers at the end of the GenStage message queue — effectively waiting for all other events to be processed before cancelling all consumers. If batching is enabled, the processors will also wait for the batches to be processed before cancelling all consumers.

@spec drain(GenServer.server()) :: :ok
def drain(producer) do
  GenStage.demand(producer, :accumulate)
  GenStage.cast(producer, {__MODULE__, :prepare_for_draining})
    # The :cancel_consumers message is added to the end of the message queue
  GenStage.async_info(producer, {__MODULE__, :cancel_consumers})
end

These mechanisms ensure that all events left in the pipeline is properly processed before the pipeline terminates, thus achieving graceful shutdowns.

Fascinating discovery!

These are some interesting bits of code that Broadway has.

`using` configurations

Like other libraries in Elixir, use Broadway is where it all begins. As discussed in the previous open-source deep dive, the behavior of use can be altered by defining the __using__ macro.

defmacro __using__(opts) do
  quote location: :keep, bind_quoted: [opts: opts, module: __CALLER__.module] do
    @behaviour Broadway

    @doc false
    def child_spec(arg) do
      default = %{
        id: unquote(module),
        start: {__MODULE__, :start_link, [arg]},
        shutdown: :infinity
      }

      Supervisor.child_spec(default, unquote(Macro.escape(opts)))
    end

    defoverridable child_spec: 1
  end
end

There are three interesting bits of code in the __using__ macro:

location: keep

Used to report runtime errors from inside the quote. Without this, errors are reported where the defined function (in quote) is invoked. This is to ensure that we are aware of where the errors are occurring. More information about this configuration can be found here.
bind_quoted

Used to create bindings within the quote. When a binding is created, the value is automatically unquoted (which includes evaluation) and the value cannot be unquoted again. This is especially used when we do not want to re-evaluate the value multiple times.

More information quoting and unquoting in Elixir can be found in the official tutorial and a simplified explanation and example of binding can be found here.

@behaviour

Used to define interface-like behavior where modules that adopt these behaviors can implement callbacks defined. In this case, when a module use Broadway, it will have to implement certain callbacks like handle_message while other callbacks like handle_batch remain optional.

@callback prepare_messages(messages :: [Message.t()], context :: term) :: [Message.t()]
@callback handle_message(processor :: atom, message :: Message.t(), context :: term) ::
            Message.t()
@callback handle_batch(
            batcher :: atom,
            messages :: [Message.t()],
            batch_info :: BatchInfo.t(),
            context :: term
          ) :: [Message.t()]
@callback handle_failed(messages :: [Message.t()], context :: term) :: [Message.t()]

@optional_callbacks prepare_messages: 2, handle_batch: 4, handle_failed: 2

More information about typespecs can be found in the official documentation.

Module metadata processing

While on the topic of meta-programming, module metadata can also be processed.

ensure_loaded? ensures that a given module is loaded. In Broadway, this is used to ensure that the :persistent_term module from Erlang is available for Elixir — the only time it will not be available is when the version of Elixir is too old. Documentation here.

unless Code.ensure_loaded?(:persistent_term) do
  require Logger
  Logger.error("Broadway requires Erlang/OTP 21.3+")
  raise "Broadway requires Erlang/OTP 21.3+"
end

function_exported? returns whether a module contains a definition for a public function with a given arity. Used to execute functions from modules if they are defined. Documentation here.

if Code.ensure_loaded?(producer_mod) and
     function_exported?(producer_mod, :prepare_for_start, 2) do
  case producer_mod.prepare_for_start(module, opts) do
    {child_specs, opts} when is_list(child_specs) ->
      {child_specs, NimbleOptions.validate!(opts, Broadway.Options.definition())}

    other ->
            # ...

Dynamic process naming

As the pipeline can comprise of any number of components, Broadway supports dynamically generated processes. These dynamically generated processes are assigned names that follow a fixed convention — comprising of the name of the pipeline, the process type, and the index of the component among the other components of the same type.

defp process_name(prefix, type, index) do
  :"#{name_prefix(prefix)}.#{type}_#{index}"
end

defp process_names(prefix, type, config) do
  for index <- 0..(config[:concurrency] - 1) do
    process_name(prefix, type, index)
  end
end

The names are returned as quoted atoms where the atom has a space in it so it has to be declared via :"" .

Storage options in Elixir

Besides the basic data structures like lists and dictionaries, Elixir and Erlang offer other ways of storing data within processes.

Atomics

:atomics are a way of performing atomic operations on a set of mutable atomic variables.

Used to maintain the rate limiting threshold.

Previously, the rate limiter used ETS instead but atomic operations are much better for concurrent systems as they avoid race conditions when multiple producer processes are attempting to modify the rate limit.
```
counter = :atomics.new(@atomics_index, [])
:atomics.put(counter, @atomics_index, allowed)
```

Persistent term

Storage for Erlang terms that is optimised for reading terms at the expense of writing and updating terms.

Used to store pipeline metadata like producer names etc.

:persistent_term.put(config.name, %{
  context: config.context,
  producer_names: process_names(config.name, "Producer", config.producer_config),
  batchers_names:
    Enum.map(config.batchers_config, &process_name(config.name, "Batcher", elem(&1, 0))),
  rate_limiter_name:
    config.producer_config[:rate_limiting] && RateLimiter.rate_limiter_name(opts[:name])
})

Queue

Manage first-in, first-out queues.

Used to manage message and demand buffers in the producer.

# A queue of "batches" of messages that we buffered.
message_buffer: :queue.new(),
# A queue of demands (integers) that we buffered.
demand_buffer: :queue.new()

Process dictionaries

Store state within a process directly although its usage is generally frowned upon.

Used to store batches in the batcher.
```
defp init_or_get_batch(batch_key, state) do
  if batch = Process.get(batch_key) do
    batch
  else
        # ...
  end
end

defp put_batch(batch_key, {_, _, _} = batch) do
  Process.put(batch_key, batch)
end
```
A better alternative might have been to use an Agent or ETS instead.

Edit! I clarified with the team about their decision to use process dictionaries over ETS, this was their response:

The correct solution here would be to simply use a map. But because this is very intensive code, we need a mutable option, and the process dictionary is the most efficient one. ETS would be slow as data has to be copied in and out of ETS.
This is one of the very cases where using the pdict for performance is justified. :)

So, the reason why they decided to use a process dictionary over ETS is due to the performance requirement of batching! Very interesting!

Options validation

Dashbit — the team behind Broadway — developed an options validation library called NimbleOptions that aims to be a small library for validating and documenting high-level options.

A set of definitions for the available options are created first and these can be used to validate a keyword list — aka the options.

If the options are invalid, an error is returned, otherwise an :ok status along with the options are returned. The returned options have default values filled in.

Default values in dictionaries

Broadway has an interesting way of fanning out default values for the options keyword list. In the options keyword list, a "parent" value for :partition_by, :hibernate_after, and :spawn_opt is provided.

options = [
    partition_by: ..., # these are parent values
    hibernate_after: ...,
    producer: [
        hibernate_after: ... # this is a child value
    ]
]

The parent value will be used for producers, processors, and batchers if no explicit child value is provided. Alternatively, we might want to fan out a parent value to only two of the three unset child values while maintaining the original child value of the third.

This is done by merging the child options into the parent options. Thus, if the child does not define a value for the option, the parent value is inherited.

opts =
  opts
  |> carry_over_one(:producer, [:hibernate_after, :spawn_opt])
  |> carry_over_many(:processors, [:partition_by, :hibernate_after, :spawn_opt])
  |> carry_over_many(:batchers, [:partition_by, :hibernate_after, :spawn_opt])

defp carry_over_one(opts, key, keys) do
  update_in(opts[key], fn value -> Keyword.merge(Keyword.take(opts, keys), value) end)
end

defp carry_over_many(opts, key, keys) do
  update_in(opts[key], fn list ->
    defaults = Keyword.take(opts, keys)
    for {k, v} <- list, do: {k, Keyword.merge(defaults, v)}
  end)
end

Closing the curtains

To conclude, Broadway is a powerful library for building data processing pipelines. These pipelines are built on top of the robust concurrency system that Elixir boasts.

Broadway is a very versatile library and the documentation contains detailed guides about using it with various data sources. Check out the Github repository and documentation!

If you want to get a basic understanding of the underlying concepts of Broadway or better visualise the architecture of a pipeline in Broadway, check out the first part here!

Open-source Deep Dive is a series where I pick apart open-source projects to explain the underlying concepts that power these projects and share my findings about the project!

Open-source Deep Dive: Hound

Jiahao — Sun, 17 Jan 2021 09:10:30 +0000

What is Hound?

For browser automation and writing integration tests in Elixir

Let's inspect this definition a little closer...

What is browser automation?

Browser automation is effectively the process of using a proxy (like Selenium or Hound) to perform browser actions on behalf of the user (like the test case). Essentially, we are automating the usage of the browser.

It is often associated with illegal applications like sneaker-botting but much like torrenting, there are positive applications and we will be exploring one of them in this post - integration testing.

What is integration testing?

When building software, we first build individual components to support given functional requirements. These individual components can be tested using unit tests - which ensure that given a set of inputs, the component returns a predictable set of outputs (predictable means that the functions tested are pure).

However, while components may work well on their own, when combined with other components (to form larger components/whole systems), unexpected behavior may be exhibited. For instance, the input from component A is transformed before it is used as input to component B, thus, the combined components returns an unexpected result.

Hence, integration tests serve to bridge the gap between individual components testing and full system testing.

When combined with browser automation, we can ensure that a website works end-to-end. We can ensure that the data validation on the front-end works as intended and that the forms submitted by users are properly sent to the back-end and saved in the database.

Approaching browser automation integration testing...

We can take two approaches to browser automation integration testing. We could either

build our own interfacing system to communicate with the browser, or
rely on existing interfacing systems

The former is time-consuming and requires a lot of care during development as we have to account for varying browser APIs and quirks. Thus, it is wiser to chose the latter when approaching browser automation integration testing. Doing so minimizes the number of components we have to manage.

Introducing Hound!

This is where Hound comes into the picture. Hound provides a clean API to build browser automation tests. It relies on Selenium, PhantomJS, and ChromeDriver as the interfacing systems to perform the "dirty" work of coordinating requests/responses to/from the browser.

This introduces a larger question, what exactly is Selenium, PhantomJS, and ChromeDriver? More importantly, in fact,

"How is browser automation performed?"

Understanding how browser automation is performed provides us with a better foundation to grasp how these technologies work and how Hound works under the hood.

The world of web drivers...

The key driving (pun intended) of browser automation is web drivers. But before we can understand what they are, we should establish some basic understanding of what a driver is in general computing terms.

What are drivers?

Drivers are pieces of software that behave as a proxy between a caller and a target. Callers can be something like the print prompt in Google Chrome or a computer peripheral. Targets can be something like the printer or computer.

In general, drivers are responsible for translating the caller's request into a given format that the target can understand.

There may be variations of a caller to the same target so each driver must be able to translate their respective caller's request into a common request format for the target. For instance, there are multiple types of keyboards that can be connected to a single computer but the computer can only understand a single request format. So the respective keyboard drivers are responsible for converting the unique keyboard's requests into the format that the computer accepts.

Back to web drivers

Similar to general drivers, web drivers behave as proxies for the caller (Hound) to communicate with the target (browser). It allows the caller to send instructions for the browser to perform. In the web development world, a browser is also referred to as a user agent.

The Selenium project proposed a W3C specification to guide the development of web drivers. For the rest of this discussion, we will be relying on this specification. The specification can be found here.

According to the specification, there must exist a separation of concern when designing a web driver. More specifically, there are two components to a web driver:

Local end - API for developers to send requests to the browser (libraries like Selenium and Hound)
Remote end - responsible for communicating with the browser, i.e. a browser driver (can you infer what this means?)

In essence, a web driver is comprised of an API and a browser driver. Ideally, the API should be able to work with different browser drivers for different browsers.

The remote end must also provide an HTTP compliant wire protocol where each endpoint maps to a command for the browser.

This means that the remote end relies on HTTP to communicate requests with the browser. The remote end is a HTTP server that the local end writes HTTP requests to. The remote end translates each HTTP request (based on endpoint and method) to a command for the browser. Note that a wire protocol is a method of getting data from one point to another. It dictates that requests should follow a given format.

The specification also provides an outline for the endpoints that the remote end must make available for the local end. This ensures standardization and ease of adoption for future browser drivers.

One advantage to using a HTTP server for the remote end is that it is possible to host the remote end on a remote machine. This means that we can delegate the job of integration testing to another machine, a process commonly known as distributed testing. By enabling distributed testing, the local machine is not burdened with the responsibility of testing potentially extensive and rigorous integration tests which the machine may not support.

So, how does Selenium work?

Selenium implements the web driver specification (they did author it). The remote end uses the JSON Wire Protocol as the HTTP compliant wire protocol to communicate with the browser driver. Note that the documentation provided by Selenium (for the JSON Wire Protocol) has been obsoleted in favor of the one defined in the specification.

How does Selenium differ from PhantomJS?

PhantomJS is a headless browser library. Headless browsers are essentially web browsers without the graphical interfaces. Selenium, on the other hand, is a web driver. The key difference between the two is the way requests are routed and managed (PhantomJS is a rather interesting project so Open-source Deep Dive: PhantomJS edition maybe?).

However, Selenium supports headless browsers as well and more importantly, Selenium is still in active development while PhantomJS has been archived due to a lack of active contributions.

What is ChromeDriver then?

ChromeDriver is a browser driver developed as part of the Chromium project. It is used by Selenium as one of the supported browser drivers. However, Hound supports raw requests to ChromeDriver as the underlying HTTP server works the same with or without the use of Selenium. It is an interesting project so I may explore it in another installment of Open-source Deep Dive!

Hound: Under the hood

Leveraging browser drivers

With a better understanding of how browser automation and web drivers work, we can in fact see that Hound doesn't rely on the entirety of Selenium (including the local end APIs). Instead, it relies on the remote end of the Selenium web driver (along with PhantomJS and ChromeDriver) to minimize the number of "moving components" that need to be managed while reaping the benefits of the existing technologies. Thus, it can focus on delivering a seamless API for developing browser automation integration tests.

Exploring a basic use case

We will inspect a basic use case of Hound before diving into how Hound works.

defmodule HoundTest do
    use ExUnit.Case
    use Hound.Helpers

    hound_session()

    test "the truth", meta do
        navigate_to("https://google.com")
        element = find_element(:class, "search")
        fill_field(element, "Apples")
        submit_element(element)

        assert page_title() == "Apples"
    end
end

There is quite a bit to unpack here. Let's first understand the core of how a test suite with Hound is setup.

First, we use ExUnit.Case as Hound works hand in hand with ExUnit, a built-in Elixir library for developing unit tests. It relies on two components of ExUnit: setup and on_exit. This allows Hound to work as expected.

Then, we use Hound.Helpers which, with the power of macros, imports all helper functions that are required to access the browser session.

Finally, we call hound_session() which creates a new session (an instance of the browser) and initializes the setup and tear down functionality of a Hound browser automation test.

Once the core of the browser automation test is built, we can write test cases as per normal, leveraging on functions like navigate_to() and fill_field() to perform browser actions. The intended behavior of these functions are easy to understand and the documentation for them can be found here.

Breaking it down

With a basic understanding of how a test suite can be setup in Hound, we can start to decompose Hound to better understand what makes it tick.

First, we need to inspect the following file: lib/hound/helpers.ex which houses the Hound.Helpers module, the same one that we use in the example above.

By overriding the __using__ macro, Hound is able to import all of the helper functions into a given file with a single use statement. This helps to minimize the boilerplate for users to get started. Macros are meta programming constructs that inject code during compile-time. More on macros here.

defmacro __using__([]) do
  quote do
    import Hound
    import Hound.Helpers.Cookie
    import Hound.Helpers.Dialog
    import Hound.Helpers.Element
    import Hound.Helpers.Navigation
    import Hound.Helpers.Orientation
    import Hound.Helpers.Page
    import Hound.Helpers.Screenshot
    import Hound.Helpers.SavePage
    import Hound.Helpers.ScriptExecution
    import Hound.Helpers.Session
    import Hound.Helpers.Window
    import Hound.Helpers.Log
    import Hound.Helpers.Mouse
    import Hound.Matchers
    import unquote(__MODULE__)
  end
end

Hound.Helpers also defines the hound_session() function which relies on setup and on_exit() of ExUnit to setup and tear down a session between every test case.

defmacro hound_session(opts \\ []) do
  quote do
    setup do
      Hound.start_session(unquote(opts))
      parent = self()
      on_exit(fn -> Hound.end_session(parent) end)

      :ok
    end
  end
end

Each helper function constructs a HTTP request to the browser driver server using Hackney . For instance, navigate_to - which opens a given URL in the session - creates the following HTTP request:

def navigate_to(url, retries \\ 0) do
  final_url = generate_final_url(url)
  session_id = Hound.current_session_id
  make_req(:post, "session/#{session_id}/url", %{url: final_url}, %{}, retries)
end

Hound rolls its own HTTP request/response management system that supports multiple retries. This can be found in the lib/hound/request_utils.ex file.

We have managed to break down the core functionality of Hound. There are additional interesting components to Hound that I would like to explore as well.

Processes

Applications are started according to standard OTP specification (here). lib/hound.ex starts a link to Hound.Supervisor which initializes two workers: Hound.ConnectionServer and Hound.SessionServer. These are child processes (Hound isn't fully up-to-date with Application convention) that the supervisor manages.

def init([options]) do
  children = [
    worker(Hound.ConnectionServer, [options]),
    worker(Hound.SessionServer, [])
  ]

  supervise(children, strategy: :one_for_one)
end

Let's explore what the connection server and session server are all about next.

More information on processes in Elixir here.

Connection server

This process is responsible for managing the details of the browser driver and providing information to construct the HTTP server endpoints. It stores the driver information using Agent to allow the information to be accessed across processes.

More information on Agent here.

Session management

Sessions, as mentioned earlier, refer to instances of the browser that we want to run our tests on. As Hound supports multiple sessions across different processes, it has rolled a session management system.

Session management in Hound relies on ETS, a built-in storage option provided by Erlang and available in Elixir. When the session server first starts, it creates a new ETS table to hold the session information. This server is setup as a GenServer which allows it to support asynchronous and synchronous callbacks from other processes.

def init(state) do
  :ets.new(@name, [:set, :named_table, :protected, read_concurrency: true])
  {:ok, state}
end

When a session is first created by hound_session(), the current process's ID (by default, it's the main process) is related to the session. The process is monitored and the new session is created. Under the process ID, multiple sessions can be created, thus allowing Hound to support multi-session testing. Each session is identified by an ID. Each session is also assigned a name. By default, we use the session name of :default. The ETS table holds the following information (mapped to JSON for illustration purposes. In reality, ETS tables store tuples of data so the actual data stored does not include any keys, just the values in the given order):

[
    {
        // Process ID
        "pid": ...,
        // Process monitoring ref
        "ref": ...,
        "session_id": ...,
        // Map containing all sessions running
        "session": {
            "<session_name>": "<session_id>",
            ...
        },
    }
]

As you can see, a session name can be assigned to the same session ID, but not the other way around (I am not too sure why this is setup as such, more investigation would be required).

With the ETS table setup to manage session information, we can avoid a major problem: passing around the session ID to various functions. If we had done so, we would have increased the overhead required when using the session ID as functions would have to be designed to accept the session ID and we would have to devise a method of passing the session ID around.

Instead, the session ID is retrieved from the server on demand using the current_session_id() function in lib/hound.ex.

As the current session ID is related to the calling process ID, multiple processes can have different sessions, thus, providing multi-session testing support. This also means that if the calling process changes, the associated session will be retrieved or a new session will be created dynamically.

If a process dies - i.e. Process.monitor() sends a DOWN message - the session server will destroy all associated sessions with that process asynchronously.

def handle_info({:DOWN, ref, _, _, _}, state) do
  if pid = state[ref] do
    destroy_sessions(pid)
  end
  {:noreply, state}
end

Configurations

Configurations are managed using Elixir's Config API which uses keyword parameter lists to manage configurations.

import Config

config :hound,
  browser: "firefox"

The configurations are stored as application environment variables which are retrieved by the connection server.

driver = options[:driver] || Application.get_env(:hound, :driver, "selenium")

More information on the Config API here.

Coding conventions

Other rather interesting bits of Elixir convention that Hound employs are:

^ (pin) operator

The pin operator ensures that a variable, when matched during assignment, is the same as the existing variable of the given name

In Hound, this is used to ensure that the retrieved process ID of session (from the session server) is the same as the given process ID (from argument).

If the retrieved pid does not match the pid argument, an error is raised.
```
def all_sessions_for_pid(pid) do
  case :ets.lookup(@name, pid) do
    [{^pid, _ref, _session_id, all_sessions}] -> all_sessions
    [] -> %{}
  end
end
```
More information on the pin operator here.
defdelegate

defdelegate dictates that a function's underlying behavior is deferred to that of another function in another module.

This allows a module to house the functionality of different modules without breaking the modularity afforded by the module system.

Interestingly, the __using__ override in Hound.Helpers can be replaced with a multitude of defdelegate to the helper functions but it would, understandably, create a lot of confusion.

More information on defdelegate here.
= (match operator) in function parameters

As = is the match operator in Elixir, it can be used to perform pattern matching while assigning the matched pattern to a variable name.

This is very useful when working with structures as you may not want to deconstruct the entire structure while ensuring that arguments follow the given structure.
```
def foo(%User{} = user) do
    IO.puts user[:name]
end
```
More information on the match operator here.

Pattern matching as enums

Pattern matching with atoms can be used as substitutes for typical enum behavior.

An enum in Kotlin may look like:

enum class MatchClause(val name: String) {
    CLASS("class"),
    CSS("css selector"),
    NAME("name"),
    ID("id"),
    ELEM("element")
}

In Elixir, it can be written as such:

def match(:class), do: "class"
def match(:css), do: "css selector"
def match(:name), do: "name"
def match(:id), do: "id"
def match(:elem), do: "element"

Conclusion

To conclude, Hound is a browser automation and integration testing library built on top of web driver - more specifically, browser driver - technologies as it leverages Selenium, PhantomJS, and ChromeDriver to build a highly abstracted and simple to use API for building integration tests.

Under the hood, Hound is an intriguing project that uses fundamental constructs to build powerful internal libraries that support complex operations.

If you are interested in the topics discussed in this post, here are some additional readings:

NOTE: I do not condone the use of browser automation or torrenting for illegal purposes. Any links or discussions about the mentioned subjects are purely for educational purposes and should remain as that.

Open-source Deep Dive is a series where I pick apart open-source projects to explain the underlying concepts that power these projects and share my findings about the project!

DEV Community: Jiahao

Migrating from Hugo Astro

Pitfalls of Metaprogramming in Elixir

Common Perils of Macros

1. Injecting Unnecessary Functions with Macros

2. Macros Over-injecting Behavior

3. Macros Used in Place of Regular Functions

Metaprogramming in Elixir: Further Reading

How to Use Macros in Elixir

Using Macro Wrappers to Extend Elixir Module Behavior

Batch Imports with Macro Wrappers

Macro Wrappers Define Functions to Override

Macro Wrappers Dynamically Generate Functions

Implement Domain-Specific Languages using Macros

Traversing Abstract Syntax Trees (ASTs)

Going Back to the Roots of ASTs

Order of Traversal in ASTs

Prewalk vs. Postwalk Macros

Compile-time Hooks with Macros

Module Attributes as Temporary Storage

Deferring Computation with Macros

Macros in Elixir: A Powerful Tool, If Used Wisely

Under the Hood of Macros in Elixir

Stages of Elixir's Compilation Process

Expansion Phase

Contexts in Macros

Order of Evaluation in Macro and Caller Contexts

Revisiting unquote

Variable Scoping in Macros

Macro Hygiene in Elixir

Understanding require

Escaping Macros in Elixir

Guard Clauses and Pattern Matching

Up Next: Applications of Macros in Elixir

An Introduction to Metaprogramming in Elixir

Introducing Metaprogramming in Elixir

Types of Metaprogramming

Introspection

Reflection

Why Do Metaprogramming?

Defining Run-time and Compile-time

Interesting Uses of Metaprogramming

Code generation

Code instrumentation

Behavioral changes

Metaprogramming in Elixir: The Basics

Defining Macros

Starting Small with Macros

AST Structure

Interacting with ASTs

quote Options in Elixir

Build a Simple Macro in Elixir

Next Up: Macros in Detail

Open-source Deep Dive: Broadway (Part 1) - Message queues, concurrency in Elixir, and Broadway architecture

Act 1, Scene 1

Lights! Camera! Action!

What are message queues?

Concurrency in Elixir

Cue the producer/consumer model

The next frontier (of concurrency): GenStage

Pipeline architecture

How it's made: Producers

How it's made: Processors

Open-source Deep Dive: Broadway (Part 2) - Inner workings of Broadway

What's the scoop?

Rate limiting

Batching

Telemetry

Built-in testing

Graceful shutdowns

Exit signals and process termination

Back to our regularly scheduled deep dive...

Fascinating discovery!

__using__ configurations

Module metadata processing

Dynamic process naming

Storage options in Elixir

Options validation

Default values in dictionaries

Closing the curtains

Revisiting `unquote`

Understanding `require`

`quote` Options in Elixir

`using` configurations