Under the Hood of Macros in Elixir

Welcome back to part two of this series on metaprogramming in Elixir. In part one, we introduced metaprogramming and gave a brief overview of macros.

In this part, we will explore the inner workings and behaviors of macros in more depth.

As discussed in the previous post, macros are compile-time constructs in Elixir. So, before diving into how macros work, it is important to understand where macros lie within Elixir's compilation process.

Stages of Elixir's Compilation Process

We can boil down the Elixir compilation process to the following basic stages:

(Note: the actual compilation process of an Elixir program is more intricate than above.)

The compilation process can be broken down into the following phases:

1. Parsing — The Elixir source code (program) is parsed into an AST, which we will call the initial AST.
2. Expansion — The initial AST is scanned and macro calls are identified. Macros are executed. Their output (AST) is injected and expanded into the callsite. Expansion occurs recursively, and a final AST is generated.
3. Bytecode generation phase — After the final AST is generated, the compiler performs an additional set of operations that eventually generate and execute BEAM VM bytecode.

As you can see, macros sit at the expansion phase, right before code is converted into bytecode. Therefore, a good knowledge of the expansion phase helps us understand how macros work.

Expansion Phase

Let's start by first examining the expansion phase on a general level.

The compiler will expand macros (as per Macro.expand) to become part of the program's pre-generated AST. Macro expansion occurs recursively, meaning that Elixir will continue expanding a macro until it reaches its most fundamental AST form.

As macros expand right before bytecode is generated, they can modify a program's behavior during compile-time.

If we dig a little deeper, we will find that the compiler first injects the output AST of a macro at its callsite. Then the AST is expanded recursively.

We can observe this behavior as follows:

defmodule Foo do
  defmacro foo do
    quote do
      IO.inspect("hello")
    end
  end
end

iex(1)> require Foo
iex(2)> ast = quote do: Foo.foo
{{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true],
 []}
iex(3)> ast |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423391, alias: false], [:IO]},
    :inspect
  ]}, [], ["hello"]}

ast represents the initial AST generated by the compiler before expansion. It holds a reference to the macro Foo.foo but it is not expanded as the macro has not been evaluated yet.

When we call Macro.expand on the given AST, the compiler begins by injecting the behavior of the macro into the callsite. We can expand the AST one step at a time using Macro.expand_once.

Contexts in Macros

Now that we understand the basics of the expansion phase, we can investigate the parts of a macro.

Macros contain two contexts — a macro context and a caller context:

defmacro foo do
  # This is the macro's context, this is executed when the macro is called

  # This is the return value of the macro (AST)
  quote do
    # This is the caller's context, this is executed when the callsite is called
  end
end

As you can see, the macro's context is any expression declared before the quote.

The caller's context is the behavior declared in the quote. The quote generated AST is the macro's output and is injected into and expanded at the callsite.

The behavior defined under the caller's context 'belongs' to the caller, not the module where the macro is defined. The following example, taken from Metaprogramming Elixir, illustrates this:

defmodule Mod do
  defmacro definfo do
    IO.puts "definfo :: Macro's context #{__MODULE__}"

    quote do
      IO.puts "definfo :: Caller's context #{__MODULE__}"

      def friendly_info do
        IO.puts "friend_info :: Module name #{__MODULE__}"
      end
    end
  end
end

defmodule MyModule do
  require Mod
  Mod.definfo
end

iex(1)> c "context.exs"
definfo :: Macro's context Elixir.Mod
definfo :: Caller's context Elixir.MyModule
[Mod, MyModule]
iex(2)> MyModule.friendly_info
friend_info :: Module name Elixir.MyModule
:ok

As you can see, the module that executes the macro's context is Mod. But the module that executes the caller's context is MyModule — the callsite where the macro is injected and expanded.

Similarly, when we declare friendly_info, we inject this function into the callsite of the macro, which is MyModule. So the function now 'belongs' to MyModule.

But why does there need to be two different contexts? What exactly makes the macro context and caller context different from one another?

Order of Evaluation in Macro and Caller Contexts

The key difference between the macro context and the caller context is that behavior is evaluated at different times.

Let's look at an example:

defmodule Foo do
  defmacro foo do
    IO.puts("macro")
    quote do
      IO.puts("caller")
    end
  end
end

iex(1)> require Foo
iex(2)> ast = quote do: Foo.foo
{{:., [], [{:__aliases__, [alias: false], [:Foo]}, :foo]}, [no_parens: true],
 []}

iex(3)> ast |> Macro.expand(__ENV__)
macro
{{:., [],
  [{:__aliases__, [counter: -576460752303423293, alias: false], [:IO]}, :puts]},
 [], ["caller"]}

When we expand the AST of a macro call, the macro context is evaluated and the caller context is injected and expanded into the callsite.

However, we can go a level deeper.

Let's look again at the first component of the expansion phase: Macros are executed. Their output (AST) is injected and expanded into the callsite.

For a compiler to know what AST needs to be injected into the callsite, it has to retrieve the output of the macro during compilation (when the expansion phase occurs). The macro call is parsed as an AST during the parsing phase. The compiler identifies and executes these macro call ASTs prior to the expansion phase.

If we think of macros as regular functions, the macro context is the function body and the caller context is the result of the function. During compilation, a macro is executed and evaluates the macro context. Then the quote is evaluated, returning the results of the function. The caller context is injected and expanded into the callsite of the macro.

The macro context is evaluated during compile-time and treated as a regular function body. It executes within and is 'owned' by its containing module.

The caller context is injected into the callsite, so it is 'owned' by the caller. It is evaluated whenever the callsite is evaluated.

The example above showcases what happens when a macro is invoked at the module level. But what if we attempt to invoke the macro in a function? When do the macro and caller contexts evaluate?

Well, we can use another example here:

defmodule Foo do
  defmacro foo do
    IO.puts("macro")

    quote do
      IO.puts("caller")
    end
  end
end

defmodule Bar do
  require Foo

  def execute do
    IO.puts("execute")
    Foo.foo
  end
end

macro
iex(1)> Bar.execute
execute
caller

The caller context evaluates when the function is called (as we established earlier).

However, something interesting happens with our macro context — it evaluates when the module compiles. This evaluation only happens once when Bar compiles, as evidenced by the lack of "macro" in our output when we call Bar.execute. Why is this the case?

Well, we only need to evaluate the macro once to retrieve its output (caller context) and inject it into the callsite (which is a function in this case). The caller context behavior evaluates every time the function is called.

This difference in the order and time of evaluation helps guide us on when to use the macro and the caller contexts.

We use the macro context when we want the behavior to be evaluated during compile-time. This is regardless of when the caller context is evaluated or where the macro is called in the code.

We use the caller context when we want to invoke behavior injected into the callsite at evaluation.

Now that we have a better grasp of the Elixir compilation process, macros, and the order of evaluation, we can revisit unquote.

Revisiting unquote

In part one of this series, we established that unquote evaluates a given expression and injects the result (as an
AST) into the AST built from quote. This is only a piece of the puzzle.

Let's dig deeper to understand the behavior of unquote during compilation and the necessity of using it.

While the rest of the quote body is evaluated at the same time as the callsite, unquote is evaluated (immediately) during compile-time — when the macro is evaluated. unquote aims to evaluate and inject the result of a given expression. This expression might contain information that is only available during the macro evaluation, including variables that are initialized during this process. unquote must be evaluated during compile-time along with the macro, so that the AST of the result injects into the quote that we build.

But why do we need to unquote the expression to inject it into the AST? To answer this, let's compare the expanded AST of a macro using unquote against one that does not:

defmodule Foo do
  defmacro foo(x) do
    quote do
      IO.inspect(x)
    end
  end
end

defmodule Bar do
  defmacro bar(y) do
    quote do
      IO.inspect(unquote(y))
    end
  end
end

iex(1)> require Foo
iex(2)> require Bar
iex(3)> ast_foo = quote do: Foo.foo(1 + 2 * 3)
iex(4)> ast_bar = quote do: Bar.bar(1 + 2 * 3)
iex(5)> ast_foo |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423448, alias: false], [:IO]},
    :inspect
  ]}, [], [{:x, [counter: -576460752303423448], Foo}]}

iex(6)> ast_bar |> Macro.expand(__ENV__)
{{:., [],
  [
    {:__aliases__, [counter: -576460752303423384, alias: false], [:IO]},
    :inspect
  ]}, [],
 [
   {:+, [context: Elixir, import: Kernel],
    [1, {:*, [context: Elixir, import: Kernel], [2, 3]}]}
 ]}

Observe that the expanded Foo.foo AST is vastly different from the Bar.bar AST even though they are both given the same variable. This is because Elixir is quite literal with variable references. If a variable is referenced without unquote, an AST of that variable reference injects into the AST.

Using unquote ensures that the underlying AST of the variable's value injects into the quote body.

Now you may ask: What is the difference in variable scoping between the evaluation of macros and the execution of the callsite? Why does it matter?

The scoping of variables in macros can be a confusing subject, so let's demystify it.

Variable Scoping in Macros

Now that we understand how macros are evaluated and expanded, we can look at the scoping of variables in macros, and when to use the options unquote and bind_quoted in quote.

Due to function clause scoping, the arguments of a variable are initialized and 'in scope' during the macro evaluation of a function.

Similarly, variables declared and assigned within a function body remain in scope until the function ceases. The same behavior applies to macros.

When the macro context is evaluated, its arguments and any initialized variables are 'in scope.' This is why unquote can evaluate variable references declared as arguments of the macro or any variables initialized in the macro context.

Any evaluation of variable initialization in the caller context will initialize these variables within the callsite during execution.

To understand this difference better, let's look at a few examples:

defmacro foo do
  quote do
    x = 1 + 1
    def bar, do: IO.inspect(unquote(x))
  end
end

In this first example, unquote will not work. The variable x has not yet been initialized, but should have been initialized during the execution of the callsite. The immediate evaluation of unquote runs too early, so we cannot reference our variable x when we need to. When unquote evaluates during compile-time, it attempts to evaluate the variable reference expression of x and finds that it is not in scope.

How can we fix this? By disabling unquoting. This means disabling the immediate evaluation of unquote. We only want unquote to evaluate when our caller context evaluates. This ensures that unquote can properly reference a variable in scope (x) as variable initialization would have occurred
during the evaluation of the callsite.

defmacro foo do
  quote unquote: false do
    x = 1 + 1
    def bar, do: IO.inspect(unquote(x))
  end
end

This example highlights the impact of scoping in macros. If we attempt to access a variable that is available during the evaluation of the macro context, unquote as-is is perfect for us.

However, suppose we try to access a variable that is only available during the evaluation of the callsite. In that case, we must disable the immediate unquoting behavior to initialize variables in scope before unquote attempts to reference them.

Let's apply this understanding to two other examples.

defmacro foo(opts) do
  quote bind_quoted: [opts: opts] do
    x = Keyword.get(opts, :x)
    def bar, do: IO.inspect(unquote(x))
  end
end

In this example, we have initialized the variable x from a keyword list. As the keyword list is initialized during compile-time (along with the evaluation of the macro context), we first have to bind it to the caller context to:

1. Generate an initialization of the variable during the evaluation of the callsite, and
2. Disable unquoting behavior.

We have to bind opts to the caller context, as the variable is no longer in scope during the evaluation of the callsite.

Finally, we have:

defmacro foo(x) do
  quote do
    def bar, do: IO.inspect(unquote(x))
  end
end

In this last example, x remains a variable in scope during the evaluation of the macro context — i.e. when the macro is called. The immediate evaluation of unquote works in our favor. It renders unquote(x) valid, as x is in scope when unquote is evaluated.

Macro Hygiene in Elixir

While we are on the topic of scopes in macros, let's discuss macro hygiene.

According to tutorialspoint.dev:

Hygienic macros are macros whose expansion is guaranteed not to cause the accidental capture of identifiers.

This means that if we inject and expand a macro into the callsite, we need not worry about the macro's variables (defined in the caller context) conflicting with the caller's variables.

Elixir ensures this by maintaining a distinction between a caller variable and macro variable. You can explore this further using an example from the official tutorial.

A macro variable is declared within the body of quote, while a caller variable is declared within the callsite of the macro.

defmodule Foo do
  defmacro change do
    quote do: a = 13
  end
end

defmodule Bar do
  require Foo

  def go do
    a = 1
    Foo.change
    a
  end
end

Bar.go
# => 1

In this example, a referenced in change is not the same variable a in the scope of go. Even when we attempt to change the value of a, the value of a in the scope of go remains untouched.

However, there may be a time where you have to reference a variable from the caller's scope in the macro's scope.
Elixir provides the macro var! to bridge the gap between these two scopes:

defmodule Foo do
  defmacro change do
    quote do: var!(a) = 13
  end
end

defmodule Bar do
  require Foo

  def go do
    a = 1
    Foo.change
    a
  end
end

Bar.go
# => 13

This distinction ensures no unintended changes to a variable due to changes within a macro (whose source code we may not have access to).

You can apply the same hygiene to aliases and imports.

Understanding require

In the code examples shown in this section, we have always used require <module> before we invoke the macros within the module. Why is that?

This is the perfect segue into how the compiler resolves modules containing macros — particularly the order in which modules are compiled.

In Elixir, modules compile in parallel, and usually — for regular modules — the compiler is smart enough to compile dependencies of functions in the proper order of use. The parallel compilation process pauses the compilation of a file until the dependency is resolved. This behavior is replicated when handling modules that contain macro invocations.

However, as macros must be available during compile-time, the module these macros belong to must be compiled beforehand.
Here's where require comes into the picture. require explicitly informs the compiler to compile and load the module containing the macro first.

We can use an example to illustrate this behavior:

defmodule Foo do
  IO.puts("Compiling Foo")

  defmacro foo do
    IO.puts("Foo.foo := macro")
    quote do
      IO.puts("Foo.foo := caller")
    end
  end
end

defmodule Bar do
  IO.puts("Compiling Bar")

  require Foo

  IO.puts("Bar := before Foo.foo")
  Foo.foo()
end

Compiling Foo
Foo.foo := macro
Compiling Bar
Bar := before Foo.foo
Foo.foo := caller

(Note that this is just an approximation of the actual compilation process, but it aims to paint a clearer picture of how require works.)

Let's try to understand why the outputs are in this order.

Firstly, Bar tries to compile. The compiler scans and finds a require for Foo before evaluating any module-level expressions within the module (such as IO.puts). So it pauses the compilation of Bar and compiles the Foo module first. As Foo is compiled, module-level code — like IO.puts — is evaluated, and the compiler prints the first line of the output.

Once Foo is compiled, the compiler returns to Bar to resume compilation. Bar is parsed, macro calls are executed, and the macro context is evaluated. Even though Foo.foo is called after IO.puts("Bar := before Foo.foo"), the evaluation of the macro call takes precedence over the evaluation of module-level code.

During expansion, Foo.foo's caller context is injected and expanded into the callsite in Bar. It then behaves just like a regular module-level function call, printing the last three output lines in order of declaration.

In essence, require instructs the compiler on the order of compilation that each module should go through if there are macro dependencies. This ensures that the macros are available during compile-time.

Escaping Macros in Elixir

Before explaining what Macro.escape does, let's look at an example:

iex(1)> x = {1, 2, 3}
iex(2)> quote do: unquote(x)
{1, 2, 3}
iex(3)> ast = quote do: IO.inspect(unquote(x))
{{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [], [{1, 2, 3}]}
iex(4)> Code.eval_quoted(ast)
** (CompileError) nofile: invalid quoted expression: {1, 2, 3}

Please make sure your quoted expressions are made of valid AST nodes. If you would like to introduce a value into the
AST, such as a four-element tuple or a map, make sure to call Macro.escape/1 before
    (stdlib 3.15) lists.erl:1358: :lists.mapfoldl/3
    (elixir 1.11.3) lib/code.ex:706: Code.eval_quoted/3

That is a strange error. Based on our understanding of unquote and macros, the code should work as intended, but it doesn't. Why is that?

Well, the answer is found on iex(2). When we attempt to unquote x, we return not an AST, but a tuple — the same one initially assigned to x. The error then points to the fact that the tuple is an invalid quoted expression.

When we unquote(x) as-is, we inject a raw tuple into the AST, which cannot be evaluated as it is not a valid AST and throws an error.

So, how do we fix it?

We need to convert the raw tuple referenced by x into a valid AST. This can be achieved by escaping this value using Macro.escape. Let's understand what Macro.escape does:

iex(1)> a = {1, 2, 3}
{1, 2, 3}
iex(2)> Macro.escape(a)
{:{}, [], [1, 2, 3]}
iex(3)> quote do: unquote(a)
{1, 2, 3}
iex(4)> quote do: unquote(Macro.escape(a))
{:{}, [], [1, 2, 3]}

In iex(2), we see that Macro.escape(a) returns an AST of the tuple, not the raw tuple — and this is exactly what we are looking for. By combining Macro.escape's behavior with unquote, we can inject the AST of the tuple into the quote as seen in iex(4).

Let's test this:

iex(1)> x = {1, 2, 3}
{1, 2, 3}
iex(2)> quote do: unquote(Macro.escape(x))
{:{}, [], [1, 2, 3]}
iex(3)> ast = quote do: IO.inspect(unquote(Macro.escape(x)))
{{:., [], [{:__aliases__, [alias: false], [:IO]}, :inspect]}, [],
 [{:{}, [], [1, 2, 3]}]}
iex(4)> Code.eval_quoted(ast)
{1, 2, 3}
{{1, 2, 3}, []}

As you can see, the code works just as intended because we escape the tuple.

Often when working with data structures like tuples and dictionaries, you may find that the injected data from unquote does not inject a valid AST. In these cases, you should use Macro.escape before unquote.

Guard Clauses and Pattern Matching

Finally, it's worth mentioning that, much like regular functions defined through def, macros can use guard clauses and pattern matching:

defmacro foo(x) when x == 4 do
  IO.inspect("macro with x being 4")
end

defmacro foo(_) do
  IO.inspect("macro with any other value")
end

Up Next: Applications of Macros in Elixir

Congratulations, you've made it to the end of this part! You should now have a better grasp of how macros work internally.

In part three, we will look at the many applications of macros in Elixir.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Jia Hao Woo is a developer from the little red dot — Singapore! He loves to tinker with various technologies and has been using Elixir and Go for about a year. Follow his programming journey at his blog and on Twitter.