[Rust Guide] 19.4 Macros

19.4.1 What Are Macros

In Rust, macro is a collective name for a group of related features:

• Declarative macros built with macro_rules!
• Three kinds of procedural macros:
  • Derive macros, used on struct or enum, which let you specify code added through the derive attribute
  • Attribute-like macros, which add custom attributes to any item
  • Function-like macros, which look like function calls and operate on the tokens passed to them as arguments

19.4.2 The Difference Between Functions and Macros

• At a fundamental level, macros are code that generates other code, which is called metaprogramming.
• Functions must declare the number and types of their parameters in their signatures; macros can handle variable numbers of arguments.
• The compiler expands macros before it interprets the code.
• Macro definitions are much more complex than function definitions and are harder to read, understand, and maintain.
• When calling a macro in a file, the macro must already be defined or imported into the current scope. A function can be defined anywhere and used anywhere.

19.4.3 Declarative Macros with macro_rules!

Declarative macros are sometimes called macro templates, sometimes macro_rules macros, and sometimes simply macros.

They are the most common form of macros in Rust. They are somewhat similar to match expression pattern matching, and we use macro_rules! when defining declarative macros.

For example:

#[macro_export]
macro_rules! vec {
    ( $( $x:expr ),* ) => {
        {
            let mut temp_vec = Vec::new();
            $(
                temp_vec.push($x);
            )*
            temp_vec
        }
    };
}

This is a simplified definition of the vec! macro, which is used to create a Vec. Let’s go through it line by line:

• #[macro_export] means this macro can be used only after the crate it belongs to is brought into scope. Without this attribute, the macro cannot be imported into scope.
• macro_rules! is the keyword for declarative macros. The macro’s name is vec, and everything inside the following {} is the macro body.
• The body is somewhat like match pattern matching, in the sense that it looks like branches. In fact, there is only one branch here. Although we say the body is similar to match pattern matching, it differs from match in an essential way: match matches patterns, while macros match Rust code structure.
• ( $( $x:expr ),* ) is its pattern, and what follows is the code. Because there is only one pattern here, any other pattern will cause a compile-time error. More complex macros may contain multiple branches.

First, we wrap the entire pattern in parentheses. Inside the macro system, we use a dollar sign ($) to declare a variable that will contain Rust code matching the pattern. The dollar sign clearly marks this as a macro variable rather than a normal Rust variable. Next comes another set of parentheses, which capture the values matching the pattern inside them so they can be used in the replacement code. Inside $() is $x:expr, which matches any Rust expression and gives it the name $x. * means the pattern can match zero or more of the preceding item.

Suppose we write let v: Vec<u32> = vec![1, 2, 3];—then $x will match 1, 2, and 3 respectively.

Now let’s look at the code body associated with the branch: temp_vec.push() inside $()* is generated for each match of the $() portion in the pattern, zero or more times depending on the number of matches. $x is replaced with each matched expression. When we call this macro with vec![1, 2, 3];, the generated code that replaces the macro call is:

{
    let mut temp_vec = Vec::new();
    temp_vec.push(1);
    temp_vec.push(2);
    temp_vec.push(3);
    temp_vec
}

For more on writing macros, see The Little Book of Rust Macros by Daniel Keep and continued by Lukas Wirth.

Most programmers only use macros and never write them, so we will not go deeper here.

19.4.4 Procedural Macros That Generate Code Based on Attributes

The second kind of macro is a procedural macro. It works more like a function, or some kind of procedure. A procedural macro takes code as input, processes it, and generates code as output, rather than matching patterns and replacing them with other code like a declarative macro does.

There are three kinds of procedural macros:

• Derive macros
• Attribute-like macros
• Function-like macros

When creating a procedural macro, the definition must live in its own crate, and that crate must use a special crate type. This is due to complicated technical reasons. Rust is working toward removing this requirement, but for now it still exists.

For example:

use proc_macro;

#[some_attribute]
pub fn some_name(input: TokenStream) -> TokenStream {
}

• some_attribute is a placeholder used to specify the procedural macro type.
• Below it is the procedural macro function, which takes a TokenStream as input and produces a TokenStream as output. TokenStream is defined in the proc_macro crate. It represents a sequence of tokens, and that is exactly what procedural macros operate on: the source code that needs to be processed becomes the input TokenStream, and the code generated by the macro becomes the output TokenStream. The attribute attached to the function determines which kind of procedural macro we are creating. A single crate can contain multiple kinds of procedural macros.

Derive Macros

Let’s look at an example:

Create a crate named hello_macro, define a HelloMacro trait with an associated function hello_macro, and provide a procedural macro that automatically implements the trait so that users can write #[derive(HelloMacro)] on a type and get the default hello_macro implementation.

First, we need to create a new workspace, and put the other projects under that workspace. Create and open Cargo.toml:

touch Cargo.toml

Write this inside it:

[workspace]

members = [
    "hello_macro",
    "hello_macro_derive",
    "pancakes",
]

First create a library crate:

cargo new hello_macro --lib

In hello_macro/src/lib.rs, write:

pub trait HelloMacro {
    fn hello_macro();
}

This gives us a hello_macro trait and a hello_macro method, but without any concrete implementation.

Then we can implement the trait in main.rs and provide the actual method body:

use hello_macro::HelloMacro;

struct Pancakes;

impl HelloMacro for Pancakes {
    fn hello_macro() {
        println!("Hello, Macro! My name is Pancakes!");
    }
}

fn main() {
    Pancakes::hello_macro();
}

This works, but it has a drawback: if users want many types to use hello_macro, they must write similar code for each type. That is very tedious.

So we want to use a procedural macro to generate the relevant code. Also, because the macro needs to print the type name, that part is variable. For example, if the type is Pancakes, it should print "Hello, Macro! My name is Pancakes!"; if it is Apple, it should print "Hello, Macro! My name is Apple!". Because Rust has no reflection, macros are the only option here.

Procedural macros need their own library, so we create another library crate in the workspace:

cargo new hello_macro_derive --lib

hello_macro_derive is the crate that holds the procedural macro. Putting the hello_macro macro code in hello_macro_derive is the naming convention.

In this crate’s Cargo.toml, add the following content without overwriting the existing content:

[lib]
proc-macro = true

[dependencies]
syn = "2.0"
quote = "1.0"

We will use the syn and quote crates, so add them as dependencies.

Then look at how lib.rs in this crate should be written:

use proc_macro::TokenStream;
use quote::quote;

#[proc_macro_derive(HelloMacro)]
pub fn hello_macro_derive(input: TokenStream) -> TokenStream {
    // Build a syntax tree representation of Rust code
    // that we can manipulate
    let ast = syn::parse(input).unwrap();

    // Build the trait implementation
    impl_hello_macro(&ast)
}

fn impl_hello_macro(ast: &syn::DeriveInput) -> TokenStream {
    let name = &ast.ident;
    let generated = quote! {
        impl HelloMacro for #name {
            fn hello_macro() {
                println!("Hello, Macro! My name is {}!", stringify!(#name));
            }
        }
    };
    generated.into()
}

• Through the compiler interface provided by proc_macro, we can read and operate on Rust code. Because it is built into Rust, we do not need to add it as a dependency.
• The syn crate is used to turn Rust code from text into a data structure that we can further manipulate.
• The quote crate turns the data structure produced by syn back into Rust code.

These three crates make parsing Rust code much easier. Writing a full Rust parser is not simple.

In short, the logic here is:

• The hello_macro_derive function parses the TokenStream
• impl_hello_macro converts the syntax tree (ast)

The code in hello_macro_derive is largely the same for every derive macro; the difference is the impl_hello_macro part. The effect is that when a user writes #[derive(HelloMacro)] on a type, the hello_macro_derive function is invoked automatically.

It is automatically invoked because we used #[proc_macro_derive(HelloMacro)] when defining the macro, and the attribute tells Rust that it applies to the HelloMacro trait.

This function first turns the input TokenStream into a data structure that we can interpret and manipulate. It passes the TokenStream into syn::parse, which outputs a DeriveInput struct representing the parsed Rust code. Using the Pancakes type above as an example, the output should look like this:

DeriveInput {
    // ...

    ident: Ident {
        ident: "Pancakes",
        span: #0 bytes(95..103)
    },
    data: Struct(
        DataStruct {
            struct_token: Struct,
            fields: Unit,
            semi_token: Some(
                Semi
            )
        }
    )
}

Its ident (identifier, meaning name) is Pancakes. The rest is not explained in detail; see the official DeriveInput documentation.

impl_hello_macro is where the final Rust code is generated and returned as TokenStream.

fn impl_hello_macro(ast: &syn::DeriveInput) -> TokenStream {
    let name = &ast.ident;
    let generated = quote! {
        impl HelloMacro for #name {
            fn hello_macro() {
                println!("Hello, Macro! My name is {}!", stringify!(#name));
            }
        }
    };
    generated.into()
}

We use ast.ident to obtain an Ident struct instance containing the name of the annotated type. Using Pancakes as an example, when we run impl_hello_macro on the code in the listing, the ident we get has an ident field whose value is "Pancakes". So the name variable contains an Ident struct instance, which will print as the string "Pancakes".

The quote! macro lets us define the Rust code we want to return. Because the result of quote! cannot be understood directly by the compiler, we need to convert it to TokenStream. We do that by calling into, which takes this intermediate representation and returns the required TokenStream value.

The quote! macro also provides a templating mechanism: we can write #name, and quote! replaces it with the value in name. You can even do repetition in ways similar to ordinary macros. See the official quote documentation.

The stringify! macro is built into Rust. It accepts a Rust expression, such as 1 + 2, but it does not evaluate it. Instead, 1 + 2 is directly converted to the string "1 + 2". That is different from format! or println!, which evaluate the expression and then convert the result into a String. The #name input may be an expression printed literally, so we use stringify!. Using stringify! also saves allocation by turning #name into a string literal at compile time.

After writing all this, compile the two crates (cargo build with the package name works; just be careful about the path, or Cargo will not find the crate). Then create a binary crate in the same workspace:

cargo new pancakes

In the pancakes crate’s Cargo.toml, add the following without overwriting anything else:

[dependencies]
hello_macro = { path = "../hello_macro" }
hello_macro_derive = { path = "../hello_macro_derive" }

Add the hello_macro and hello_macro_derive dependencies.

In pancakes/src/main.rs, write this:

use hello_macro::HelloMacro;
use hello_macro_derive::HelloMacro;

#[derive(HelloMacro)]
struct Pancakes;

fn main() {
    Pancakes::hello_macro();
}

That does the job. Run it and see:

Hello, Macro! My name is Pancakes!

Attribute-Like Macros

Attribute-like macros are also called attribute macros. They are similar to custom derive macros, but instead of generating code for a derive attribute, they let you create new attributes. They are also more flexible: derive only applies to structs and enums, while attributes can also be applied to other items, such as functions.

Here is an example of an attribute-like macro:

There is an attribute named route (representing a route), and it annotates a function when using a web application framework.

#[route(GET, "/")]
fn index() {

This code is only a fragment and is incomplete. It means that if the path is / and the method is Get, the index function will be executed. The route attribute is defined by a procedural macro, and the macro’s function signature looks like this:

#[proc_macro_attribute]
pub fn route(attr: TokenStream, item: TokenStream) -> TokenStream {

There are two TokenStreams as parameters: attr corresponds to (GET, "/"), and item corresponds to the function body, which is the index function.

Aside from that, attribute macros work almost exactly like derive macros. They also require a proc_macro crate and a function that generates the corresponding code.

Function-Like Macros

Function-like macros are also called function macros. They are invoked with the macro_name!(...) call style, similar to macro_rules! macros, but they are more flexible than macro_rules! macros: they take a TokenStream as input and use Rust code in the definition to operate on it, like the other two procedural macro forms.

For example:

let sql = sql!(SELECT * FROM posts WHERE id=1);

This is only a fragment and is incomplete. Suppose we want to define a macro that parses SQL statements, specifically SELECT * FROM posts WHERE id=1. The macro definition could be:

#[proc_macro]
pub fn sql(input: TokenStream) -> TokenStream {

Its signature is also similar to a derive macro: it takes a TokenStream and returns a TokenStream with the required behavior.