How to abstract away library interfaces working directly through syntax in Rust with procedural macros

In this post I describe how I wrote two macros to separate the bevy_ecs interface from my app core even though bevy_ecs' system interface works directly via the function syntax.
This isn't a complete tutorial for Rust's syn and quote libraries, the Entity Component System pattern or Rust macros. I won't explain every struct that I use. Read the library documentations for that.

You can find my source code here.

general info about procedural macros

In both cases I'm using procedural macros to remain flexible. With bevy_ecs a derive Component macro would probably suffice, but some ECS libraries presumably use another way which would cause problems if I tried to switch to them.

Procedural macros get two token streams (one for the macro arguments like Debug in #[derive(Debug)], one for the block that has the macro) and return one (the code that replaces the code block which is affected by the macro). They look like this:

#[proc_macro_attribute]
fn macro_name (args: TokenStream, input: TokenStream) -> TokenStream {
    //...
}

I haven't used macro arguments yet, so I won't describe them.

To parse the incoming token streams into something usable like a token tree, Rust offers the syn package. For the opposite direction Rust offers the quote package.

To get from the token stream to a token tree, syn has multiple parsing macros. I used parse_macro_input!. The main difference to the others is that parse_macro_input! enforces what type of syntax block you get. E.g. parse_quote! just parses any string.
A call would look like this:

let parsed_input = parse_macro_input!(input as ItemStruct);

In this case my custom macro is expecting a struct. I parse the input parameter to ItemStruct and parse_macro_input! checks if input actually is a struct and returns an ItemStruct instance. ItemStruct is a normal Rust struct provided by syn that can be nicely worked with!

After you've edited your token tree or created a new one you have to create a new token stream that will be the output of the custom macro. There are multiple ways for different scenarios.
The most simple one is using the quote! macro.

let parsed_input = something_that_implements_ToTokens;
let input_struct_identifier = again_something_that_implements_ToTokens;

let new_tokenstream = quote! {
    #parsed_input

    impl #input_struct_identifier {
        fn im_just_here_to_show_what_quote_can_do (a: i32) -> bool {
            a < 100_00
        }
    }
};

In quote! you can directly write Rust syntax and use variables that implement the ToTokens trait (parsed_input and input_struct_identifier in this example). quote! substitutes these variables with their token streams.
The example above could produce a token stream resembling something like this:

struct SomeStruct {
    //...
}

impl SomeStruct {
    fn im_just_here_to_show_what_quote_can_do (a: i32) -> bool {
        a < 100_00
    }
}

However, if you created a struct that already implements the ToToken trait you can directly parse it into a token stream.
E.g. like that:

item_struct.into_token_stream()

component macro

The component macro is rather simple. bevy_ecs uses the same style that I want, so the macro simply adds a #[derive(bevy_ecs::component::Component)].

let expanded = quote! {
    #[derive(bevy_ecs::component::Component)]
    #input
};

There is probably a way to add the derive macro in the ItemStruct instance, but I didn't try that.

system macro

The system macro is far more complex and interesting.
In general, I want my system functions like this:

#[system]
fn some_system (a: &ComponentA, b: &mut ComponentB, entity: Entity) {
    //a,b and entity are used here
}

bevy_ecs on the other hand, uses this style:

fn some_system (query: Query<&ComponentA, &mut ComponentB, Entity>) {
    for (a, mut b, entity) in &mut query {
        //a, b and entity are used here
    }
}

This has quite some syntax overhead that I don't like. Thus, I combined keeping dependencies away from my program with getting an interface more of my liking.

This comparison already shows how the token stream input looks like and how the output is supposed to be.

The macro can be seperated in creating a new function signature and creating the for loop. Then you create a new function that copies all the other properties from the old function and insert the new signature and new function body.
It could look like this:

pub fn create_fn_item (input: ItemFn) -> Result<ItemFn, TokenStream> {
    let new_sig = create_new_sig(&input.sig)?;
    let for_loop_stmts = create_new_loop_stmts(&input)?;

    let new_fn = ItemFn {
        attrs: input.attrs, // attributes like #[repr(transparent)]
        vis: input.vis, // visibility
        sig: new_sig, // signature
        block: Box::new(syn::Block{ // function body
            brace_token: Default::default(),
            stmts: for_loop_stmts
        }),
    };

    return Ok(new_fn);
}

I won't explain the steps for the signature and the for loop in detail because it would mainly be copying source code and writing some redundant comment. My biggest advise is to parse the structures you want and print them to the console. Then look how they are build up and combined with the syn documentation build your wanted syntax structures. You can use my source code as an example.

debugging procedural macros

Currently procedural macros can only be executed in their default way with #macro_name, even though they're just special functions. This means quite some problems for debugging because macros are executed at compile time. There are some debugging options but not a usual IDE debugger.
However, there is a workaround for that. The library proc_macro2. This is a wrapper around the procedural macro API, and it can be used outside a procedural macro. That means you can outsource your macro logic into plain functions and call these from your macro. The actual macro only does the initial syntax parsing, the conversion between proc_macro and proc_macro2 and calls the outsourced function. Because the latter is a plain and normal Rust function, you can unit test it or do whatever else you want.
This is an example how it might look:

#[proc_macro_attribute]
pub fn to_bevy_component(
_args: TokenStream,
input: TokenStream,
) -> TokenStream {
    let input = parse_macro_input!(input as ItemStruct); 

    let result = bevy_component::to_bevy_component(input);

    proc_macro::TokenStream::from(result)
}

pub fn to_bevy_component (input: ItemStruct) -> proc_macro2::TokenStream {
    let expanded = quote! {
        #[derive(bevy_ecs::component::Component)]
        #input
    };

    return expanded;
}

#[cfg(test)]
mod tests {
    //...
}

throwing errors in procedural macros

In case I mess up and use a macro on some invalid syntax thing I wanted error management via Result like usual. This doesn't mean syntactical problems but e.g. some syntax thing that isn't supposed to appear in the syntax things affected by the macro and that I therefore don't want to have to handle.
You can do it quite easily. You just have to create a syn::Error, call .into_compile_error() on it, parse it into a token stream and return it from your macro. For Rust to be able to determine the right spot that caused the error, you have to get the correct span. A span is some part of your parsed code that starts and ends somewhere. Usually you can get it from the syn structs.

I used this technique to exclude self parameters, because systems will never be associated to a struct (or at least don't offer any additional value than normal functions).

 match fn_arg {
        FnArg::Receiver(rec) => Err(
            syn::Error::new(rec.self_token.span, "systems can't have a self parameter")
                .into_compile_error()
        ),
    //...