Utilizing Macros & Annotations in a Web Framework (Part 2)

When I first wrote Utilizing Macros & Annotations in a Web Framework I didn't intend on it being a series. However since its creation, Athena has evolved quite a bit and so has my knowledge and understanding of both macros and annotations. Because of this, I wanted to give an updated look at some of the patterns, approaches, and experiences I used/had with macros and annotations when creating the framework.

Stay tuned for part 3 for a look into how I how annotations and macros to build a compile time service container for Athena's DependencyInjection component.

Registering Routes

In the previous article I mentioned that my ideal syntax would be annotating my controllers with a @[ART::Controller] annotation, then iterate over these controller types to build out the routes. However, after thinking about it more I actually like how it ended up, using an abstract class and iterating over the children of it. This not only makes it a bit simpler to define a controller as you don't have to remember to add the annotation, but also allows defining common methods within that abstract class that could be useful to all controllers. When paired with DI, it also makes it possible to define additional abstract controllers that inherit from the default one that could be reused for endpoints requiring similar dependencies/logic.

I also found a workaround to being able to iterate over types with a specific annotation without the need for a parent type or module. Since all types in Crystal inherit from an Object class, we can combine the all_subclasses and the select methods to get the desired output.

annotation MyAnn; end

@[MyAnn]
class A; end

class B; end

@[MyAnn]
class C; end

{{Object.all_subclasses.select &.annotation MyAnn}} # => [A, C]

Annotation Data at Runtime

One of the main uses of annotations is to store metadata related to a type, method, or instance variable that is accessible at compile time. This metadata could then be consumed at compile time to do various things. However there is not a built in way, such as reflection, to also have access to this data at runtime. A pattern I started using quite a lot combines the record macro, modules, and the macro includedhook.

The idea is that you define a module that defines methods within the including type that exposes an array, or some other data structure, of structs representing the data, most likely instance variables, related to that type. The main benefits of this are:

1. You are using methods, so you are able to access all the instance variables within that type
2. You are using structs, so there is minimal performance overhead
3. You are using modules, so it works for both structs and classes
4. It allows you to use the annotations as a DSL that determine how each struct is instantiated

Lets take a look at an example from Athena's Serializer component. This component defines an ASR::PropertyMetadata struct. The purpose of this struct is pretty clear; it stores metadata related to a property. In other words, it describes the state of an instance variable; including its name, type, class it belongs to, etc. The serializer component also defines the ASR::Serializable module, similar to JSON::Serializable, which defines two main methods: #serialization_properties and .deserialization_properties. Instead of having the module define methods tied to a specific format, the ASR::Serializable module defines generic methods that return arrays of our ASR::PropertyMetadata object. This provides a generic interface that could be consumed in a format agnostic manner.

A example of this concept (cut down for brevity) would look like:

# Define an abstract struct to type our methods with
abstract struct PropertyMetadataBase; end

# Define a struct to store information about each instance variable
record PropertyMetadata(IvarType, ClassType) < PropertyMetadataBase,
  name : String,
  type : IvarType.class = IvarType,
  class : ClassType.class = ClassType

# Define an annotation similar to `JSON::Field`'s `skip` option
annotation Skip; end

# Define our module
module Serializable
  macro included
    # :nodoc:
    def self.deserialization_properties : Array(PropertyMetadataBase)
      {% verbatim do %}
        {% begin %}
          {{@type.instance_vars.reject(&.annotation(Skip)).map do |ivar|
              %(PropertyMetadata(#{ivar.type}, #{@type}).new(
                name: #{ivar.name.stringify}
              )).id
            end}} of PropertyMetadataBase
        {% end %}
      {% end %}
    end
  end
end

# Define a type to test with
record User, id : Int32, name : String, age : Int32 do
  include Serializable

  @[Skip]
  @id : Int32
end

pp Example.deserialization_properties # =>
# [
#   PropertyMetadata(String, User)(@name="name", @type=String, @class=User),
#   PropertyMetadata(Int32, User)(@name="age", @type=Int32, @class=User)
# ]

The main benefit of this approach is it combines the benefits of using annotations, with having runtime types to expose the same data. For example, notice how we reject any instance variable that has the Skip annotation, and that a PropertyMetadata instance isn't created for that property. The logic that consumes the data from this method need not worry about doing that filtering since it all happens at compile time.

However, there is a limitation with this approach. The logic for handling the Skip annotation for example, is hardcoded within the module. Or in other words, there isn't a way to also expose user defined annotations as part of the metadata struct. Mainly because there isn't currently way to get all the annotations on a given type, method, or instance variable (see this issue). But also there isn't a built in way to represent an annotation (and its data) at runtime. After a bit of thinking I managed to figure out a pretty decent workaround; use a record defined by the user to represent the data read off of their custom annotation.

If you read the Property Validation section from my previous article, you would remember how I had to define that hacky ASSERTIONS constant so I knew the types of annotations to read, and what properties I should read off of each. This was because there wasn't a way to get all of the data within an annotation, you had to access each value by name/index using Annotation#[]. Fortunately since that last article, Annotation#args and Annotation#named_args methods have been added (by me 😉 PR). An example implementation of this would look like:

# Define our annotation
annotation MyAnn; end

# Define a record to store the data related to our annotation
record MyAnnConfig, id : Int32, name : String, active : Bool = true

class Foo
  # Apply our annotation to a method,
  # have it return an instance of our struct.
  @[MyAnn(1, name: "Jim")]
  def config : MyAnnConfig
    {% begin %}
      {% ann = @def.annotation MyAnn %}
      MyAnnConfig.new {% unless ann.args.empty? %}{{ann.args.splat}},{% end %}{{ann.named_args.double_splat}}
    {% end %}
  end
end

pp Foo.new.config # => MyAnnConfig(@active=true, @id=1, @name="Jim")

There is quite a few noteworthy things to mention here. The most obvious of which is that we can use default values. Since the active property was not supplied in the annotation, the default value of true was used. We are also able to use both positional and named arguments; positional annotation arguments will be provided as positional arguments and named annotation arguments will be provided as named arguments to the configuration struct.

Another benefit is we get type safety related to the arguments supplied within the annotation. If we defined our annotation as @[MyAnn(1, name: "Jim", unknown_value: 192.1)], it wouldn't compile with the error:

> 1 |
> 2 |
> 3 |       MyAnnConfig.new 1,name: "Jim", unknown_value: 192.1
^--
Error: no overload matches 'w' with types Int32, name: String, unknown_value: Float64

Overloads are:
- MyAnnConfig.new(id : Int32, name : String, active : Bool = true)

It also wouldn't compile if it was missing a value that didn't have a default, or if one of the values was not the correct type. You would also be able to define additional methods on the configuration struct in additional to the standard getters.

We can go one step further to allow for creating our configuration structs more dynamically:

# Define our annotation
annotation MyAnn; end

# Define a record to store the data related to our annotation
record MyAnnConfig, id : Int32, name : String, active : Bool = true

# Define a const to hold our annotation types
CUSTOM_ANNS = [] of Nil

# Define a macro to "register" our types
macro register_ann(name)
  {% CUSTOM_ANNS << name %}
end

# Register our annotation
register_ann MyAnn

class Foo
  @[MyAnn(1, name: "Jim")]
  def config
    {% begin %}
      # Iterate over each of our registered annotations
      # assuming the struct name is ann name + "Config"
      {% for ann_name in CUSTOM_ANNS %}
        {% ann = @def.annotation ann_name.resolve %}
        {{ann_name.id}}Config.new {% unless ann.args.empty? %}{{ann.args.splat}},{% end %}{{ann.named_args.double_splat}}
      {% end %}
    {% end %}
  end
end

pp Foo.new.config # => MyAnnConfig(@active=true, @id=1, @name="Jim")

This does create a naming convention related to our configuration structs, but it's manageable. I think ideally the configuration struct would be merged with the annotation definition. This would centralize things to make the annotation itself the type that is applied and that stores the data at runtime.

I had the idea to apply this concept of exposing user defined annotations at runtime to both the serializer and routing components of Athena. The feature would allow for using more advanced annotation based runtime logic within your ART::Listeners/ART::ParamConverterInterfaces and/or ASR::ExclusionStrategies::ExclusionStrategyInterfaces. Since I wanted this feature in two separate shards, I ended up creating a framework of sorts to make defining, registering, and using these annotation configurations easier as well as share code between the shards.

This has since been released as part of the Athena Config component; which can also be used outside of the Athena ecosystem. The main concepts are:

• ACF.configuration_annotation - Used to both define and register an annotation, but also register the related configuration struct behind the scenes
• ACF::AnnotationConfigurations - Wraps a hash keyed by annotation class that stores the configuration structs related to each annotation applied to a specific type, method, or instance variable

The wrapping type defines a custom #[] method that allows accessing the configuration structs in a type safe way. Athena's serializer component exposes annotation configurations as part of the PropertyMetadata object. Lets combine some of the examples to demonstrate how it all fits together.

# Define an abstract struct to type our methods with
abstract struct PropertyMetadataBase; end

# Define a struct to store information about each instance variable
record PropertyMetadata(IvarType, ClassType) < PropertyMetadataBase,
  name : String,
  annotation_configurations : ACF::AnnotationConfigurations,
  type : IvarType.class = IvarType,
  class : ClassType.class = ClassType

# Define our module
module Serializable
  macro included
    # :nodoc:
    def self.deserialization_properties : Array(PropertyMetadataBase)
      {% verbatim do %}
        {% begin %}
          {{@type.instance_vars.map do |ivar|
              # Use the code from the `ACF::AnnotationConfigurations` type to resolve the applied annotations.
              annotation_configurations = {} of Nil => Nil

              ACF::CUSTOM_ANNOTATIONS.each do |ann_class|
                ann_class = ann_class.resolve
                annotations = [] of Nil

                ivar.annotations(ann_class).each do |ann|
                  pos_args = ann.args.empty? ? "Tuple.new".id : ann.args
                  named_args = ann.named_args.empty? ? "NamedTuple.new".id : ann.named_args

                  annotations << "#{ann_class}Configuration.new(#{ann.args.empty? ? "".id : "#{ann.args.splat},".id}#{ann.named_args.double_splat})".id
                end

                annotation_configurations[ann_class] = "#{annotations} of ACF::AnnotationConfigurations::ConfigurationBase".id unless annotations.empty?
              end

              %(PropertyMetadata(#{ivar.type}, #{@type}).new(
                name: #{ivar.name.stringify},
                annotation_configurations: ACF::AnnotationConfigurations.new(#{annotation_configurations} of ACF::AnnotationConfigurations::Classes => Array(ACF::AnnotationConfigurations::ConfigurationBase)),
              )).id
            end}} of PropertyMetadataBase
        {% end %}
      {% end %}
    end
  end
end

# Define and register our custom annotations
ACF.configuration_annotation MyAnn, id : Int32, active : Bool = true
ACF.configuration_annotation OtherAnn

# Define a type to test with
record User, id : Int32, age : Int32 do
  include Serializable

  @[MyAnn(1)]
  @id : Int32

  @[MyAnn(2, active: false)]
  @age : Int32
end

properties = User.deserialization_properties

properties[0].annotation_configurations[MyAnn].active # => true
properties[1].annotation_configurations[MyAnn].active # => false

properties[0].annotation_configurations[OtherAnn].active # => Error: undefined method 'active' for OtherAnnConfiguration
properties[0].annotation_configurations[OtherAnn]?       # => nil

I added an annotation_configurations instance variable to our PropertyMetadata type, used the code from ACF::AnnotationConfigurations create the annotation configurations instance to provide to the metadata object. I then used the deserialization_properties method to access our configurations, using the annotation_configurations to get a reference to the wrapping type. The #[] method accepts the annotation class you wish to fetch. Notice that you get type safety when using getters for the properties you specified when registering the annotation. #[]? is also defined that returns nil if the specified annotation was not applied. A #has?(type) method is also available that returns true or false depending on if any annotations of the provided type were applied to that property. In the end this abstraction layer provides a common standardized interface to use in both shards, as well as makes the required naming convention of the configuration structs an implementation detail.

This feature greatly increases the flexibility of both the serializer and routing components (or whatever it is implemented within). An example of how this feature could be used is imagine you define IgnoreOnCreate and/or IgnoreOnUpdate configuration annotations. You could then define an exclusion strategy that injects the ART::RequestStore to know what HTTP method the current request is. Logic could then be implemented that would skip properties with the IgnoreOnCreate annotation during POST requests and skip IgnoreOnUpdate properties on PUT requests, while still exposing them on GET requests. On the routing side of things, you could imagine a Pagination or RateLimited annotation used to configure a related event listener.

As usual feel free to join me in the Athena Gitter channel if you have any suggestions, questions, or ideas. I'm also available on Discord (Blacksmoke16#0016) or via Email.