DEV Community: Blacksmoke16

oq - A portable/performant jq wrapper Part 2

Blacksmoke16 — Sat, 27 Feb 2021 15:35:03 +0000

OQ Revisited

Over a year ago I published a blog post introducing my jq wrapper oq. Given it has been quite a while since then I decided to rerun the benchmarks I did in the original post with the goal of seeing if things changed, both in regards to oq itself and the greater ecosystem.

Methodology

I tested the latest versions of the packages in the first post. I also added the Go version of yq.

The tested packages, and versions, for the benchmarks include:

jq (1.6) - Installed via pacman -S jq
yq (go) (4.6.1) - Downloaded latest binary from the latest GH Release
yq (python) (2.12.0) - Installed via pip3 install yq
oq (1.2.0) - Built locally with Crystal 0.36.1 via shards build --production --release

Each package was tested against the same 3 benchmarks in my original post, plus an additional benchmark for XML => JSON, mainly since yq (python) has a dedicated command for handling XML input/output that I didn't account for in the first post. The data is gathered via the GNU Time command. E.g. /usr/bin/time -v.

Setup

OS: 5.10.15-1-MANJARO x86_64
CPU: Intel i7-7700k
Memory: 32GB @ 3,000 MHz
SSD: Samsung 970 EVO Plus - 1TB

Results

This section summarizes the results, with key metrics in table form. See the appendix for the raw benchmark data.

Simple

Simply doing a pass through JSON => JSON then counting the lines on a pretty small input file:

{
  "guests": [
    {
      "name": "Jim",
      "age": 17,
      "numbers": [
        1,
        2,
        3
      ]
    },
    {
      "name": "Bob",
      "age": 51,
      "numbers": [
        4,
        5,
        6
      ]
    },
    {
      "name": "Susan",
      "age": 85,
      "numbers": [
        7,
        8,
        9
      ]
    }
  ]
}

Name	Max Memory (MB)	Wall Clock Time (seconds)
jq	3.04	0.02
yq (go)	6.02	0.01
yq (python)	14.728	0.07
oq	6.536	0.07

All packages performed relatively equally, nothing noteworthy.

Jeopardy.json

Getting the length of a 53MB JSON file.

The file used: jeopardy.json.

Name	Max Memory (MB)	Wall Clock Time (seconds)
jq	230.32	0.66
yq (go)	705.292	2.54
yq (python)	2,922.996	108.89
oq	230.304	0.74

This test resulted in some more useful data, causing the true performance of each package to be more obvious. jq and oq fared the best, being pretty comparable in both memory usage and execution time. yq (go) was the next best, using 3x the memory and took ~3.5x longer than oq. yq (python) was by far the worse; using 12.7x the memory and taking 147x longer than oq.

YAML => XML/JSON

Consuming a ~57MB YAML file, and outputting XML and JSON.

The file used: sde/bsd/invItems.yaml from the EVE Online SDE Export.

Name	Max Memory (MB)	Wall Clock Time (seconds)
jq	N/A	N/A
yq (go) - JSON	5,202.032	32.77
yq (python) - JSON	5,742.736	206.2
yq (python) - XML	5,742.704	217.17
oq - JSON	575.6	14.01
oq - JSON - SimpleYAML	334.632	14.16
oq - XML	649.984	15.77
oq - XML - SimpleYAML	334.504	15.83

The final test showed similar metrics as the previous one. jq doesn't support non JSON formats so it was excluded. oq did the best with the execution time being fairly stable, while the memory usage was halved when using the new SimpleYAML format for the input. Interestingly yq (go) fared better execution wise in this test, being only 2.34x slower, but using ~15.5x more memory. yq (python) once again was the worse, with memory usage slightly worse than yq (go), but with an execution time of 15.5x slower than oq.

XML => JSON

Consume the output of the last benchmark, and convert it back to JSON. This test is mainly to not be biased as yq (python) includes a dedicated sub-package xq for handling XML input/output.

Name	Max Memory (MB)	Wall Clock Time (seconds)
jq	N/A	N/A
yq (go)	N/A	N/A
yq (python)	795.06	15.59
oq	2600.964	20.28

Neither jq or yq (go) support XML input, so they were excluded. The dedicated sub-package paid off for yq (python), beating oq by a fair margin in both memory consumption and execution time. It looks like it's able to stream the XML input, while XML input for oq is the only input format that cannot be streamed (yet).

Conclusion

It seems the performance of yq (python) has gotten a bit better since last time. Shaving off over 2GB of memory and 2 minutes of execution time in the jeopardy.json benchmark. However, it still is by far the slowest, and most memory hungry package (when not using XML as the input format). yq (go) is in a better spot, but still has pretty high memory usage depending on the exact use case. oq seems to still be the most efficient overall, with both memory and execution time being on par with jq. Thanks to its focus on streaming of data when possible (all but XML input at the moment); oq is able to use much less memory comparatively.

Appendix

Simple

jq

Command being timed: "jq . data.json | wc -l"
User time (seconds): 0.02
System time (seconds): 0.00
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.02
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3040
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 288
Voluntary context switches: 1
Involuntary context switches: 0
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31

yq (go)

Command being timed: "./go-yq -Pj e . data.json | wc -l"
User time (seconds): 0.01
System time (seconds): 0.00
Percent of CPU this job got: 105%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.01
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 6020
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 617
Voluntary context switches: 76
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31

Using the -j and -P options to replicate the default behavior of the other binaries.

yq (python)

Command being timed: "yq . data.json | wc -l"
User time (seconds): 0.06
System time (seconds): 0.00
Percent of CPU this job got: 87%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 14728
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 4
Minor (reclaiming a frame) page faults: 3637
Voluntary context switches: 43
Involuntary context switches: 1
Swaps: 0
File system inputs: 1328
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31

oq

Command being timed: "oq . data.json | wc -l"
User time (seconds): 0.06
System time (seconds): 0.01
Percent of CPU this job got: 104%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 6536
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 808
Voluntary context switches: 58
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31

Jeopardy.json

jq

216930
Command being timed: "jq length jeopardy.json"
User time (seconds): 0.59
System time (seconds): 0.06
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.66
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 230320
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 59454
Voluntary context switches: 1
Involuntary context switches: 26
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

yq (go)

216930
Command being timed: "./go-yq e length jeopardy.json"
User time (seconds): 3.41
System time (seconds): 0.22
Percent of CPU this job got: 143%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.54
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 705292
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 176999
Voluntary context switches: 787
Involuntary context switches: 163
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

yq (python)

216930
Command being timed: "yq length jeopardy.json"
User time (seconds): 108.62
System time (seconds): 0.86
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:48.89
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2922996
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 816903
Voluntary context switches: 13512
Involuntary context switches: 586
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

oq

216930
Command being timed: "oq length jeopardy.json"
User time (seconds): 0.70
System time (seconds): 0.12
Percent of CPU this job got: 110%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.74
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 230304
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 59966
Voluntary context switches: 13569
Involuntary context switches: 7
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

YAML => XML/JSON

jq

N/A

yq (go)

JSON

Command being timed: "./go-yq -Pj e . invItems.yaml > invItems.go-yq.json"
User time (seconds): 47.66
System time (seconds): 1.46
Percent of CPU this job got: 149%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5202032
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1395040
Voluntary context switches: 37244
Involuntary context switches: 1922
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

yq (python)

JSON

Command being timed: "yq . invItems.yaml > invItems.python-yq.json"
User time (seconds): 205.35
System time (seconds): 1.82
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:26.20
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5742736
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1587436
Voluntary context switches: 13384
Involuntary context switches: 1836
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

XML

Command being timed: "yq -s -x --xml-root items --xml-dtd {"item": .[] | .} invItems.yaml > invItems.python-yq.xml"
User time (seconds): 215.17
System time (seconds): 2.03
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:37.17
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5742704
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1683522
Voluntary context switches: 32842
Involuntary context switches: 1031
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

oq

JSON

Command being timed: "./oq -i yaml . invItems.yaml > invItems.oq.json"
User time (seconds): 12.90
System time (seconds): 12.24
Percent of CPU this job got: 179%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.01
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 575600
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 231391
Voluntary context switches: 289799
Involuntary context switches: 166
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Command being timed: "./oq -i simpleyaml . invItems.yaml > invItems.oq.simple.json"
User time (seconds): 12.27
System time (seconds): 12.18
Percent of CPU this job got: 172%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.16
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 334632
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 89229
Voluntary context switches: 2981745
Involuntary context switches: 246
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

XML

Command being timed: "./oq -i yaml -o xml --xml-root items . invItems.yaml"
User time (seconds): 16.00
System time (seconds): 12.04
Percent of CPU this job got: 177%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 649984
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 249933
Voluntary context switches: 381174
Involuntary context switches: 266
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Example output:

<?xml version="1.0" encoding="utf-8"?>
<items>
  <item>
    <flagID>0</flagID>
    <itemID>0</itemID>
    <locationID>0</locationID>
    <ownerID>0</ownerID>
    <quantity>-1</quantity>
    <typeID>0</typeID>
  </item>
  <item>
    <flagID>0</flagID>
    <itemID>1</itemID>
    <locationID>0</locationID>
    <ownerID>0</ownerID>
    <quantity>-1</quantity>
    <typeID>0</typeID>
  </item>
  ...
</items>

Command being timed: "oq -i simpleyaml -o xml --xml-root items . invItems.yaml"
User time (seconds): 15.27
System time (seconds): 11.99
Percent of CPU this job got: 172%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.83
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 334504
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 89233
Voluntary context switches: 3029485
Involuntary context switches: 298
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

XML => JSON

jq

N/A

yq (go)

N/A

yq (python)

Command being timed: "xq .items invItems.python-yq.xml > invItems.python-yq.xml.json"
User time (seconds): 16.37
System time (seconds): 0.39
Percent of CPU this job got: 107%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.59
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 795060
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 341577
Voluntary context switches: 14927
Involuntary context switches: 87
Swaps: 0
File system inputs: 0
File system outputs: 168064
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

oq

Command being timed: "./oq -i xml .items invItems.oq.xml > invItems.oq.xml.json"
User time (seconds): 20.25
System time (seconds): 16.87
Percent of CPU this job got: 183%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:20.28
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2600964
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 766666
Voluntary context switches: 2117666
Involuntary context switches: 1249
Swaps: 0
File system inputs: 0
File system outputs: 168064
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Utilizing Macros & Annotations in a Web Framework (Part 2)

Blacksmoke16 — Mon, 27 Jul 2020 17:26:16 +0000

When I first wrote Utilizing Macros & Annotations in a Web Framework I didn't intend on it being a series. However since its creation, Athena has evolved quite a bit and so has my knowledge and understanding of both macros and annotations. Because of this, I wanted to give an updated look at some of the patterns, approaches, and experiences I used/had with macros and annotations when creating the framework.

Stay tuned for part 3 for a look into how I how annotations and macros to build a compile time service container for Athena's DependencyInjection component.

Registering Routes

In the previous article I mentioned that my ideal syntax would be annotating my controllers with a @[ART::Controller] annotation, then iterate over these controller types to build out the routes. However, after thinking about it more I actually like how it ended up, using an abstract class and iterating over the children of it. This not only makes it a bit simpler to define a controller as you don't have to remember to add the annotation, but also allows defining common methods within that abstract class that could be useful to all controllers. When paired with DI, it also makes it possible to define additional abstract controllers that inherit from the default one that could be reused for endpoints requiring similar dependencies/logic.

I also found a workaround to being able to iterate over types with a specific annotation without the need for a parent type or module. Since all types in Crystal inherit from an Object class, we can combine the all_subclasses and the select methods to get the desired output.

annotation MyAnn; end

@[MyAnn]
class A; end

class B; end

@[MyAnn]
class C; end

{{Object.all_subclasses.select &.annotation MyAnn}} # => [A, C]

Annotation Data at Runtime

One of the main uses of annotations is to store metadata related to a type, method, or instance variable that is accessible at compile time. This metadata could then be consumed at compile time to do various things. However there is not a built in way, such as reflection, to also have access to this data at runtime. A pattern I started using quite a lot combines the record macro, modules, and the macro includedhook.

The idea is that you define a module that defines methods within the including type that exposes an array, or some other data structure, of structs representing the data, most likely instance variables, related to that type. The main benefits of this are:

You are using methods, so you are able to access all the instance variables within that type
You are using structs, so there is minimal performance overhead
You are using modules, so it works for both structs and classes
It allows you to use the annotations as a DSL that determine how each struct is instantiated

Lets take a look at an example from Athena's Serializer component. This component defines an ASR::PropertyMetadata struct. The purpose of this struct is pretty clear; it stores metadata related to a property. In other words, it describes the state of an instance variable; including its name, type, class it belongs to, etc. The serializer component also defines the ASR::Serializable module, similar to JSON::Serializable, which defines two main methods: #serialization_properties and .deserialization_properties. Instead of having the module define methods tied to a specific format, the ASR::Serializable module defines generic methods that return arrays of our ASR::PropertyMetadata object. This provides a generic interface that could be consumed in a format agnostic manner.

A example of this concept (cut down for brevity) would look like:

# Define an abstract struct to type our methods with
abstract struct PropertyMetadataBase; end

# Define a struct to store information about each instance variable
record PropertyMetadata(IvarType, ClassType) < PropertyMetadataBase,
  name : String,
  type : IvarType.class = IvarType,
  class : ClassType.class = ClassType

# Define an annotation similar to `JSON::Field`'s `skip` option
annotation Skip; end

# Define our module
module Serializable
  macro included
    # :nodoc:
    def self.deserialization_properties : Array(PropertyMetadataBase)
      {% verbatim do %}
        {% begin %}
          {{@type.instance_vars.reject(&.annotation(Skip)).map do |ivar|
              %(PropertyMetadata(#{ivar.type}, #{@type}).new(
                name: #{ivar.name.stringify}
              )).id
            end}} of PropertyMetadataBase
        {% end %}
      {% end %}
    end
  end
end

# Define a type to test with
record User, id : Int32, name : String, age : Int32 do
  include Serializable

  @[Skip]
  @id : Int32
end

pp Example.deserialization_properties # =>
# [
#   PropertyMetadata(String, User)(@name="name", @type=String, @class=User),
#   PropertyMetadata(Int32, User)(@name="age", @type=Int32, @class=User)
# ]

The main benefit of this approach is it combines the benefits of using annotations, with having runtime types to expose the same data. For example, notice how we reject any instance variable that has the Skip annotation, and that a PropertyMetadata instance isn't created for that property. The logic that consumes the data from this method need not worry about doing that filtering since it all happens at compile time.

However, there is a limitation with this approach. The logic for handling the Skip annotation for example, is hardcoded within the module. Or in other words, there isn't a way to also expose user defined annotations as part of the metadata struct. Mainly because there isn't currently way to get all the annotations on a given type, method, or instance variable (see this issue). But also there isn't a built in way to represent an annotation (and its data) at runtime. After a bit of thinking I managed to figure out a pretty decent workaround; use a record defined by the user to represent the data read off of their custom annotation.

If you read the Property Validation section from my previous article, you would remember how I had to define that hacky ASSERTIONS constant so I knew the types of annotations to read, and what properties I should read off of each. This was because there wasn't a way to get all of the data within an annotation, you had to access each value by name/index using Annotation#[]. Fortunately since that last article, Annotation#args and Annotation#named_args methods have been added (by me 😉 PR). An example implementation of this would look like:

# Define our annotation
annotation MyAnn; end

# Define a record to store the data related to our annotation
record MyAnnConfig, id : Int32, name : String, active : Bool = true

class Foo
  # Apply our annotation to a method,
  # have it return an instance of our struct.
  @[MyAnn(1, name: "Jim")]
  def config : MyAnnConfig
    {% begin %}
      {% ann = @def.annotation MyAnn %}
      MyAnnConfig.new {% unless ann.args.empty? %}{{ann.args.splat}},{% end %}{{ann.named_args.double_splat}}
    {% end %}
  end
end

pp Foo.new.config # => MyAnnConfig(@active=true, @id=1, @name="Jim")

There is quite a few noteworthy things to mention here. The most obvious of which is that we can use default values. Since the active property was not supplied in the annotation, the default value of true was used. We are also able to use both positional and named arguments; positional annotation arguments will be provided as positional arguments and named annotation arguments will be provided as named arguments to the configuration struct.

Another benefit is we get type safety related to the arguments supplied within the annotation. If we defined our annotation as @[MyAnn(1, name: "Jim", unknown_value: 192.1)], it wouldn't compile with the error:

> 1 |
> 2 |
> 3 |       MyAnnConfig.new 1,name: "Jim", unknown_value: 192.1
^--
Error: no overload matches 'w' with types Int32, name: String, unknown_value: Float64

Overloads are:
- MyAnnConfig.new(id : Int32, name : String, active : Bool = true)

It also wouldn't compile if it was missing a value that didn't have a default, or if one of the values was not the correct type. You would also be able to define additional methods on the configuration struct in additional to the standard getters.

We can go one step further to allow for creating our configuration structs more dynamically:

# Define our annotation
annotation MyAnn; end

# Define a record to store the data related to our annotation
record MyAnnConfig, id : Int32, name : String, active : Bool = true

# Define a const to hold our annotation types
CUSTOM_ANNS = [] of Nil

# Define a macro to "register" our types
macro register_ann(name)
  {% CUSTOM_ANNS << name %}
end

# Register our annotation
register_ann MyAnn

class Foo
  @[MyAnn(1, name: "Jim")]
  def config
    {% begin %}
      # Iterate over each of our registered annotations
      # assuming the struct name is ann name + "Config"
      {% for ann_name in CUSTOM_ANNS %}
        {% ann = @def.annotation ann_name.resolve %}
        {{ann_name.id}}Config.new {% unless ann.args.empty? %}{{ann.args.splat}},{% end %}{{ann.named_args.double_splat}}
      {% end %}
    {% end %}
  end
end

pp Foo.new.config # => MyAnnConfig(@active=true, @id=1, @name="Jim")

This does create a naming convention related to our configuration structs, but it's manageable. I think ideally the configuration struct would be merged with the annotation definition. This would centralize things to make the annotation itself the type that is applied and that stores the data at runtime.

I had the idea to apply this concept of exposing user defined annotations at runtime to both the serializer and routing components of Athena. The feature would allow for using more advanced annotation based runtime logic within your ART::Listeners/ART::ParamConverterInterfaces and/or ASR::ExclusionStrategies::ExclusionStrategyInterfaces. Since I wanted this feature in two separate shards, I ended up creating a framework of sorts to make defining, registering, and using these annotation configurations easier as well as share code between the shards.

This has since been released as part of the Athena Config component; which can also be used outside of the Athena ecosystem. The main concepts are:

ACF.configuration_annotation - Used to both define and register an annotation, but also register the related configuration struct behind the scenes
ACF::AnnotationConfigurations - Wraps a hash keyed by annotation class that stores the configuration structs related to each annotation applied to a specific type, method, or instance variable

The wrapping type defines a custom #[] method that allows accessing the configuration structs in a type safe way. Athena's serializer component exposes annotation configurations as part of the PropertyMetadata object. Lets combine some of the examples to demonstrate how it all fits together.

# Define an abstract struct to type our methods with
abstract struct PropertyMetadataBase; end

# Define a struct to store information about each instance variable
record PropertyMetadata(IvarType, ClassType) < PropertyMetadataBase,
  name : String,
  annotation_configurations : ACF::AnnotationConfigurations,
  type : IvarType.class = IvarType,
  class : ClassType.class = ClassType

# Define our module
module Serializable
  macro included
    # :nodoc:
    def self.deserialization_properties : Array(PropertyMetadataBase)
      {% verbatim do %}
        {% begin %}
          {{@type.instance_vars.map do |ivar|
              # Use the code from the `ACF::AnnotationConfigurations` type to resolve the applied annotations.
              annotation_configurations = {} of Nil => Nil

              ACF::CUSTOM_ANNOTATIONS.each do |ann_class|
                ann_class = ann_class.resolve
                annotations = [] of Nil

                ivar.annotations(ann_class).each do |ann|
                  pos_args = ann.args.empty? ? "Tuple.new".id : ann.args
                  named_args = ann.named_args.empty? ? "NamedTuple.new".id : ann.named_args

                  annotations << "#{ann_class}Configuration.new(#{ann.args.empty? ? "".id : "#{ann.args.splat},".id}#{ann.named_args.double_splat})".id
                end

                annotation_configurations[ann_class] = "#{annotations} of ACF::AnnotationConfigurations::ConfigurationBase".id unless annotations.empty?
              end

              %(PropertyMetadata(#{ivar.type}, #{@type}).new(
                name: #{ivar.name.stringify},
                annotation_configurations: ACF::AnnotationConfigurations.new(#{annotation_configurations} of ACF::AnnotationConfigurations::Classes => Array(ACF::AnnotationConfigurations::ConfigurationBase)),
              )).id
            end}} of PropertyMetadataBase
        {% end %}
      {% end %}
    end
  end
end

# Define and register our custom annotations
ACF.configuration_annotation MyAnn, id : Int32, active : Bool = true
ACF.configuration_annotation OtherAnn

# Define a type to test with
record User, id : Int32, age : Int32 do
  include Serializable

  @[MyAnn(1)]
  @id : Int32

  @[MyAnn(2, active: false)]
  @age : Int32
end

properties = User.deserialization_properties

properties[0].annotation_configurations[MyAnn].active # => true
properties[1].annotation_configurations[MyAnn].active # => false

properties[0].annotation_configurations[OtherAnn].active # => Error: undefined method 'active' for OtherAnnConfiguration
properties[0].annotation_configurations[OtherAnn]?       # => nil

I added an annotation_configurations instance variable to our PropertyMetadata type, used the code from ACF::AnnotationConfigurations create the annotation configurations instance to provide to the metadata object. I then used the deserialization_properties method to access our configurations, using the annotation_configurations to get a reference to the wrapping type. The #[] method accepts the annotation class you wish to fetch. Notice that you get type safety when using getters for the properties you specified when registering the annotation. #[]? is also defined that returns nil if the specified annotation was not applied. A #has?(type) method is also available that returns true or false depending on if any annotations of the provided type were applied to that property. In the end this abstraction layer provides a common standardized interface to use in both shards, as well as makes the required naming convention of the configuration structs an implementation detail.

This feature greatly increases the flexibility of both the serializer and routing components (or whatever it is implemented within). An example of how this feature could be used is imagine you define IgnoreOnCreate and/or IgnoreOnUpdate configuration annotations. You could then define an exclusion strategy that injects the ART::RequestStore to know what HTTP method the current request is. Logic could then be implemented that would skip properties with the IgnoreOnCreate annotation during POST requests and skip IgnoreOnUpdate properties on PUT requests, while still exposing them on GET requests. On the routing side of things, you could imagine a Pagination or RateLimited annotation used to configure a related event listener.

As usual feel free to join me in the Athena Gitter channel if you have any suggestions, questions, or ideas. I'm also available on Discord (Blacksmoke16#0016) or via Email.

Athena 0.9.0

Blacksmoke16 — Sat, 13 Jun 2020 01:18:42 +0000

Athena 0.9.0

With Crystal 0.35.0 finally released, I'm happy to also announce the release of Athena 0.9.0!

This release focused on further refining the overall foundation of the framework. This includes:

A new Getting Started section
An overhaul of the DI framework
A refactor of how controller action arguments are resolved
A refactor of how param converters work
Compile time route collision detection
Some additional minor QoL improvements

See the linked types for more details/examples.

DI Overhaul

Athena 0.9.0 comes bundled with Athena Dependency Injection 0.2.0, which overhauls the implementation of the service container; making the usage of it simpler, more feature rich, easier to maintain, and more robust.

In version 0.1.x, services were required to include the ADI::Service module, as well as specify any service dependencies within the ADI::Register annotation as position arguments. This is no longer required as dependencies are now resolved automatically based on type restrictions. The module is also no longer required.

NOTE: I'm using records for brevity, a struct service behaves differently than a class service. Be sure to pick the right one for your use case. See the docs for more details.

# Before
@[ADI::Register]
record ServiceOne do
  include ADI::Service
end

@[ADI::Register("@my_service", true)]
record ServiceTwo, service : ServiceOne, debug : Bool do
  include ADI::Service
end

# After
@[ADI::Register]
record ServiceOne

@[ADI::Register(_debug: true)]
record ServiceTwo, service : ServiceOne, debug : Bool

By default, only services can be auto resolved, however ADI.bind can be use to allow auto resolving Scalar Arguments and Tagged Services. Service Aliases can be used to define a "default" service for a given interface. Service dependencies can also be declared as optional or even based on Generic Services.

Action Handling Refactor

The logic that resolves a specific action argument has been decoupled from the logic that resolves the arguments for an action. This is mainly a behind the scenes change, but does come with some user facing changes. The main one being the request's attributes. Prior versions of Athena exposed a Hash on the HTTP::Request object that could be used to store arbitrary data specific to the request's life-cycle. This has been replaced with ART::ParameterBag; the concept is the same, but the API is better and is more robust.

The ART::ParameterBag is also where the path/query parameters are stored. In fact, any value stored within it is able to be automatically provided to the controller action. This is accomplished via ART::Arguments::Resolvers::RequestAttribute which looks for a value in the bag with the same name as the controller argument. Custom resolves can also be defined by creating an implementation of ART::Arguments::Resolvers::ArgumentValueResolverInterface.

Param Converter Refactor

ART::ParamConverterInterfaces have undergone a rewrite. The API has changed to no longer return the converted value. It should instead, most commonly, be stored in the request's attributes. Param converters are now also services themselves; this allows using DI to supply any require dependencies needed for the conversion process. The concept of ART::ParamConverterInterface::ConfigurationInterface has also been introduced to allow defining extra configuration data that should be read from the ART::ParamConverter annotation.

require "athena"

@[ADI::Register]
struct MultiplyConverter < ART::ParamConverterInterface
  configuration by : Int32

  # :inherit:
  def apply(request : HTTP::Request, configuration : Configuration) : Nil
    arg_name = configuration.name
    return unless request.attributes.has? arg_name
    value = request.attributes.get arg_name, Int32
    request.attributes.set arg_name, value * configuration.by, Int32
  end
end

class ParamConverterController < ART::Controller
  @[ART::Get(path: "/multiply/:num")]
  @[ART::ParamConverter("num", converter: MultiplyConverter, by: 4)]
  def multiply(num : Int32) : Int32
    num
  end
end

ART.run

# GET /multiply/3 # => 12

A built in ART::TimeConverter that converts a date(time) string into a Time instance has also been added.

Route Collision Detection

Athena will now raise a compile time error if two routes share the same path; either statically, or with path arguments in the same locations.

class TestController < ART::Controller
  @[ART::Get(path: "some/path/:id")]
  def action1(id : Int64) : Int64
    id
  end
end

class OtherController < ART::Controller
  @[ART::Get(path: "some/path/:id")]
  def action2(id : Int64) : Int64
    id
  end
end

ART.run

# #=> Route action OtherController#action2's path "/some/path/:id" conflicts with TestController#action1's path "/some/path/:id".

Other Minor Improvements

Introduced the concept of ART::Response::Writer to control how the response content is written to the response IO
Allow ART::Response to write directly to the response IO
Add some additional ART::Exceptions types, and make defining custom types easier

As usual feel free to join me in the Athena Gitter channel if you have any suggestions, questions, or ideas. I'm also available on Discord (Blacksmoke16#0016) or via Email.

The Rebirth of Athena

Blacksmoke16 — Tue, 11 Feb 2020 19:46:35 +0000

Rebirth

The past year has been quite the journey for Athena. It felt like not too long ago when I released version 0.1.0, however now, after a lot of thinking, multiple iterations, and 7 minor versions later, I'm proud to announce the release of 0.8.0, or what I like to call Rebirth. This release brings about changes to nearly every part of Athena, both user facing and internally, as well as changes to the vision and organizational future of the Athena Framework itself.

Since its been a while since my last post, I thought I would take this opportunity to highlight some of the changes introduced in this latest version as well as give an update on the roadmap/vision for the framework as a whole. This blog post isn't intended to be a tutorial on how to use these features, I will however provide plenty of links out to examples or documentation either in the Blog Demo App/Blog Post or the API docs. Also, feel free to join me in the Athena Gitter channel.

Organizational

As part of this latest release, I took the time to make some organizational changes; this resulted in the creation of the athena-framework Github organization, which will be the home to all of the frameworks components. Related to this, I moved some of the components that were included in athena core into the organization; namely dependency-injection and config. The main benefit of this is it allows other shards, outside of the Athena Framework to use each component independently from one another. The longer term goal is creating a common set of useful components that can be used by others, versus everyone creating a slightly different version of the same thing.

0.8.0 also removed the direct dependencies on crylog and CrSerializer. These, along with assert are going to be reworked to be more DI friendly, as well as work out some issues with their implementations I found along the way. They will then be moved into the athena-framework organization. The plan for how these will be reintegrated into Athena will be discussed in the DependencyInjection section.

Athena makes use of namespaces to group common types together, both for organizational and documentation purposes. This practice however can lead to really long names, such as Athena::Routing::Exceptions::NotFoundException.new "" as you might have used once or twice. Unfortunately without an import system, like ES6 for example, there isn't really a way around this. In order to help alleviate this issue in the mean time, I created various three letter acronyms that point to each component's namespace. For example, ART for Athena::Routing, or AED for Athena::EventDispatcher. Using these aliases in 0.8.0 the previous example would now be ART::Exceptions::NotFound.new "".

Event Dispatcher

The most impactful change was the implementation of a Mediator and Observer pattern event library to handle the core logic of handling a request, as well as a replacement for HTTP::Handler style middleware. Athena internally now operates by emitting various events that can be listened on in order to handle a request. Athena defines various listeners itself as well; including for routing, CORS, exceptions, and a view layer, which I will get to soon. Each event contains information related to the event, for example the Request event contains the current request, while the Exception event contains the current request and the exception that was raised.

The main advantage of this approach is it is much easier to add middleware to the framework since an array of HTTP::Handler do not have to be supplied up front. The listener approach is also much more flexible since the order of execution can be more easily controlled, versus being based on the order of the handler array. Listeners could be defined in external shards and just by requiring it, the listener(s) would automatically be registered.

The other big advantage is listeners support DI, so other dependencies can be injected into the listener as needed. An example of this could be a security listener on the Request event in order to authenticate a request; lookup the corresponding user, and set it within a service that exposes that user to the rest of the application. Another example could be listening on exceptions in order to log information about them for analytics or monitoring.

@[ADI::Register("@logger", tags: ["athena.event_dispatcher.listener"])]
# Define a listener to handle authenticating requests.
# Also register it as a service and give it the proper tag for it to be automatically registered.
struct MonitoringListener
  # Define the interface to implement the required methods
  include AED::EventListenerInterface

  # Define this type as a service for DI to pick up.
  include ADI::Service

  # Specify that we want to listen on the `Exception` event.
  # The value of the has represents this listener's priority;
  # the higher the value the sooner it gets executed.
  def self.subscribed_events : AED::SubscribedEvents
    AED::SubscribedEvents{
      ART::Events::Exception => 0,
    }
  end

  # Define our initializer for DI to inject a logger instance.
  def initialize(@logger : LoggerInterface); end

  # Define a `#call` method scoped to the `Exception` event.
  def call(event : ART::Events::Exception, _dispatcher : AED::EventDispatcherInterface) : Nil
     # Log the exception message
     @logger.error event.exception.message

    # Do whatever else you want to do with the event, including emitting other events
  end
end

Custom events can also be defined that can be emitted from user code, either from another listener, controller action, etc. The EventDispatcher component can also be used independently outside of Athena.

Dependency Injection (DI)

The change to a listener based approach originated due to another flaw in the HTTP::Handler approach; they do not play nicely with DI. The main issue is that they are instantiated once when you create the HTTP::Server, outside of the request life-cycle, thus the service container is not available to inject dependencies, nor can the middleware exist as a service itself.

This fact, along with how common middleware is, made me want to rethink the overall design of Athena to be more DI friendly. As mentioned in the Interfaces & DI section in the vision issue, the ultimate goal is to make Athena solely depend on interfaces versus concrete types. This not only allows for a better DI implementation, but also make custom implementations, and testing easier.

The plan for the serializer and logger components is that they are optional; but if installed and required, have an ext, or plugin, or something, file that would better integrate it into the rest of the framework. An example of this could be defining some types from that component as services, or defining a basic implementation of an interface for use within Athena. For example:

# ext/logger.cr
@[ADI::Register(name: "logger")]
# Define a basic implementation of a logger service for services to inject
struct Logger
  # Be sure it adheres to the interface.  This also would allow
  # external/third-party code to define their own implementation
  # of the logger service to use
  include LoggerInterface

  ...
end

# some_controller.cr
require "athena/ext/logger"

class SomeController < ART::Controller
  include ADI::Injectable

  # The logger could be injected into anything logging is needed.
  # Each request would have its own logger instance
  def initialize(@logger : LoggerInterface); end

  @[ART::Get("some/path")]
  def some_path : String
    @logger.info "Some message"
    ...
  end
end

Internally, Athena would utilize optional services to manage this. I.e. by default no logging, but if a logger service is registered and available, use that; such as for debug information on each request, or logging exceptions etc.

ErrorRendererInterface

The ErrorRendererInterface is a perfect example of how DI fits into the framework. The default error renderer will JSON serialize the exception and return that string to the client. However, this behavior is customizable by redefining the error_renderer service; if for example you wish to return an HTML error page or something.

View Layer

In previous versions of Athena, the response format was not very customizable. The implementation was tied to some type that defines a render class method on it. Once again this does not fit into the DI oriented framework I so desire, so I had to rethink the implementation to better handle that as well as improve the overall flexibility of the framework.

The solution to this is two fold, the ART::Response type and the view event.

ART::Response

The concept behind the ART::Response type is that it represents a "pending" response to the client. It can be mutated, body rewritten etc, all without affecting the actual HTTP::Server::Response. The idea behind it is that a controller action can either return an ART::Response or not; the behavior of the framework changes slightly if the return value is an ART::Response.

If a controller action returns an ART::Response, or a subclass of it like ART::RedirectResponse, then that object is used directly as the response to the client; body, status, and headers are simply copied over to the actual HTTP::Server::Response. This allows an action to return arbitrary data easily to fulfill simple use cases.

class TestController < ART::Controller
  @[ART::Get("/css")]
  def css : ART::Response
    # A controller action returning CSS.
    ART::Response.new ".some_class { color: blue; }", headers: HTTP::Headers{"content-type" => MIME.from_extension(".css")}
  end
end

Another thing that came from this is the render macro; it provides a simple way to render ECR templates. The variables used in the template can come directly from the action's arguments, such as the example in the API docs. This feature combined with ParamConverterInterface can make for a very easy way to render a user's profile for example:

@[ART::Get("/user/:id/profile")]
@[ART::ParamConverter("user", converter: DBConverter(User))]
def user_profile(user : User) : ART::Response
  # Render the user profile for the user with the provided ID.
  # The template has access to the full User object.
  render "user_profile.ecr"
end

ART::Events::View

The second part of the view layer is the view event, and the corresponding default listener. When a controller action's return value is NOT an ART::Response, the view event is emitted. The job of the listeners listening on the event is to convert the resulting value into an ART::Response. The default listener does this by JSON serializing the value, by default using the standard library's #to_json method, or in the future, optionally by the serializer component if it is installed

In the future, a format negotiation algorithm will be implemented to call the correct renderer based on the request's Accept headers. For now, if you wish to define a global custom format for your routes, you have a few options.

Define a custom view listener that runs before the default one (future listeners will not run once #response= is called)
Subclass ART::Response to encapsulate your logic
Define a helper method/macro on ART::Controller to encapsulate your logic

Overall these changes make Athena a whole lot more flexible, making it viable to render HTML, or anything else that is required.

Routing

While not much changed in the routing aspect of Athena, I do want to point out some of the minor changes/additions that did occur.

QueryParams

In previous versions, query parameters were included directly within the HTTP method annotation as a NamedTuple. In 0.8.0, QueryParams are now defined by their own dedicated annotation. They still support constraints, and param converters.

@[ART::Get("/example")]
@[ART::QueryParam("value")] # Is typed as a string due to `name : String`
def get_user(name : String) : Nil
end

Macro DSL

One of the draws of Sinatra, and by extension, Kemal is the super simple syntax.

get "/index"
  "INDEX"
end

One of the downsides of Athena that I've heard is its verbosity. To help with this, I created a similar macro DSL that simply abstracts the creation of a method and addition of the annotation.

class ExampleController < ART::Controller
  # Super simple right?
  get "index" do
    "INDEX"
  end

  # It also works with arguments, and other annotations like ParamConverters/QueryParams
  @[ART::QueryParam("value3")]
  get "values/:value1/:value2", value1 : Int32, value2 : Float64, value3 : Int8 do
    "Value1: #{value1} - Value2: #{value2} - Value3: #{value3}"
  end
end

It can still get pretty verbose if you have many path arguments, with a non String return type, and some route constraints, but this and the three letter acronym aliases will help reduce the learning curve, and make life a little bit easier.

Access Raw `HTTP::Request`

Before there was not really a way to access the current request object outside of the RequestStore, and related, not really an easy way to access the raw response body of said request. In addition, the action argument's name for POST requests had to be body. This was far from ideal and has been improved greatly in 0.8.0. The raw HTTP::Request and by extension, its body IO, can now be accessed by simply typing an action argument to HTTP::Request. Athena will see that and provide the raw object to that action.

@[ART::Post(path: "/foo")]
# The name of the argument can be anything as long as its type is `HTTP::Request`.
def post_body(request : HTTP::Request) : String
  request.body.try &.gets_to_end
end

The Future

The overall roadmap for Athena is outlined in the vision issue. For the short term I will continue with fixing any issues, and improving the documentation as needed. I will also continue working on reworking and moving the remaining components into the Github organization.

The medium to long term includes the creation of additional components, such as resurrecting the CLI component, as well as introducing a more structured framework to handle authentication and access control of a given route.

As usual, any issues/comments/questions, feel free to drop a comment on this article, or come join me in the Athena Gitter channel.

Object Validation with Assert

Blacksmoke16 — Thu, 29 Aug 2019 16:43:41 +0000

UPDATE: Oct 21, 2019 Update examples for Assert v0.2.0.

Assert

Assert is an annotation based object validation library heavily inspired by Symfony Validation Constraint Annotations.

@[Assert::NotBlank]
property name : String

@[Assert::GreaterThanOrEqual(value: 0)]
property age : Int32

Introduction

Validations in Crystal, up until now, have mostly been shard specific. One shard could have their own implementation for validating POST body data, while an ORM in the same application could be completely different. Assert was initially included in CrSerializer but has since gone through a rewrite and re-released as its own shard with the idea of brining some level of standardization that every shard could benefit from. The main benefit of this would be the ability to use the same validations throughout your application.

This article is intended to be an overview of features/example use cases. Most of the example code comes from the API Docs which contains more detailed documentation on each feature.

Features

Assert was created with flexibility and extensibility in mind, by default it includes:

Multitude of built-in assertions:
- Email
- Ip
- URL
- Choice
- Sub object/array of objects are all valid
- Etc.
Ability to create/use custom assertions
- Use generics within custom assertions
Ability to run a subset of assertions on an object
Control how and when the object is validated

Usage

In order to use Assert:

Add it to your shard.yml
Require it - require "assert"
Include it in your object - include Assert

An example usage can be seen in the API docs.

Groups

Each assertion can also be assigned to a group(s) in order to run subsets of assertions.

class Groups
  include Assert

  def initialize(@group_1 : Int32, @group_2 : Int32, @default_group : Int32); end

  @[Assert::EqualTo(value: 100, groups: ["group1"])]
  property group_1 : Int32

  @[Assert::EqualTo(value: 200, groups: ["group2"])]
  property group_2 : Int32

  @[Assert::EqualTo(value: 300)]
  property default_group : Int32
end

Groups.new(100, 200, 300).valid?                      # => true
Groups.new(100, 100, 100).valid?                      # => false
Groups.new(100, 100, 100).valid?(["group1"])          # => true
Groups.new(200, 100, 300).valid?(["default"])         # => true
Groups.new(100, 200, 200).valid?("group1", "default") # => false

This allows you to reuse the same class/struct in various scenarios since you can control which assertions run. For example, running a different set of assertions when a user registers for the first time, versus when they update some of their information.

Custom Assertions & Generics

If your application has some unique validation requirements that the included assertions do not cover; you can create custom assertions.

A custom assertion is simply a class that inherits from Assert::Assertions::Assertion, applies the Assert::Assertions::Register annotation, and implements some methods.

@[Assert::Assertions::Register(annotation: Assert::Exists)]
# A custom assertion that validates if a record exists with the given *id*.
#
# For example, an ORM model where `.exists?` checks if a record exists with the given PK.
# I.e. `SELECT exists(select 1 from "users" WHERE id = 123);`
class Exists(PropertyType, Model) < Assert::Assertions::Assertion
  # This is a helper macro to make defining the initialize method easier
  initializer(
    actual: PropertyType
  )

  # :inherit:
  def default_message_template : String
    "'%{actual}' is not a valid %{property_name}."
  end

  # :inherit:
  def valid? : Bool
    Model.exists? @actual
  end
end

In this example, we're defining a custom assertion called Exists with the following methods:

initializer - A helper macro that defines the initializer.
- See the API Docs for more information on what the initializer is used for.
- See Assertion.initializer for more information on the macro itself
default_message_template - The error message template to use if the assertion fails and no custom message was provided.
- Instance variables on the assertion can be used within the template by surrounding the ivar's name in double curly braces.
valid? - Implements the logic that determines if the property is valid or not.
- The implementation could be whatever you wish as long as it returns true or false.

We also need to apply the Assert::Assertions::Register annotation, which is used to define the name of the annotation that should be applied to properties. In this case Assert::Exists. Now this assertion is ready to be used.

class Post < SomeORM::Model
  include Assert

  def initialize; end

  @[Assert::Exists(User)]
  property author_id : Int64 = 17
end

In this example, the assertion would run a SQL query to determine if a User with an id of 17 exists.

A Dependency Injection shard, such as Athena's DI Module could also be used to inject the current user/request, or some ACL service into the assertion.

Example Use Cases

Assert is not useful unless there are objects that need validating. For some projects/applications it might not be necessary. However web frameworks and ORMs could easily benefit, as validations are an important part of both.

Web Framework

Web applications have the most apparent need for validations. This could either be validating the POST body in an API, or validating user input from a form submission. However, Assert is not for client side validation. It would not, for example, prevent a user from entering a string in a numeric form field. It would be caught when that form was submitted, and the server runs the validations.

Lets go over an example of how Assert could be used within an API using Kemal.

require "kemal"
require "assert"
require "json"

# user.cr
class User
  include JSON::Serializable
  include Assert

  # Asserts that their age is >= 0 AND not nil
  @[Assert::NotNil]
  @[Assert::GreaterThanOrEqual(value: 0)]
  property age : Int32?

  # Assert their name is not blank
  @[Assert::NotBlank]
  property name : String

  # Assert their email is not blank AND is a valid format
  @[Assert::Email(message: "'%{actual}' is not a proper email")]
  @[Assert::NotBlank]
  property email : String

  # Assert their password is between 7 and 25 characters
  @[Assert::Size(Range(Int32, Int32), range: 7..25)]
  property password : String

  # Have the object be validated after the it is deserialized
  def after_initialize
    validate!
  end
end

# users_controller.cr
post "/users" do |env|
  env.response.content_type = "application/json"
  user = User.from_json env.request.body.not_nil!
  # Do stuff with a valid user

  # Return the user as JSON in the response
  user.to_json
rescue ex : Assert::Exceptions::ValidationError
  env.response.status_code = 400
  env.response.print ex.to_json
end

Kemal.run

This setup makes the validations on User run after the object has been deserialized from the JSON POST body. If the object is not valid #validate! will raise a ValidationError exception. We are catching that exception and returning the JSON error message back to the user.

Assert also defines #valid? and #validate methods which return a Bool if it the object is valid, or an array of failed assertions respectively. The former being most useful if you just want to know if an object is valid, with the latter being most useful if you wanted to do something with the assertion data, such as formatting an error message.

curl -X POST \
  http://localhost:3000/users \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Jim",
    "age": -1,
    "email": "foobar",
    "password": "monkey123"
}'

Would return the following error

{
    "code": 400,
    "message": "Validation tests failed",
    "errors": [
        "'age' should be greater than or equal to '0'",
        "'foobar' is not a proper email"
    ]
}

The JSON error response format can be changed by overriding the #to_json(builder : JSON::Builder) method within the exception class; if you wanted to group the errors by the property name for example.

class Assert::Exceptions::ValidationError
  def to_json(builder : JSON::Builder)
    builder.object do
      builder.field "code", 400
      builder.field "message", @message
      builder.field "errors" do
        builder.object do
          @failed_assertions.group_by(&.property_name).each do |prop, errors|
            builder.field prop, errors.map &.message
          end
        end
      end
    end
  end
end

Would output:

{
    "code": 400,
    "message": "Validation tests failed",
    "errors": {
        "age": [
            "'age' should be greater than or equal to '0'"
        ],
        "email": [
            "'foobar' is not a proper email"
        ]
    }
}

This, of course, is just a demonstration. A framework could easily integrate Assert to make the process more streamlined. Such as how its used within Athena.

@[Athena::Routing::Post(path: "users")]
@[Athena::Routing::ParamConverter(param: "body", type: User, converter: Athena::Routing::Converters::RequestBody)]
def new_user(body : User) : User
  body
end

The ParamConverter handles converting the request body into a User object, any validations on User will also be executed after it has been deserialized. The JSON error will be returned automatically if it is not valid. The controller action will also never execute if the object is not valid.

While using Assert may make your objects larger, it removes the need for validation within the controller code as you know can be assured that invalid objects will not make it far. It also allows you to share the same validation logic from other parts of your application. Such as an ORM.

ORM Models

ORMs models inherently require validation to make sure what is being persisted to the database is correct. Continuing along with our Kemal example, we could easily convert our User object into a Granite model.

NOTE: This is just an example and does not include setting up the ORM

require "granite"
require "assert"

class User < Granite::Base
  include Assert

  table "users"

  column id : Int64?, primary: true

  # Asserts that their age is >= 0 AND not nil
  @[Assert::NotNil]
  @[Assert::GreaterThanOrEqual(value: 0)]
  column age : Int32?

  # Assert their name is not blank
  @[Assert::NotBlank]
  column name : String

  # Assert their email is not blank AND is a valid format
  @[Assert::Email(message: "'%{actual}' is not a proper email")]
  @[Assert::NotBlank]
  column email : String

  # Assert their password is between 7 and 25 characters
  @[Assert::Size(Range(Int32, Int32), range: 7..25)]
  column password : String

  # Granite includes `JSON::Serializble` by default
  def after_initialize
    validate!
  end
end

Our controller action would also slightly change.

post "/users" do |env|
  env.response.content_type = "application/json"
  user = User.from_json env.request.body.not_nil!
  # We can now just save the user since we know its valid
  user.save

  # Return the user as JSON in the response
  user.to_json
rescue ex : Assert::Exceptions::ValidationError
  env.response.status_code = 400
  env.response.print ex.to_json
end

POSTing a valid user would return:

{
    "id": 1,
    "age": 17,
    "name": "Jim",
    "email": "test@gmail.com",
    "password": "monkey123"
}

Conclusion

Assert's benefits:

Having a framework agnostic validation library that can be used across shards.
- Sharing validation logic between your API controllers and ORM models for example.
Easily extensible by the user.
- Custom assertions can be easily defined without needing to edit any shard's source code.
Easy to integrate into existing frameworks/shards.
- Current projects/frameworks could easily add Assert to their objects and control when/how they get validated.
  - Such as the Kemal example. Moving validations into the objects and out of the controller actions
  - Or an ORM that would make sure the model is valid before saving

As usual, if you have any questions, feedback, or ideas for new assertions; feel free to message me on the Crystal Gitter, or create an issue on the Github Repo.

Dependency Injection in Crystal

Blacksmoke16 — Sun, 28 Jul 2019 14:17:53 +0000

UPDATE: January 12, 2020 for Athena version 0.8.0
UPDATE: June 16, 2020 for Athena version 0.9.0

Dependency Injection (DI)

Dependency Injection can be a powerful tool in making an application easier to test, more expandable, and increase the flexibility of the system's components.

This article assumes you are somewhat familiar with the concepts of DI. If not, check out some of these other articles:

DI is usable in every language, although it is more common in some than others. Crystal, and Ruby by extension, seem to not utilize DI as much as other languages. While its out of scope of this article to figure out why that is, lets go through some useful patterns and examples of how DI can be useful in your application.

We'll be using Crystal as our language of choice, along with Athena as our framework. Athena's DI component is also available as a standalone shard.

Manager

The first example is what I call the "manager" pattern. This pattern is most useful when dealing with multiple object instances based on a singular class/interface. I.e. multiple objects instantiated from the same class.

Problem Statement

Say you are going to partner with n other companies. You want to setup an API endpoint that would expose partner specific data within your system for them to consume. You also want it to be easy to add partners; requiring minimal to no code changes to do so.

Solution

Define a Partner class that implements common methods/properties that each partner would have in common.
Define a PartnerManager class that would be responsible for "managing" the Partner instances.
Define a PartnerParamConverter used to convert a partner's id (obtained from the API route) into a Partner instance.
Define a PartnerController to group of the logic of partner related API endpoints.

The Code

require "athena"

private PARTNER_TAG = "partner"

# Define a type that all partners wil be based off of, then register some partners.
@[ADI::Register(_id: "GOOGLE", name: "google")]
@[ADI::Register(_id: "FACEBOOK", name: "facebook")]
record Partner, id : String

# We can also use `ADI.auto_configure` to handle applying the
# correct tag to each `Partner` instance.
ADI.auto_configure Partner, {tags: [PARTNER_TAG]}

# Define another service that will have all partners injected into it.
# This manager will be injected into our param converter to handle
# resolving a `Partner` from their id.
@[ADI::Register(_partners: "!partner")]
struct PartnerManager
  @partners : Hash(String, Partner) = {} of String => Partner

  def initialize(partners : Array(Partner))
    # Create a mapping of partner ID to the partner instance.
    partners.each do |partner|
      @partners[partner.id] = partner
    end
  end

  # Returns a `Partner` with the provided *partner_id* or raises an
  # `ART::Exceptions::NotFound` exception if one could not be found.
  def get(partner_id : String) : Partner
    @partners[partner_id]? || raise ART::Exceptions::NotFound.new "No partner with an ID '#{partner_id}' has been registered."
  end
end

@[ADI::Register]
struct PartnerParamConverter < ART::ParamConverterInterface
  def initialize(@partner_manager : PartnerManager); end

  # :inherit:
  def apply(request : HTTP::Request, configuration : Configuration) : Nil
    # Grab the partner's ID from the request's attributes, then resolve it into a Partner.
    # Path/query params are automatically populated into the attributes.
    partner = @partner_manager.get request.attributes.get "id", String

    # Add the resolved partner object to the request's attributes
    # for it to later be resolved for the controller action argument.
    request.attributes.set configuration.name, partner, Partner
  end
end

# The controller that would house all partner related endpoints.
class PartnerController < ART::Controller
  @[ART::ParamConverter("partner", converter: PartnerParamConverter)]
  get "partner/:id", partner : Partner do
    # Notice we have access to the actual `Partner` object.
    # From here you can do whatever you need with the object.
    "Resolved #{partner.id}!"
  end
end

ART.run

# GET /partner/FOO    # => {"code":404,"message":"No partner with an ID 'FOO' has been registered."}
# GET /partner/GOOGLE # => "Resolved Google!"

Why It Matters

From here, if you wanted to add another partner you would simply register another partner service and everything would just work. E.x. @[ADI::Register(_id: "YAHOO", name: "yahoo")].

This also makes the code generic. Nothing specific to any particular partner is defined anywhere other than their instance of the Partner class.

Alternate solutions

Hard code an array of Partner objects within the controller.
- This wouldn't be an ideal solution since it would tightly couple the Partner struct and the PartnerController. It shouldn't be the responsibility of the PartnerController to know all possible partners. It also would make updating/maintaining/testing of the controller harder due to that extra dependency.
Store the partner data in a database.
- Since our system isn't actually doing anything in the partner's system, this would just be mostly dead data. It would only exist for the sole purpose of resolving the ID. Having unnecessary data is just wasteful.

Plug and Play

The next example isn't so much a pattern, but a core feature of DI. The ability to change the functionality of a class by just changing what service gets injected into it.

Problem Statement

You have a Worker class that will do some work, then write the output to x. x is an external service like Amazon S3, the local file system, Redis, etc. To start this worker will only support one of them, say S3. However, we want to be able to design the class in such a way where the implementation is generic and could work with any other service in the future if so desired.

Solution

Define an "interface", in Crystal's case we'll use a module, to define the public API of our writers
Register an initial implementation of the interface for S3
- Register another implementation for Redis, but default to S3 as the default implementation
Handle injecting the proper writer into our Worker class
Create a test for our class.

The Code

Initial Interface Implementation

require "athena"

# Define an abstract class that will act as our base interface.
# It also will ensure our subclasses implement the correct method.
module WriterInterface
  abstract def write(content : String) : Nil
end

# Create an implementation of `WriterInterface` for S3 and register it.
@[ADI::Register]
class S3Writer
  include WriterInterface

  # :inherit:
  def write(content : String) : String
    # Write the content
    "Wrote data to S3"
  end
end

# Register our worker also as a service, and set it as public.
# Ideally everything would be a service, however in most cases
# at least one service will need to be public in order to be
# the "entrypoint" into the application.
@[ADI::Register(public: true)]
class Worker
  @writer : WriterInterface

  # DI will automatically resolve the correct `WriterInterface` based on the
  # type restriction of the argument, and optionally the argument's name (more on that later).
  def initialize(writer : WriterInterface)
    # Manually assign the ivar to make changing the writer instance easier;
    # as we would only have to update these two variables within initialize versus
    # throughout the class.
    @writer = writer
  end

  def do_work
    # Do some work
    @writer.write "did work"
  end
end

# Grab out worker instance from the container and see what work it does.
ADI.container.worker.do_work # => Wrote data to S3

At the moment, since we only have one implementation of the WriterInterface, this example isn't that much more beneficial than just instantiating the S3Writer within the Worker. However, this changes when we introduce more than one implementation.

Multiple Interface Implementations

Based off the previous example:

# Add another implementation for `Redis`
@[ADI::Register]
class RedisWriter
  include WriterInterface

  # :inherit:
  def write(content : String) : String
    # Write the content
    "Wrote content to Redis"
  end
end

However, we now have a little problem. Since ADI resolves services based on argument type restrictions, and we now have multiple implementations of WriterInterface, how does it know which one to inject? We have two options:

Aliases

ADI has the concept of Service Aliases that allow defining a "default" service to use when the WriterInterface type restriction is encountered.

# The `S3Writer` will now be injected by default
# when the `WriterInterface` type restriction is encountered.
@[ADI::Register(alias: WriterInterface)]
class S3Writer
  ...
end

Argument Name

While having a default implementation is good the majority of the time, what if we want the other implementation? This case can be handled by updating the name of the argument so that the resolution logic would now be based on both the type restriction AND the name of the argument.

@[ADI::Register(public: true)]
class Worker
  @writer : WriterInterface

  # DI would now automatically inject `RedisWriter` since its a `WriterInterface` instance
  # AND it's service name is `redis_writer`.
  def initialize(redis_writer : WriterInterface)
    @writer = redis_writer
  end

  ...
end

Testing

One of the major benefits DI allows for is easier testing since nothing is too tightly coupled with anything else, i.e. everything is built upon abstractions aka interfaces.

require "spec"

# Create a mock writer.
# This allows mocking the response from the external service as we shouldn't be worried about
# how the other service works since we should only be testing our worker, not its dependencies.
class MockWriter
  include WriterInterface

  def write(content : String) : String
    "WROTE_MOCK_DATA"
  end
end

# We can now test the functionality of our `Writer` type in isolation.
describe Worker do
  describe "#do_work" do
    it "should do work" do
      Worker.new(MockWriter.new).do_work.should eq "WROTE_MOCK_DATA"
    end
  end
end

Why It Matters

Since we defined the type restriction as our WriterInterface module, our Writer class is not tightly coupled with any one specific implementation of it. We are able to easily change which implementation gets injected by either updating our alias, or simply changing the name of the initializer argument.

As mentioned previously, one of the main benefits of this is to make the Worker class depend on upon abstractions as opposed to concrete classes. Or in other words, prevent a singular implementation from being tightly coupled with the Worker class. As long as each implementation correctly implements WriterInterface, the Worker class shouldn't care about which implementation it's using, just that calling .write on it, writes the content correctly.

Each WriterInterface implementation could also easily inject their own dependencies, such as credentials, API clients etc.

Another benefit of this pattern is if you have a service that has a singular purpose such as sending an email; you could easily reuse the service. For example, simply inject the EmailProvider service, then use it to send emails.

DI removes the need to worry about how and where objects are getting instantiated. If the EmailProvider had dependencies of its own and you were doing sender = EmailProvider.new xxx each time an email needed sent; that is less than ideal. DI allows the class to be instantiated once with the given arguments; then it can simply be injected where it's needed. E.x. def initialize(@sender : EmailProvider); end.

The EmailProvider service would ideally be tested so you can have confidence that anywhere you inject it, the emails will be sent properly without having to test the same logic multiple times. Speaking of tests, DI allows us to define a test implementation of WriterInterface . When testing a class with dependencies on other services, the dependencies should always be mocked. This gives you control over how those services act. If you didn't mock them out and used the actual implementations your tests would be testing much more than they should.

Sharing Data

The last example is going to show how DI can be used to share data between disparate types.

Problem Statement

You are creating a JSON API. You recently got to the point of where you need to handle user authentication. You want to design things in such a way that allows you to access the current user outside of the request context.

Solution

Define a service that will store current user object
Set the user on that service within your SecurityListener handles authorization.
- Also add some Log context to include this user's information
Use this service to expose user information via an endpoint

The Code

require "athena"

# Our user object, this would most likely be an ORM model.
class User
  include JSON::Serializable

  getter id, customer_id, name

  private def initialize(@id : Int64, @customer_id : Int64, @name : String); end

  # Simulate a ORM query method.
  def self.find(id : Int) : self
    new 1, 12, "Fred"
  end
end

@[ADI::Register]
# Our service that will store user the currently logged in user.
class UserStorage
  property! user : User
end

@[ADI::Register]
# Our custom listener that listens on the Request event.
#
# It'll handle making sure the user's token is valid, and setting the current user.
struct SecurityListener
  include AED::EventListenerInterface

  def self.subscribed_events : AED::SubscribedEvents
    AED::SubscribedEvents{
      ART::Events::Request => 30, # Set the priority higher so it runs before routing
    }
  end

  def initialize(@user_storage : UserStorage); end

  def call(event : ART::Events::Request, dispatcher : AED::EventDispatcherInterface) : Nil
    # Logic to make sure a token is provided.
    token = "PARSED_TOKEN"

    # Logic to validate the token and get the stored user_id from it.
    user_id = 1

    # Fetch the user from the database
    user = User.find user_id

    # Add some logging context for future logs
    Log.context.set user_id: user.id, customer_id: user.customer_id

    # Set the user in user storage.
    @user_storage.user = user
  end
end

# Register the controller itself as a service,
# NOTE: Controller services must be declared as public.
@[ADI::Register(public: true)]
class UserController < ART::Controller
  # Inject our `UserStorage` object
  def initialize(@user_storage : UserStorage); end

  # The user storage service could also be injected anywhere else you need the current user.
  get "user/me", return_type: User do
    Log.info { "Returning data for #{@user_storage.user.name}" }
    @user_storage.user
  end
end

# Start the server
ART.run

# GET /user/me # => {"id":1,"customer_id":12,"name":"Fred"}
# 2020-06-16T02:25:51.593601Z   INFO - athena.routing: Matched route /user/me -- uri: "/user/me", method: "GET", path_params: {}, query_params: {} -- user_id: 1, customer_id: 12
# 2020-06-16T02:25:51.593677Z   INFO - Returning data for Fred -- user_id: 1, customer_id: 12

Why It Matters

Notice that since we set some Log context with our SecurityListener, all logs after that point, within the same fiber, will include that data. Also, since our SecurityListener has a priority of 30, it runs before Athena's routing logic which has a priority of 25. This is beneficial since it means we don't even have to invoke the router if the request is unauthorized.

Similarly to how Log::Context is fiber specific, the DI container is also fiber specific, or in other words unique per request. The benefit of this is it allows sharing arbitrary data between your services, without having to worry about resetting when the request is finished. Notice that the User object set on UserStorage within SecurityListener, is the same one provided to our UserController. Since the DI container handles instantiating and providing objects to our services, we can easily provide references to the same object instance in multiple services without needing to manually pass it through as part of the public API.

The most common way frameworks handle the "current_user" is by reopening HTTP::Server::Context and adding it there, which works fine most of the time. But what happens when you want to access the current user somewhere that isn't a controller action or an HTTP::Handler? Having something that is able to be easily accessed anywhere it is needed is much more flexible.

The UserStorage class could also be expanded into say, TokenStorage which would store the token, which would have a user method; to handle the logic of resolving a User object from the token/data in the token. There could also be different implementations of the Token class, such as AnonymousToken if there is not currently a logged in user. Overall, this approach is just much more flexible than being tied to the request context to access common data, like the current user.

Alternate Solutions

Define the current_user as a class variable.
- This effectively makes the system not fiber safe and unable to handle more than 1 request at a time.
Store a reference within the database.
- This would solve the problem of being able to access the user anywhere, but is less than ideal. If a system is handling many requests, that would equate to a lot of extra unnecessary DB queries.
Store the user within the HTTP::Request object
- This is a viable solution assuming you don't need to access said user anywhere outside of the request context. If for example you wanted to include some user data within an asynchronous message context; how would you provide the user object? In sort there wouldn't be a clean way to do it without making it part of the public enqueuing API.

Conclusion

As shown, DI can make your Crystal application less dependent on concrete classes and more dependent on abstractions. This prevents tight coupling, makes testing easier, and encourages code reuse. However, this does not mean DI is the best pattern to use 100% of the time. Being aware of its capabilities and being able to apply the pattern to a well suited problem is the key.

If anyone has any other alternate solutions (both good and bad), feel free to share them in the comments. I would be quite interested in seeing how people, those mainly coming from a Ruby background, would solve some of these scenarios.

As usual feel free to join me in the Athena Gitter channel if you have any suggestions, questions, or ideas. I'm also available on Discord (Blacksmoke16#0016) or via Email.

oq - A portable/performant jq wrapper

Blacksmoke16 — Sat, 13 Jul 2019 17:48:58 +0000

oq

A performant, and portable jq wrapper thats facilitates the consumption and output of formats other than JSON; using jq filters to transform the data.

Background

I've been using jq for a while for transforming a master JSON document into partner dependent structures for their consumption. However, up until recently all of the partner structures have also been in JSON. Since jq does not support outputting XML on its own, I began to look around to see if there were any libraries that would allow using jq filters to transform the data, but output XML in addition to JSON. I ended up finding a Python library called yq that seemed to be perfect.

It supports outputting to XML and JSON while being able to use the same jq filter for both. After playing around with it for a while it became clear that, while quite speedy for smaller files, it really struggled with some of the larger documents I needed to process. The fact that it's Python also complicated things as Python needs to be installed to use it, without going through some extra process to make it a singular binary. Thus, the idea for a more performant and portable option began to take shape.

Introduction

Using the relatively new Crystal language; I created oq with the primary goals being portability, performance, and to extend the formats that jq supports.

Usage

oq has three additional arguments that sets the input/output formats to use, in additional to the name of the root element if serializing to XML. All other arguments are passed on to jq.

Examples

Consuming JSON and output XML

echo '{"name": "Jim"}' | oq -o xml .
<?xml version="1.0" encoding="UTF-8"?>
<root>
  <name>Jim</name>
</root>

Consuming JSON and output YAML

echo '{"name": "Jim"}' | oq -o yaml .
---
name: Jim

Consume YAML from a file and output XML

data.yaml

---
name: Jim
numbers:
  - 1
  - 2
  - 3

oq -i yaml -o xml . data.yaml 
<?xml version="1.0" encoding="UTF-8"?>
<root>
  <name>Jim</name>
  <numbers>1</numbers>
  <numbers>2</numbers>
  <numbers>3</numbers>
</root>

Consume JSON, transform it, and output XML

data.json

{
  "guests": [
    {
      "name": "Jim",
      "age": 17,
      "numbers": [
        1,
        2,
        3
      ]
    },
    {
      "name": "Bob",
      "age": 51,
      "numbers": [
        4,
        5,
        6
      ]
    },
    {
      "name": "Susan",
      "age": 85,
      "numbers": [
        7,
        8,
        9
      ]
    }
  ]
}

filter

.guests | 
{ 
  "person": [
    .[] | {
      "age": {
        "@scale": .scale,
        "#text": .age
      },
      "name": .name,
      "favorite_numbers": {
        "number": .numbers 
      }
    }
  ]
}

oq -o xml --xml-root people -f filter data.json
<?xml version="1.0" encoding="UTF-8"?>
<people>
  <person>
    <age scale="months">289</age>
    <name>Jim</name>
    <favorite_numbers>
      <number>1</number>
      <number>2</number>
      <number>3</number>
    </favorite_numbers>
  </person>
  <person>
    <age scale="years">51</age>
    <name>Bob</name>
    <favorite_numbers>
      <number>4</number>
      <number>5</number>
      <number>6</number>
    </favorite_numbers>
  </person>
  <person>
    <age scale="days">31025</age>
    <name>Susan</name>
    <favorite_numbers>
      <number>7</number>
      <number>8</number>
      <number>9</number>
    </favorite_numbers>
  </person>
</people>

The approach on handling the JSON to XML transcoding is based on this article.

Benchmarks

I also ran some benchmarks for jq, yq, and oq to show how they compare in various situations.

Setup

OS: #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16)
CPU: Intel i7-7700k
Memory: 32GB @ 3,000 MHz
SSD: Samsung 850 PRO - 512GB

Benchmarks are done via the /usr/bin/time -v command

Simple

First, I used the data.json file to see how they perform simply parsing the file and output itself via the . filter.

jq

jq . data.json | wc -l
    Command being timed: "jq . data.json"
    User time (seconds): 0.02
    System time (seconds): 0.01
    Percent of CPU this job got: 68%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.06
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 16236
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 3860
    Voluntary context switches: 224
    Involuntary context switches: 8
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0
31

yq

yq . spec/assets/data1.json | wc -l
    Command being timed: "yq . data.json"
    User time (seconds): 0.08
    System time (seconds): 0.01
    Percent of CPU this job got: 77%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.11
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 16252
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 1
    Minor (reclaiming a frame) page faults: 7179
    Voluntary context switches: 189
    Involuntary context switches: 10
    Swaps: 0
    File system inputs: 1672
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0
31

oq

oq . data.json | wc -l
    Command being timed: "oq . data.json"
    User time (seconds): 0.02
    System time (seconds): 0.04
    Percent of CPU this job got: 74%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.10
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 16140
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 4499
    Voluntary context switches: 306
    Involuntary context switches: 13
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0
31

For this first test, all three are pretty much equal, with only a negligible difference in wallclock/memory used.

Jeopardy.json (#2)

The next benchmark uses the jeopardy.json ~56mb file as retrieved in jq's benchmark wiki page.

First up, a simple length jeopardy.json command.

jq

jq length jeopardy.json 
216930
    Command being timed: "jq length jeopardy.json"
    User time (seconds): 0.64
    System time (seconds): 0.10
    Percent of CPU this job got: 97%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.76
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 230080
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 63213
    Voluntary context switches: 240
    Involuntary context switches: 13
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

yq

yq length jeopardy.json 
216930
    Command being timed: "yq length jeopardy.json"
    User time (seconds): 152.45
    System time (seconds): 1.27
    Percent of CPU this job got: 100%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 2:33.04
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 3853532
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 1117041
    Voluntary context switches: 13708
    Involuntary context switches: 3189
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

oq

oq length jeopardy.json 
216930
    Command being timed: "oq length jeopardy.json"
    User time (seconds): 0.67
    System time (seconds): 0.17
    Percent of CPU this job got: 105%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.80
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 230224
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 63839
    Voluntary context switches: 13832
    Involuntary context switches: 12
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

The big files do not bode well with yq, with it taking ~190x longer than either oq or jq, while also using almost 17x more memory.

YAML => XML

The last benchmark I did was giving both yq and oq a large yaml file (~57mb), then having them convert it to XML. Since jq can't consume YAML, I excluded it.

The file used: invItems.yaml from the EVE Online SDE Export.

Example Input:

-   flagID: 0
    itemID: 0
    locationID: 0
    ownerID: 0
    quantity: -1
    typeID: 0
-   flagID: 0
    itemID: 1
    locationID: 0
    ownerID: 0
    quantity: -1
    typeID: 0
 ...

yq

For yq, I had to give it a filter and some extra args for it to output correctly

yq -s -x --xml-root items --xml-dtd '{"item": .[] | .}' invItems.yaml > invItems.yq.xml
    Command being timed: "yq -s -x --xml-root items --xml-dtd {"item": .[] | .} invItems.yaml"
    User time (seconds): 309.21
    System time (seconds): 2.76
    Percent of CPU this job got: 100%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 5:11.90
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 7817608
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 2262904
    Voluntary context switches: 32918
    Involuntary context switches: 2504
    Swaps: 0
    File system inputs: 0
    File system outputs: 195072
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

Example Output

<?xml version="1.0" encoding="utf-8"?>
<items>
  <item>
    <flagID>0</flagID>
    <itemID>0</itemID>
    <locationID>0</locationID>
    <ownerID>0</ownerID>
    <quantity>-1</quantity>
    <typeID>0</typeID>
  </item>
  <item>
    <flagID>0</flagID>
    <itemID>1</itemID>
    <locationID>0</locationID>
    <ownerID>0</ownerID>
    <quantity>-1</quantity>
    <typeID>0</typeID>
  </item>
  ...
</items>

oq

oq -i yaml -o xml --xml-root items . invItems.yaml > invItems.oq.xml
    Command being timed: "oq -i yaml -o xml --xml-root items . invItems.yaml"
    User time (seconds): 20.08
    System time (seconds): 0.48
    Percent of CPU this job got: 107%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:19.13
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 1332328
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 522235
    Voluntary context switches: 30478
    Involuntary context switches: 974
    Swaps: 0
    File system inputs: 0
    File system outputs: 195072
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

Example Output

<?xml version="1.0" encoding="UTF-8"?>
<items>
  <item>
    <flagID>0</flagID>
    <itemID>0</itemID>
    <locationID>0</locationID>
    <ownerID>0</ownerID>
    <quantity>-1</quantity>
    <typeID>0</typeID>
  </item>
  <item>
    <flagID>0</flagID>
    <itemID>1</itemID>
    <locationID>0</locationID>
    <ownerID>0</ownerID>
    <quantity>-1</quantity>
    <typeID>0</typeID>
  </item>
  ...
</items>

Similarly to the jeopary.json benchmark, yq just has a hard time dealing with the larger inputs with this test case taking ~16x longer and using almost 6x the memory than oq.

Road to 1.0.0

Since this project is still early in its development, I put together a roadmap of what I would like to get done before calling it 1.0.0:

Support XML input format
Address bugs/issues that arise
Small feature requests
Possibly additional formats

Feel free to submit issues/PRs.

Productive logging with Athena & Crylog

Blacksmoke16 — Sat, 01 Jun 2019 20:03:33 +0000

This article was written for an older version of Athena and is no longer valid. Its content has been integrated into Creating a JSON API with Athena & Granite and Dependency Injection in Crystal.

Also, `Crylog` has been deprecated in favor of the standard libraries' Log module.

Logging

Logging is an important tool when creating any application. Logging adds information that could be useful for debugging issues, doing analytics, or just gathering user behavior data; all of which can be used to inform future decisions.

However not all logging is equal. Logging should be setup in such a way so that only useful information is logged, depending on the environment.
For example, a developer working on an application would love to see debug level information so that they have all of the possible information available in order to track down issues and find bottlenecks. The application running in the production environment should NOT see debug level logs so that the log files are not cluttered, which causes actual important issues to be lost in the shuffle.

Crylog

Crylog is a new flexible logging framework for Crystal based on Monolog. It allows applications to have multiple named loggers, each with various handlers, processors, formatters.

A handler would be something that does something with a logged message. For example logging to a file or STDOUT. But it could also be sending the message to Sentry, or a database etc.

A processor adds extra data to each message. A good use case of this would be adding user/customer ids to each logged message that would represent the user which caused that message to be logged.

Lastly, a formatter of course determines how a message gets serialized. Formatters allow different styles to be used depending on the handler, as HTML would be great for email handlers but not so much for log files.

When used together, Crylog allows for various loggers to be customized to handle the various aspects of an application. Such as only logging messages that are warnings or higher in the production environment, but all messages during development.

Athena

Athena is an annotation based web framework which uses Crylog as its logging solution. For this article I'm going to be demonstrating some of the benefits that Crylog enables when combined with some neat Athena features, such as Dependency Injection. This will be a continuation of the Blog Article I wrote a few months ago.

Dependency Injection (DI)

One of the newer features of Athena is a service container layer. This allows a project to share useful objects, aka services, throughout the project. These objects live in a special class called the Service Container (SC). Object instances can be retrieved from the container, or even injected directly into classes as a form of constructor DI. While this article is not focused on DI specifically, it will be using it. For more information related to it, see the Athena Docs.

Agenda

First, lets define some goals we wish to achieve via our logging.

Log some informational messages when articles are created/deleted, or a user logins, or a new user registers, along with some contextual information.
Both scenarios should be logged to a file, but the development environment should also log to STDOUT.

By default Athena will log all messages to STDOUT and a development.log file when in the development environment, while the production environment logs to a production.log file, but only for warnings and higher. Also since we want to add some custom logic into our loggers, we'll need to define our own configuration.

Security HTTP Handler

In the last tutorial, we created a HTTP::Handler class to handle authorization around the token used in each request. I do have plans for some security related features that I use in this article to be built into Athena itself, however for now it is up to the user to define.

However there are some problems with this approach. As I left off in the previous tutorial, how would we know what user is logged in within our controller actions? To solve this we can create a UserStorage class that will store the current user, so that other classes would have access to it via DI. Lets create a new directory under src called services that I'll put the UserStorage service. Next create a new file called user_storage.cr with the following code. Be sure to require it in your blog.cr file as well.

module Blog
  @[Athena::DI::Register]
  class UserStorage < Athena::DI::ClassService
    # Use a ! property since they'll always be a user defined in our use case.
    # It also defines a nilable getter method to make sure there is a user
    property! user : Blog::Models::User
  end
end

Notice how we are inheriting from Athena::DI::ClassService, this allows Athena to get a list of all services to register, while the @[Athena::DI::Register] allows some configuration on how the service is registered.

Now we can go back to our security handler, get the user storage service, and set the user.

module Blog
  class SecurityHandler
    include HTTP::Handler

    def call(ctx : HTTP::Server::Context) : Nil
      ...
      # Get the user storage service from the container
      user_storage = Athena::DI.get_container.get("user_storage").as(UserStorage)

      # Set the user in user storage
      user_storage.user = Blog::Models::User.find! body[0]["user_id"]

      # Call the next handler
      call_next ctx
    end
  end
end

Then we can go do a similar thing and update the references in the ArticleController.

module Blog::Controllers
  @[Athena::Routing::ControllerOptions(prefix: "article")]
  class ArticleController < Athena::Routing::Controller
    include Athena::DI::Injectable

    def initialize(@request_stack : Athena::Routing::RequestStack, @user_storage : UserStorage); end

    @[Athena::Routing::Post(path: "")]
    @[Athena::Routing::ParamConverter(param: "body", type: Blog::Models::Article, converter: Athena::Routing::Converters::RequestBody)]
    def new_article(body : Blog::Models::Article) : Blog::Models::Article
      # Sets the owner of the blog post as the current authed user
      body.user = @user_storage.user
      body.save
      body
    end

    @[Athena::Routing::Get(path: "")]
    def get_articles : Array(Blog::Models::Article)
      # We are also using the user in UserStorage as an additional conditional in our query when fetching articles
      # this allows us to only returns articles that belong to the current user.
      Blog::Models::Article.where(:deleted_at, :neq, nil).where(:user_id, :eq, @user_storage.user.id).select
    end

    @[Athena::Routing::Put(path: "")]
    @[Athena::Routing::ParamConverter(param: "body", type: Blog::Models::Article, converter: Athena::Routing::Converters::RequestBody)]
    def update_article(body : Blog::Models::Article) : Blog::Models::Article
      body.user = @user_storage.user
      body.save
      body
    end

    @[Athena::Routing::Get(path: "/:article_id")]
    @[Athena::Routing::ParamConverter(param: "article", pk_type: Int64, type: Blog::Models::Article, converter: Athena::Routing::Converters::Exists)]
    def get_article(article : Blog::Models::Article) : Blog::Models::Article
      article
    end

    @[Athena::Routing::Delete(path: "/:article_id")]
    @[Athena::Routing::ParamConverter(param: "article", pk_type: Int64, type: Blog::Models::Article, converter: Athena::Routing::Converters::Exists)]
    def delete_article(article : Blog::Models::Article) : Nil
      article.deleted_at = Time.utc
      article.save
      @request_stack.response.status = HTTP::Status::ACCEPTED
    end
  end
end

We are also able to inject the RequestStack in order to have access to the request/response.

NOTE: Services can only be auto injected, like the ArticleController if the class gets instantiated within the request/response cycle. Classes instantiated outside of it, like the SecurityHandler, do not have access to the same container and services must be fetched manually.

Now that we have our authentication problem resolved we can move onto adding some logging.

Adding Logs

We'll do the two public routes first. Simply go to your AuthController and add a Athena.logger.info "User logged in", Crylog::LogContext{"user_id" => user.id} before the {token: user.generate_jwt}. This would log

[2019-11-20T01:40:40Z] main.INFO: User logged in {"user_id":1}

our message with the id of the user that logged in. We can do a similar thing in our UserController. Add a Athena.logger.info "New user registered", Crylog::LogContext{"user_id" => body.id, "email" => body.email, "first_name" => body.first_name, "last_name" => body.last_name} after saving the user model. This would log that someone registered with some information about that user.

We could also do the same thing in our ArticleController, adding like Athena.logger.info "Article ##{body.id} was created", Crylog::LogContext{"user_id" => @user_storage.user.id}. However we could keep things more DRY by using a Crylog Processor since the ArticleController are authenticated endpoints, which would have a user in the UserStorage.

Let's create a new directory under src called logger that I'll put the processor. Next create a new file called user_processor.cr with the following code. Be sure to require it in your blog.cr file as well.

module Blog
  struct UserProcessor < Crylog::Processors::LogProcessor
    def call(message : Crylog::Message) : Nil
      user_storage = Athena::DI.get_container.get("user_storage").as(UserStorage)

      # Return early if a message was logged in a public endpoint there won't be a user in storage
      return unless user = user_storage.user?

      # Add the current user's id to all log messages
      message.extra["user_id"] = user.id
    end
  end
end

This would add the current user's id to every logged message as part of the message's extra property. It also handles the case where we were logging users logging in and registering, which would not have a user in storage. Another option would be to define a logger specifically for public stuff, but that is a bit overkill for now.

Next we need to tell Crylog to use our processor. We will also handle the 2nd bullet point in our agenda. Within blog.cr module add the following code:

  def Athena.configure_logger
    # Create the logs dir if it doesn't exist already.
    Dir.mkdir Athena.logs_dir unless Dir.exists? Athena.logs_dir
    Crylog.configure do |registry|
      registry.register "main" do |logger|
        handlers = [] of Crylog::Handlers::LogHandler

        if Athena.environment == "development"
          # Log to STDOUT and development log file if in develop env.
          handlers << Crylog::Handlers::IOHandler.new(STDOUT)
          handlers << Crylog::Handlers::IOHandler.new(File.open("#{Athena.logs_dir}/development.log", "a"))
        elsif Athena.environment == "production"
          # Log only to a file if in production env.
          handlers << Crylog::Handlers::IOHandler.new(File.open("#{Athena.logs_dir}/production.log", "a"))
        end

        # Tell crylog to use our processor on the main logger.
        logger.processors = [Blog::UserProcessor.new] of Crylog::Processors::LogProcessors

        logger.handlers = handlers
      end
    end
  end

This method tells Crylog how to configure the loggers to use. As you can see, we are logging to STDOUT and a development.log file when in the dev environment, but only a production file when in the prod environment. Notice we are opening the log file with the a option; this makes makes it so messages are appended to the log instead of completely overwriting it. We are also telling it to use our UserProcessor. For more information regarding the possible logging configurations, checkout the Crylog documentation.

After that, start the server and start doing things to see messages logged. Messages logged in the ArticleController will have the current user id included in the message; just like when a user logs in, but without having to explicitly specify it.

From here additional handlers/processors etc could be defined to also log these messages to Greylog, Sentry, or even send emails if they are of a high enough severity. Crylog allows for all sorts of flexibility when deciding on how best implement logging for your project.

Update November 19, 2019 for some minor polish & changes based on previous tutorial/new Athena version

Utilizing Macros & Annotations in a Web Framework

Blacksmoke16 — Thu, 04 Apr 2019 12:29:23 +0000

Macros, and especially annotations, are among the least documented features of Crystal. Because of this, I wanted to take the time and write a little post about how I used annotations and frameworks to build my web framework, focusing on how they are used, not necessarily the framework itself.

A few months ago I started on, yet another, web framework project. However I wanted to take a different approach, taking into consideration the experiences I have from my other side projects as well as those at my work. The result of this was Athena. Unlike other frameworks that rely mostly on macros to define the routes, Athena uses annotations with macros. This allows for code that is very readable, with route actions being just class methods. This has a few benefits such as:

Easier to document - Are just methods so doc comments work just fine.
Easier to test - Are just methods again, can just call your actions directly to test.
Easier to read/understand - There is no special syntax needed (other than learning the annotations)
Inheritance.

Annotations are use heavily, as the main mechanism to define routes, callbacks, and even CLI commands. I also used them in my other serialization/validation shard CrSerializer as an alternative way to manage property serialization/validations of models.

Macros

Macros are defined as (from the git book), "Methods that receive AST nodes at compile-time and produce code that is pasted into a program." Or in other words, code that you write that expands into Crystal code at compile time and gets inserted into the program. Macros are almost a language on their own; quite powerful and flexible. Macros allow you to reduce boilerplate by writing code that can define classes, methods, etc.

Annotations

Annotations, one of the least documented but most powerful features in Crystal. In short, annotations allow types (classes, structs, methods, and properties) to have information added to them that is available at compile time, and readable via macros.

Annotations are defined similarly to a class or struct, but using the annotation keyword.

annotation Get; end

I'm going to define some scenarios I wanted to solve when setting out with creating Athena. Then do a walk-through of my thinking and how I used macros and annotations to solve those problems.

Develop a way of dynamically registering routes based on a specific annotation as opposed to using a standard get "/path" do { } macro for example.
Ability to easily validate properties on the ORM models by applying annotations to each property, instead of using a validate xxx macro.
Similar to 1, but to have self registering CLI commands that get automatically exposed to an OptionParser interface.

Registering Routes

In order to solve this first obstacle, I had to think. "How can I get an Array of controllers with routes defined?"

My ideal solution would have been like:

@[Athena::Routing::Controller]
class MyController
  @[Athena::Routing::Get(path: "/me")]
  def get_me : String
    "Jim"
  end

  @[Athena::Routing::Post(path: "/user")]
  def new_me(body : String) : String
    body
  end
end

{% for controller in Athena:::Routing::Controller.types %}
  # Iterate over all the classes with the `Controller` annotation
{% end %}

Thus, any class that has the Athena:::Routing::Controller would be known to be a controller, class methods would be annotated to tie a given route to the code that should be executed when the route is matched. However, this is not currently possible. See #7274. So I had to think of plan B.

The Crystal generated API docs were super helpful in this regard, especially TypeNode. A TypeNode represents a type in a program, such as String, or MyController, etc. Reading through the available methods, I noticed there was an all_subclasses method; which returns an array of all the subclasses of that type. Ah ha! I then had the idea, similar to Rails and their ApplicationController, that I could make my own class, have controllers inherit from that, and use all_subclasses to iterate over all controllers at compile time. Great!

This discovery lead to the syntax that is used today:

class MyController < Athena::Routing::Controller
  @[Athena::Routing::Get(path: "/me")]
  def get_me : String
    "Jim"
  end

  @[Athena::Routing::Post(path: "/user")]
  def new_me(body : String) : String
    body
  end
end

However there was a new problem. Where do I do this? Turning my attention to the various routers available, such as Amber Router. In the examples, each router required the user to define a tree, add routes to it, and find those routes all in the same file. The key problem here was I needed a way so the user doesn't have to know about how to do any of that. I needed a way to hide the tree creation, adding of routes, and resolving routes behind the scenes.

I went back to the Crystal API docs, specifically the HTTP::Handler class. This module, when included, requires a class to define a call method, that will be executed on each HTTP request. Also, since it is nothing more than a class, I thought I could add an initialize method that would run once when newing up the handler for the HTTP::Server.

Perfect! A simplified cut down version of this ultimately ended up looking like:

class RouteHandler < Athena::Routing::Handlers::Handler
  @routes : Amber::Router::RouteSet(Action) = Amber::Router::RouteSet(Action).new

  def initialize
    {% for c in Athena::Routing::Controller.all_subclasses %}
      {% methods = c.class.methods.select { |m| m.annotation(Get) || m.annotation(Post) || m.annotation(Put) || m.annotation(Delete) } %}
      {% for m in methods %}
        {% if d = m.annotation(Get) %}
          {% method = "GET" %}
          {% route_def = d %}
        {% elsif d = m.annotation(Post) %}
          {% method = "POST" %}
          {% route_def = d %}
        {% end %}

        # Action is a record that contains data about the action.
        # Such as params, the action to execute, path, controller etc. 
        {% action = xxx %}

        @routes.add {{route_def[:path]}}, {{action}}
      {% end %}
    {% end %}
  end
end

I first defined an instance variable routes, which is instantiated to be a new instance of my router tree. I then defined an initialize method. Within this method, I used a macro loop to iterate over the subclasses of the base controller class. I then used .select to get an array of actions defined on each controller, specifically looking for class methods annotated with one of the HTTP verb annotations. This allowed for private class methods to be used within a class, but not expected to be related to a route. I then set the method and route_definition based on what annotation was matched. Finally I register the route with the router.

Annotation fields can be access using the [] method, either using a String or Symbol for named fields, or an Int32 for index based.

Since this is all macro code, the actual code that gets added to the program at compile time would look like:

class RouteHandler < Athena::Routing::Handlers::Handler
  @routes : Amber::Router::RouteSet(Action) = Amber::Router::RouteSet(Action).new

  def initialize
    @routes.add "/GET/me", RouteAction.new("get_me", xxx)
    @routes.add "/POST/user", RouteAction.new("new_user", xxx)
  end
end

Cool! Now I could define the call method to handle routing. Again, a simplified trimmed down version of this ended up looking like:

def call(ctx : HTTP::Server::Context)
  search_key = '/' + ctx.request.method + ctx.request.path
  route = @routes.find search_key

  if route.found?
    action = route.payload.not_nil!
  else
    raise Athena::Routing::Exceptions::NotFoundException.new "No route found for '#{ctx.request.method} #{ctx.request.path}'"
  end

  call_next ctx
end

This calls other handlers I defined for CORS, and executing the actual action tied to the given route. However that doesn't involve any other notable uses of annotations/macros so I'm just going to skip it. However, If you're interested in learning more about how Athena is setup, feel free to message me on the Crystal Gitter.

This portion is now complete and could be used similarly to:

server = HTTP::Server.new([RouteHandler.new])

server.bind_tcp "127.0.0.1", 8080
server.listen

Property Validation

The next challenge I wanted to solve was validations, also following an annotation based approach. The idea was you could add annotations to a property, then on a POST endpoint for example, run those validations. If the validations pass, pass an instance of your model to the route action, otherwise return a 400.

I wanted to have this built on top of JSON::Serializable, so that validations would work out of the box for any class/struct using that as its serialization/deserialization method.

While it sounds simple. "Just create some validation annotations, add a validate method that iterates over those annotations like you did for the routes, and there you go." However, it was not that simple :P

Originally I came up with the syntax of:

@[CrSerializer::Assertions(greater_than_or_equal: 0, nil: false)] 
property age : Int32?

Then, I could read the fields off the annotation, and run corresponding logic to run the assertions.

However it quickly became apparent this approach has some flaws.

The line quickly gets long if there are a lot of assertions.
There is no easy way to force required fields, must supply min AND max for example.
It's not expandable. If a user wanted to add their own assertion, I could not possibly know the keys they would define let alone what to do with them.

So after some thinking I pivoted in favor of this syntax:

@[Assert::NotNil] 
@[Assert::GreaterThanOrEqual(value: 0)] 
property age : Int32?

The benefits of this:

Easier to read, as the lines do not get as long
More expandable
Easier to see related fields

But of course there was some challenges with this approach. There is currently no way to just get an array of annotations applied to a type. Each annotations has to be read by its name. I now needed a way to know the names of each assertion annotation, what fields to read off of it, and a way to tie a given annotation to some specific logic.

I worked around the first two problems by knowing that constants are available at compile time. I defined a constant hash, mapping annotation names to an array of fields that would be read off of it.

ASSERTIONS = {
  Assert::Email              => [:mode],
  Assert::IP                 => [:version],
  Assert::Uuid               => [:versions, :variants, :strict],
  Assert::Url                => [:protocols, :relative_protocol],
  Assert::RegexMatch         => [:pattern, :match],
  ...
}

Since constants are available at compile time, it is possible to use them in a macro loop; which I did to iterate over the name/fields of each assertion.

{% for ivar in @type.instance_vars %}
  {% for t, v in CrSerializer::Assertions::ASSERTIONS %}
    {% ann = ivar.annotation(t.resolve) %}
    {% if ann %}
      assertions << {{t.resolve.name.split("::").last.id}}Assertion({{ivar.type.id}}).new({{ivar.stringify}},{{ann[:message]}},{{ivar.id}},{{v.select { |fi| ann[fi] != nil }.map { |f| %(#{f.id}: #{ann[f]}) }.join(',').id}})
    {% end %}
  {% end %}
{% end %}

This example introduces a new macro concept, @type. When used within a macro, @type represents the current scope or type as a TypeNode. In my case, @type would represent the class that included CrSerializer. I used @type.instance_vars to be able to iterate over each instance variable, read the annotations from it, and add the assertions if any exist. I also use it to add the type and name of each instance variable to the assertion instantiation.

NOTE: @type.instance_vars only currently works in the context of a instance/class method. See #7504.

Now that I had a way to handle the first two issues, I still needed a way to tie a given assertion to the logic specific to that assertion. To do this, I used the key of the ASSERTIONS hash, plus Assertion as the class name to instantiate a new class of that type. For example the logic for Assert::NotNil would be handled in a class called NotNilAssertion; which looks like:

class NotNilAssertion(ActualValueType) < Assertion
  @message : String = "'{{field}}' should not be null"

  def initialize(field : String, message : String?, @actual : ActualValueType)
    super field, message
  end

  def valid? : Bool
    @actual.nil? == false
  end
end

Each assertion class has instance variables that map to the fields in the array from the ASSERTIONS hash. I use a macro to iterate over those fields, find the ones that are set, then new up an instance of that assertion with those values. I added this to a validate method that gets added to the type when including the CrSerializer module. Again, since this is all macro code, the actual code that gets inserted into the program would look like, based on the example above:

def validate : Nil
  assertions = [] of CrSerializer::Assertions::Assertion
  assertions << NotNilAssertion(Int32?).new
  assertions << GreaterThanOrEqualAssertion(Int32?).new(value: 0)
  @validator = CrSerializer::Validator.new assertions
end

But wait, if the ASSERTIONS is a constant, how would this help with allowing users to add their own assertions?

Using a macro I'm able to define the annotation with the given name, and add properties to the hash at compile time, so that the hash contains the standard assertions plus any assertions defined by the user at runtime.

macro register_assertion(name, fields)
  module CrSerializer::Assertions
    annotation {{name.id}}; end
    {% CrSerializer::Assertions::ASSERTIONS[name] = fields %}
  end
end

The user would use this like register_assertion Assert::MyCustom, [:value, :high_is_good], and of course have a MyCustomAssertion class to handle the logic.

@[Assert::MyCustom(value: 100, high_is_good: true)]
property age : Int32

Auto Registering CLI Commands

This is quite similar to the regiserting routes problem, but I'll include it as another example of what macros can do.

struct Registry
  macro finished
    class_getter commands : Array(Athena::Cli::Command.class) = {{Athena::Cli::Command.subclasses}}{% if Athena::Cli::Command.subclasses.size > 0 %} of Athena::Cli::Command.class {% end %}
  end
end

This defines a class getter with a value of an array of classes that inherit from the parent Command class. The value of the array is built out at compile time by simply calling {{Athena::Cli::Command.subclasses}}, which returns an array of classes. This has to be done within a finished macro, so that the types are known to prevent type mismatches.

I then used a macro that inserts an OptionParser that acts as an interface to list/run commands

macro register_commands
  OptionParser.parse! do |parser|
    parser.banner = "Usage: YOUR_BINARY [arguments]"
    parser.on("-h", "--help", "Show this help") { puts parser; exit }
    parser.on("-l", "--list", "List available commands") { puts Athena::Cli::Registry.to_s; exit }
    parser.on("-c NAME", "--command=NAME", "Run a command with the given name") do |name|
      Athena::Cli::Registry.find(name).command.call ARGV
      exit
    end
  end
end

struct MigrateEventsCommand < Athena::Cli::Command
  self.command_name = "migrate:events"
  self.description = "Migrates legacy events for a given customer"

  def self.execute(customer_id : Int32, event_ids : Array(Int64)) : Nil
    # Do stuff for the migration
    # Params are converted from strings to their expected types
  end
end

All that has to be done to use the commands is calling the register_commands macro in your main app file, and create your command struct inheriting from the base command struct. This will then get picked up by Athena::Cli::Command.subclasses and registered at compile time.

Ultimately it could then be used like ./MyApp -c migrate:events --customer_id=83726 --event_ids=1,2,3,4,5. Which would search the registry for the command and pass the args to it if it was defined.

Addendums

I didn't really know what to cover, as it's kind of hard to give examples out of the blue. I'll leave this section open for future additions. If there is something specific you would like to see an example of, or explained further feel free to ask.

ORM

@jwoertink made me think of another example I can walkthrough to go over the idea of "a reason I'd use them." I have been working on an annotation version of Granite, mostly for the lols, that I can give some examples from.

Annotations require a different view of how to structure an application as compared to simply using normal macros. We are all aware of the macro syntax ORMs use to define columns, i.e. field name : String. Behind the scenes the macro would add that field's name, type, and any options supplied to a constant so that at a later stage, once the macro inherited and finished hooks run, the constant could be used to create properties for those fields, applying the proper types/options to the property definition.

Begin Personal Opinion

However, in my opinion, this is a bit convoluted and is a good example of what type of situation annotations could be useful. The main point being, what advantage does using field name : String get you over property name : String then applying an annotation? The latter allows you to just have syntax like every other class in your app, inheritance works, can document properties, and third party annotations could be easily applied (like JSON::Field). Annotations would allow you to remove these extra unnecessary steps of going from macro to constant to property instead of straight to property.

End Personal Opinion

However when taking an annotation based approach, you have to look at it from a different point of view. In the case of an ORM, the main challenge becomes "how can i pass column data around, without adding everything to a mutable constant?" My solution to this is using records and building out, at compile time, an immutable array of objects that represents the columns in a model. These objects would contain data like the name of the column, the type, if its the PK, if its nullable etc. The type, name, nullable can all come directly from an instance variables TypeNode without touching annotations. Annotations come into play when you want to attach non-autogeneratable data to a column, aka property. Annotations would allow fields/values to be defined on it, that can also be read at compile time when building out our array of records.

    private record ColumnInfo(T) < ColumnBase, name : String, nilable : Bool, auto : Bool = false, primary : Bool = false, default : T? = nil, type : T.class = T

    protected def self.columns : Array(ColumnBase)
      columns = [] of ColumnBase
      {% begin %}
        {% fields = @type.instance_vars.select { |ivar| ivar.annotation(Granite::Column) } %}
        {% raise "Composite primary keys are not yet supported." if fields.select { |ivar| ann = ivar.annotation(Granite::Column); ann && ann[:primary] }.size > 1 %}
        {% for field in fields %}
          {% type = field.type.union? ? field.type.union_types.reject { |t| t == Nil }.first : field.type %}
          {% col_ann = field.annotation(Granite::Column) %}
          {% auto = col_ann && col_ann[:auto] ? col_ann[:auto] : false %}
          {% primary = col_ann && col_ann[:primary] ? col_ann[:primary] : false %}
          {% auto = col_ann[:auto] == nil && primary %}
          {% raise "Primary key '#{field.name}' of '#{@type.name}' must be nilable." if primary && !field.type.nilable? %}
          columns << ColumnInfo({{type}}).new({{field.stringify}}, {{field.type.nilable?}}, {{auto}}, {{primary}}, {{field.default_value}})
        {% end %}
      {% end %}
      columns
    end

And this is what I came up with. Similar to building out routes, I iterate over the instance variables of the current class, find those with a Granite::Column annotation, then check various annotation fields to see if they are set, otherwise use defaults. All the while generating compile time errors if a user adds two primary keys, or has a non nilable PK.

An example of what the model could look like.

  @[Granite::Model(table: "parents", adapter: "my_db")]
  class Parent < Granite::Base
    @[Granite::Column(primary: true)]
    property id : Int64?

    @[Granite::Column]
    property name : String

    @[Granite::Column(name: "createdAt")]
    property created_at : Time?

    @[Granite::Column(name: "updatedAt")]
    property updated_at : Time?

    # A property that would be read from a request body but _not_ persisted to DB.
    property meta : String
  end

IMHO, much cleaner and not dependent on shard specific macro syntax.

Hope this helps as well.

Creating a JSON API with Athena & Granite

Blacksmoke16 — Wed, 27 Feb 2019 02:04:13 +0000

UPDATE: April 22, 2019 for Athena version 0.6.0
UPDATE: November 24, 2019 for Athena version 0.7.0 and Crystal 0.31.1
UPDATE: February 7, 2020 for Athena version 0.8.0
UPDATE: June 10, 2020 for Athena version 0.9.0
UPDATE: July 11, 2020 for Athena version 0.10.0

Athena

Intro

A ~~few months ago~~ (over a year ago at this point) I set out to create a new web framework, but with a few key differences. I wanted something that would allow for a route's action to be easily documented, tested, and flexible. I also didn't want to have to deal with the boilerplate of converting route/query/body params into their expected types manually all the time in every route. Finally, I wanted to take advantage of Crystal's annotations to provide a simple yet flexible DSL for defining routes.

Taking the all of the frameworks I have experienced into consideration, with big help from Symfony. The outcome of this idea was Athena.

Now, I wanted to write a blog post showing how a JSON API with Athena would look like in an actual app, outside of general documentation.

Tutorial

This tutorial will not cover any front end work (UI/UX). It will just assume that the requests coming to the API are coming from a frontend JS framework or something. Requests are JSON, so it is pretty framework agnostic.

I'll be taking a pretty slow approach, as to make this tutorial applicable to both new crystalers as well as veterans.

Requirements

Crystal installed on your machine. (Latest version as of writing is 0.35.1)
Your HTTP client of preference. I'll just be using cURL
Your IDE/editor of preference.
Your DB of preference. I will be using Postgres.
- (Optional) Docker. Is what I will be running my DB with.

Agenda

Add the ability to:

Register/Login
- Validate users
Create/Read/Update/Delete articles
- Validate articles

Scaffolding the blog

We can utilize the crystal binary to scaffold out our application. This will create a new directory with the given name, with the required files for a crystal app; including the basic directory structure, a shard.yml, and a .gitignore, all auto generated for us. I will go ahead and create this in my home directory, then cd into the newly created directory; ready for the next steps.

$ cd ~/
$ crystal init app blog
$ cd ./blog

Dependencies

I will be using Granite as our ORM of choice to pair with Athena. Start off by adding the following to your shard.yml file.

I will also be requiring the jwt shard to generate JWTs to use as our authentication method of choice.

NOTE: I am using Postgres, and as such am installing the PG shard for use with Granite. If you are using another DB adapter, you will need to install that shard instead.

dependencies:
  granite:
    github: amberframework/granite
    version: 0.21.1
  pg:
    github: will/crystal-pg
    version: 0.21.1
  athena:
    github: athena-framework/athena
    version: 0.10.0
  jwt:
    github: crystal-community/jwt
    version: 1.4.2
  assert:
    github: blacksmoke16/assert
    version: 0.2.0

Then install the required dependencies:

$ shards install

Defining Our Models & Controllers

Our blog will have two models, and two database tables:

User - Stores users that have registered with our blog
Article - A blog post authored by a user.

For the purpose of this tutorial, I will just be executing the raw SQL in Postgres to create the tables. There are some migration shards out there that can automate this; could be a future iteration.

First lets create a new schema to hold our tables, as well as give our DB user access to that database.

I will be running my PG database using docker, with the following compose file:

version: '3.1'
services:
  pg:
    image: postgres:11.2-alpine
    container_name: pg
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=blog_user
      - POSTGRES_PASSWORD=mYAw3s0meB!log
      - POSTGRES_DB=blog
    volumes:
      - pg-data:/var/lib/postgresql/data

volumes:
  pg-data:

CREATE SCHEMA "blog";
ALTER ROLE "blog_user" SET SEARCH_PATH TO "blog";

NOTE: Using Docker is optional, as long as you have a DB to connect to you'll be fine.

Now that our dependencies are installed, we need to require them, as well as setup our DB connection in our blog.cr file. It should look something like:

# Register an adapter to connect to our DB
Granite::Connections << Granite::Adapter::Pg.new(name: "my_blog", url: "postgres://blog_user:mYAw3s0meB!log@localhost:5432/blog?currentSchema=blog")

# Require some standard library things we'll need
require "crypto/bcrypt/password"

# Require our ORM and DB adapter
require "granite"
require "granite/adapter/pg"

# Require Athena
require "athena"

# This will eventually be replaced by Athena's validation component
require "assert"

# Require JWT shard
require "jwt"

module Blog
  VERSION = "0.10.0"

  # Runs the HTTP server with the default settings
  ART.run
end

User Model

Next lets think about what columns would be required for our user:

id : Int64 - auto-generated ID to uniquely identity each user
first_name : String - first name of the user
last_name : String - last name of the user
email : String - email of the user, also used for login. Should be unique for each user
password : String - the user's password
created_at : Time - when the user was created
updated_at : Time - when the user was updated (name/email/password change)
deleted_at : Time - when the user was deleted

Translating this into a SQL statement:

CREATE TABLE "blog"."users"
(
    "id"         BIGSERIAL NOT NULL PRIMARY KEY,
    "first_name"  TEXT      NOT NULL,
    "last_name"  TEXT      NOT NULL,
    "email"      TEXT      NOT NULL,
    "password"   TEXT      NOT NULL,
    "created_at" TIMESTAMP NOT NULL DEFAULT NOW(),
    "updated_at" TIMESTAMP NOT NULL DEFAULT NOW(),
    "deleted_at" TIMESTAMP NULL
);

Now that our table is created, we can move on to create our first model. I will start by making a new directory to store our models; as well as creating our user.cr file.

$ mkdir ./src/models
$ touch ./src/models/user.cr

Using our list as reference I will create the user model.

@[ASRA::ExclusionPolicy(:all)]
class Blog::Models::User < Granite::Base
  include ASR::Serializable
  include Assert

  connection "my_blog"
  table "users"

  column id : Int64, primary: true
  column first_name : String
  column last_name : String
  column email : String
  column password : String
  column created_at : Time
  column updated_at : Time
  column deleted_at : Time?
end

A few things to point out:

The connection macro defines which adapter this model should use to connect to the database. The value passed to the macro is the same as the name set when registering the DB adapter in blog.cr.
I added a Models namespace just to add some organization and help separate the docs. Because of this be sure to add a include Models within the Blog module in blog.cr as well as a require "./models/*".
We'll get back to the include ASR::Serializable and include Assert later.

Let's do a little recap. What have we done so far?

Registered our adapter to connect to our DB.
Required all the needed shards.
Created our User model and table.

Now that all of these beginning steps are done, we can now create our user_controller to hold our routes to create a user.

User Controller

Similarly as before, I will create a new directory to hold our controllers, as well as create our user_controller.cr file.

$ mkdir ./src/controllers
$ touch ./src/controllers/user_controller.cr

class Blog::Controllers::UserController < ART::Controller
end

I'm also namespacing the controllers, so be sure to include Controllers, as well as a require "./controllers/*" in your blog.cr file. The first endpoint I will create will be a POST /user endpoint in order to add users to our database. To do this, add the following code to the UserController class.

@[ART::Post("user")]
def new_user(user : Blog::Models::User) : Blog::Models::User
  user
end

Athena's route definitions are a bit different than what you may be used to. Athena uses Crystal's annotations. The top annotation defines a POST endpoint with the path /user and sets the route's action to the new_user method. However, Athena is not able to automatically provide complex types, such as our User object to our action; we must make use of a ParamConverter to accomplish this. A ParamConverter allows defining custom logic responsible for converting data within a request into another type for the action to use. In this example, convert the request body into an instance of our User model; lets go ahead and create that now.

# Define our converter, register it as a service, inheriting from the base interface struct.
@[ADI::Register]
struct Blog::Converters::RequestBody < ART::ParamConverterInterface
  # Define a customer configuration for this converter.
  # This allows us to provide a `model` field within the annotation
  # in order to define _what_ model should be used on deserialization.
  configuration model : Granite::Base.class

  # :inherit:
  def apply(request : HTTP::Request, configuration : Configuration) : Nil
    # Be sure to handle any possible exceptions here to return more helpful errors to the client.
    raise ART::Exceptions::BadRequest.new "Request body is empty" unless body = request.body.try &.gets_to_end

    # Deserialize the object, based on the type provided in the annotation
    obj = configuration.model.from_json body

    # Run the validations
    obj.validate!

    # Add the resolved object to the request's attributes
    request.attributes.set configuration.name, obj, configuration.model
  rescue ex : Assert::Exceptions::ValidationError
    # Raise a 422 error if the object failed its validations
    raise ART::Exceptions::UnprocessableEntity.new ex.to_s
  end
end

Athena uses a lot of interfaces in order to make types more DI friendly, easier to test, etc. The interface only requires a single method apply(request : HTTP::Request, configuration : Configuration) : Nil whose sole purpose is to apply the conversion logic to the provided request, based on the provided configuration. A converter is simply a struct that inherits from ART::ParamConverterInterface. We'll cover the @[ADI::Register] annotation a bit later.

Our RequestBody converter also makes use of Athena's error handling system. Athena provides a set of common HTTP exceptions inheriting from ART::Exceptions::HTTPException, children of this type are assumed to map to an HTTP error; custom children can also be added. Non HTTPExceptions return a 500 unless rescued as you would normally. By default the exceptions are JSON serialized, but can be customized if so desired.

Most commonly, param converters will want to store the converted values within the request's attributes. The attributes are held within an ART::ParameterBag instance. The ParameterBag is a container for storing key/value pairs; which can be used to store arbitrary data within the context of a request. By default, Athena will look in the request's attributes for a value with the same name as an action argument; this include path/query params, or any custom values stored in it.

Now that we have our converter defined we can go ahead to implement it on our new_user route. Simply apply the annotation, the first argument maps to the name of the action argument the converter should be applied against, while the converter named argument accepts the specific ParamConverter.class we want to use. Any extra configuration for this converter can also be defined. In this case we are specifying that we want to deserialize the request body into a User object.

Since the param converter supplies an actual User model object, we can just call .save in our action to save the given object, then just return the user object. We can also use the ART::View annotation to make our action a bit more REST friendly by having the action return a 201 Created status code instead of the standard 200 OK. Our new_user action now looks like:

@[ART::Post("user")]
@[ART::View(status: :created)]
@[ART::ParamConverter("user", converter: Blog::Converters::RequestBody, model: Blog::Models::User)]
def new_user(user : Blog::Models::User) : Blog::Models::User
  user.save
  user
end

At this point we are now able to create users via our POST /user endpoint. Lets give it a try.

Start the HTTP server.

$ crystal ./src/blog.cr

Lets register a user:

curl --request POST \
  --url http://localhost:3000/user \
  --header 'content-type: application/json' \
  --data '{
  "first_name": "foo",
  "last_name": "bar",
  "email": "fakeemail@domain.com",
  "password": "monkey123"
}'

Success! The user was persisted and now has an id and timestamps.

{
  "id": 1,
  "first_name": "foo",
  "last_name": "bar",
  "email": "fakeemail@domain.com",
  "password": "monkey123",
  "created_at": "2020-07-11T22:42:33Z",
  "updated_at": "2020-07-11T22:42:33Z"
}

However there are a few problems with the current implementation.

What would stop someone from setting their password/name/email as an empty string?
1. Sure we could rely upon the front end for the validation, but that is easy to bypass.
We probably shouldn't be displaying the user's password in cleartext, let alone return it in the response.
What happens if someone were to POST twice with the same email? Since we are using the email as our user facing unique identifier, we should handle this.

Lets go back to our User model to address some of these issues. This is where ASR::Serializable and Assert comes into use.

NOTE: Assert will eventually be moved into the athena-framework organization as an independent component for validation.

We can update our model to look like:

@[ASRA::ExclusionPolicy(:all)]
class Blog::Models::User < Granite::Base
  include ASR::Serializable
  include Assert

  connection "my_blog"
  table "users"

  has_many articles : Article

  @[ASRA::Expose]
  @[ASRA::ReadOnly]
  column id : Int64, primary: true

  @[ASRA::Expose]
  @[Assert::NotBlank]
  column first_name : String

  @[ASRA::Expose]
  @[Assert::NotBlank]
  column last_name : String

  @[ASRA::Expose]
  @[Assert::NotBlank]
  @[Assert::Email(mode: :html5)]
  column email : String

  @[ASRA::IgnoreOnSerialize]
  @[Assert::Size(Range(Int32, Int32), range: 8..25, min_message: "Your password is too short", max_message: "Your password is too long")]
  column password : String

  @[ASRA::Expose]
  @[ASRA::ReadOnly]
  column created_at : Time

  @[ASRA::Expose]
  @[ASRA::ReadOnly]
  column updated_at : Time

  column deleted_at : Time?
end

Also update your new_user action to be:

@[ART::Post("user")]
@[ART::ParamConverter("user", converter: Blog::Converters::RequestBody, model: Blog::Models::User)]
def new_user(user : Blog::Models::User) : Blog::Models::User
  raise ART::Exceptions::Conflict.new "A user with this email already exists." if User.exists? email: user.email
  user.save
  user
end

With these changes we have addressed issues 1 and 3 that we identified earlier. We will solve issue 2 shortly after an explanation of what is going on.

Firstly we included ASR::Serializable and Assert in order to add enhanced serialization and assertion functionality. Next, we added an annotation to the class of our User model. @[ASRA::ExclusionPolicy(:all)]. This annotation alters the overall serialization strategy for the model. In this case, it will only serialize fields that are exposed via @[ASRA::Expose]. This is handy, especially for larger models, to make it easier to only serialize the expected fields, as well as prevent the serialization of other instance variables included via other modules for example.

I also added the @[ASRA::IgnoreOnSerialize] annotation to the password property. This tells Athena's serializer that the password is allowed to be deserialized, but should NOT be serialized.

Next, I have added annotations to expose the fields that we wish to be returned. I also added a @[ASRA::ReadOnly] to the id field and exposed timestamp fields, which prevents that property from being deserialized; since it's managed by the database.

I also added additional annotations to the fields we wish to validate. I am asserting that:

The first_name field is not blank
The last_name field is not blank
The email is not blank AND is a valid email
The password is between 8 and 25 characters long

Finally, I added a User.exists? email: user.email query in the UserController to check if a user exists with the given email, and throw a proper error message if one does.

Lets test it out!

curl --request POST \
  --url http://localhost:3000/user \
  --header 'content-type: application/json' \
  --data '{
  "first_name": "foo",
  "last_name": "",
  "email": "",
  "password": "monkey"
}'

produces the following response:

{
  "code": 422,
  "message": "Validation tests failed: 'last_name' should not be blank, 'email' is not a valid email address, 'email' should not be blank, Your password is too short"
}

Tada! Easy validation of your models. Also, trying to POST a user with an email that was used before now produces this error

{
  "code": 409,
  "message": "A user with this email already exists."
}

Issue 2 can be solved by adding a before_save callback on our model

@[ASRA::ExclusionPolicy(:all)]
class Blog::Models::User < Granite::Base
  ...

  before_save :hash_password

  def hash_password : Nil
    if p = @password
      @password = Crypto::Bcrypt::Password.create(p).to_s
    end
  end
end

This will execute and hash the password before the model is saved.

Auth Controller

At this point we now have a POST /user endpoint that would be paired with a front end form for user registration. But in order for the user to be "logged in" we need to do something to tell the front end that there is an active session. There are a multiple of ways to do this: setting a JWT token in a cookie, generating a session key and storing that in our user table, or returning a JWT token to the front end after receiving a correct username and password for the front end to store in some form of HTML5 storage. For the purposes of this I am going to go with the latter option, and return a JWT token for the front end to handle.

To start, I am going to create a new controller file under our ./src/controllers directory to hold our logic for signing in.

$ touch ./src/controllers/auth_controller.cr

I am also going to take this time to show off some additional features; namely working with the raw HTTP::Request object, and introduce ART::Response.

class Blog::Controllers::AuthController < ART::Controller
  # Type hinting an action argument to `HTTP::Request` will supply the current request object.
  @[ART::Post("login")]
  def login(request : HTTP::Request) : ART::Response
    # Raise an exception if there is no request body
    raise ART::Exceptions::BadRequest.new "Missing request body." unless body = request.body

    # Parse the request body into an HTTP::Params object
    form_data = HTTP::Params.parse body.gets_to_end

    # Handle missing form values
    handle_invalid_auth_credentials unless email = form_data["email"]?
    handle_invalid_auth_credentials unless password = form_data["password"]?

    # Find a user with the given ID
    user = Blog::Models::User.find_by email: email

    # Raise a 401 error if either a user isn't found or the password does not match
    handle_invalid_auth_credentials if !user || !(Crypto::Bcrypt::Password.new(user.password).verify password)

    # If an `ART::Response` is returned then it is used as is for the response,
    # otherwise, like the other endpoints, the response value is by default JSON serialized
    ART::Response.new({token: user.generate_jwt}.to_json, headers: HTTP::Headers{"content-type" => "application/json"})
  end

  private def handle_invalid_auth_credentials : Nil
    # Raise a 401 error if values are missing, or are invalid;
    # this also handles setting an appropiate `www-authenticate` header
    raise ART::Exceptions::Unauthorized.new "Invalid username and/or password.", "Basic realm=\"My Blog\""
  end
end

The raw request object can be obtained by type hinting an action argument as HTTP::Request, Athena will then know to provide the request object when executing the action. We then validate all the required fields are present, and a user was found with the given credentials; otherwise we return a 401 error for the front end to handle.

One thing to note is the return type in this action is ART::Response. At a high level, the implementation of Athena is simply attempting to convert an HTTP::Request into an ART::Response. If an action returns an ART::Response then the request is essentially finished and returned as is, (assuming no listeners alter it further, more on that later). Otherwise, like our other actions, if the return type is not an ART::Response, the resulting value goes through the view layer in order to have that value converted into an ART::Response. By default this is JSON serializing it, but it can be customized if so desired.

Next, we will need to implement the generate_jwt method on our user object, which will look like:

def generate_jwt : String
  JWT.encode({
    "user_id" => @id,
    "exp"     => (Time.utc + 1.week).to_unix,
    "iat"     => Time.utc.to_unix,
  },
    ENV["SECRET"],
    :hs512
  )
end

This method will generate a JWT with a body including:

The id of the user
An expiration date of now + 1 week
The time the JWT was created

I generated a secure string and exported it as an env variable to act as the secret key to sign the token.

$ export SECRET=MY_SECURE_STRING

This is just a simple example, and not representative of how to best use JWT tokens. If this were for real you could include other claims or change the details to best fit your use cases.

After restarting the server and sending a request:

curl --request POST \
  --url http://localhost:3000/login \
  --header 'content-type: application/x-www-form-urlencoded' \
  --data 'email=fakeemail%40domain.com&password=monkey123'

You should get a JSON object back with your JWT token within it. Success! However, if you used invalid credentials you would get a 401 error back.

{
  "code": 401,
  "message": "Invalid username and/or password"
}

Since our imaginary front end will be storing this token in local storage on the browser, we don't really have a use for a /logout endpoint. However, if you wanted to make one you could have it issue a request before deleting the token from the front end. This way you tell your server that a given user logged out if you had other tasks/cleanup to do.

Article Model

At this point we are able to register new users, allow users to login, all the while validating and throwing helpful errors.

The next item on the agenda will be to create our Article model, table, and controller. I'll go a bit faster now as it'll be quite similar as before.

Next lets think about what columns would be required for an article:

id : Int64 - auto-generated ID to uniquely identity each article
user_id : Int64 - the user that authored the article
title : String - title of the article
body : String - the body
created_at : Time - when the article was created
updated_at : Time - when the article was updated
deleted_at : Time - when the article was deleted

Translating this into a SQL statement:

CREATE TABLE "blog"."articles"
(
    "id"         BIGSERIAL NOT NULL PRIMARY KEY,
    "user_id"    BIGINT    NOT NULL REFERENCES "blog"."users",
    "title"      TEXT      NOT NULL,
    "body"       TEXT      NOT NULL,
    "created_at" TIMESTAMP NOT NULL DEFAULT NOW(),
    "updated_at" TIMESTAMP NOT NULL DEFAULT NOW(),
    "deleted_at" TIMESTAMP NULL
);

Now that our table is created, we can move on to create our second model. I will first create the article.cr file.

$ touch ./src/models/article.cr

@[ASRA::ExclusionPolicy(:all)]
class Blog::Models::Article < Granite::Base
  include ASR::Serializable
  include Assert

  connection my_blog
  table "articles"

  @[ASRA::Expose]
  @[ASRA::ReadOnly]
  belongs_to user : User

  @[ASRA::Expose]
  @[ASRA::ReadOnly]
  column id : Int64, primary: true

  @[ASRA::Expose]
  @[Assert::NotBlank]
  @[Assert::NotNil]
  column title : String

  @[ASRA::Expose]
  @[Assert::NotBlank]
  @[Assert::NotNil]
  column body : String

  @[ASRA::Expose]
  @[ASRA::ReadOnly]
  column updated_at : Time

  @[ASRA::Expose]
  @[ASRA::ReadOnly]
  column created_at : Time

  @[ASRA::ReadOnly]
  column deleted_at : Time?
end

This model is very similar to our User model with the main difference being the belongs_to user : User macro. This macro expands and creates the user_id field. It also creates a getter and setter to retrieve and set the related user object. In this case, the person who authored the article.

We'll also want to go back to the User model and add has_many articles : Article to it, below the table definition. This defines a method that would return an array of that user's articles. E.x. articles = user.articles.

Next up, the ArticleController.

Article Controller

$ touch ./src/controllers/article_controller.cr

class Blog::Controllers::ArticleController < ART::Controller
  @[ART::Post("article")]
  @[ART::View(status: :created)]
  @[ART::ParamConverter("article", converter: Blog::Converters::RequestBody, model: Blog::Models::Article)]
  def new_article(article : Blog::Models::Article) : Blog::Models::Article
    article.save
    article
  end
end

Again, this looks nearly the same as the new_user action in our UserController.

Making a request to create an article with a user_id of the id of your user, and the token retrieved earlier:

curl --request POST \
  --url http://localhost:3000/article \
  --header 'content-type: application/json' \
  --header 'authorization: Bearer TOKEN' \
  --data '{
    "user_id": 1,
    "title": "My Athena Blog",
    "body": "Athena makes developing JSON APIs easy!"
}'

Successfully creates the article:

{
  "user_id": 1,
  "id": 1,
  "title": "My Athena Blog",
  "body": "Athena makes developing JSON APIs easy!",
  "updated_at": "2020-07-11T22:57:56Z",
  "created_at": "2020-07-11T22:57:56Z"
}

Great! We can now create articles. However, do you see a problem with this implementation? There is no validation around the user_id, nor is there any authorization to prevent random people from creating articles for any user they want. Lets work on adding some authorization checks into our request flow, utilizing the generated JWT token we got a little while ago when we "logged in".

Authorization

One of the core points of JWT is that once verified, using our secret key and checking the claims in the body, it can be assured that it is a valid token and that we should process the request. Also, since this is a REST API, we'll need to enable CORS to allow our front end to actually make requests to it. We can accomplish the latter by enabling Athena's CORS listener.

We just need to simply define some configuration on how we want the listener to operate, see ART::Config::CORS for additional configuration information. Create a file in the root of your application named athena.yml with the following content:

---
routing:
  cors:
    allow_credentials: true
    allow_origin: 
      - https://api.myblog.com
    allow_methods:
      - GET
      - POST
      - PUT
      - DELETE

Athena uses Athena::EventDispatcher to handle tapping into the request/response life-cycle, as opposed to the more standard HTTP::Handler approach.

When processing a request, Athena emits various events that can be listened on to handle the request early, like the CORS listener, or for adding additional information to the response, like headers/cookies etc. A good example of this would be to tap into when an unhandled exception occurs for logging purposes.

For our goal of authenticating a user, we will create a listener on the Request event. This will be used to validate there is a token present and that it is valid. Lets get started.

First, lets make a new directory to store our handler.

$ mkdir ./src/listeners
$ touch ./src/listeners/security_listener.cr

struct Blog::Listeners::SecurityListener
  # Define the interface to implement the required methods
  include AED::EventListenerInterface

  # Specify that we want to listen on the `Request` event.
  # The value of the has represents this listener's priority;
  # the higher the value the sooner it gets executed.
  def self.subscribed_events : AED::SubscribedEvents
    AED::SubscribedEvents{
      ART::Events::Request => 10,
    }
  end

  # Define a `#call` method scoped to the `Request` event.
  def call(event : ART::Events::Request, _dispatcher : AED::EventDispatcherInterface) : Nil
    # Allow POST user and POST login through since they are public
    # In the future Athena will most likely have a more structured way to handle auth
    if event.request.method == "POST" && {"/user", "/login"}.includes? event.request.path
      return
    end

    # Return a 401 error if the token is missing or malformed
    raise ART::Exceptions::Unauthorized.new "Missing bearer token", "Bearer realm=\"My Blog\"" unless (auth_header = event.request.headers.get?("Authorization").try &.first) && auth_header.starts_with? "Bearer "

    # Get the JWT token from the Bearer header
    token = auth_header.lchop "Bearer "

    begin
      # Validate the token
      body = JWT.decode token, ENV["SECRET"], :hs512
    rescue decode_error : JWT::DecodeError
      # Throw a 401 error if the JWT token is invalid
      raise ART::Exceptions::Unauthorized.new "Invalid token", "Bearer realm=\"My Blog\""
    end
  end
end

Now that our listener is defined, we're faced with some new problems.

How do we tell Athena to use it?
How can we make the rest of our application aware of the currently authenticated user?

Both of these problems are solved via another feature of Athena, dependency injection (DI). Athena uses Athena::DependencyInjection to make sharing useful objects easy. While I'm going to cover the main points of DI in this article, see Dependency Injection in Crystal, in addition to the API docs within the shard, for a more detailed example of it in action.

A service container contains instances of various useful object, aka services. These services can then be supplied to other services without having to manually instantiate everything. It also allows for types to be tested more easily since they can depend on abstractions (interfaces) versus concrete types. In our case it'll allow us to define a service that will store the currently authenticated user in order to have access to it in the rest of the application.

First let's define a type to store the user, aka UserStorage. We'll make a new directory to store our services that don't fit better anywhere else.

$ mkdir ./src/services
$ touch ./src/services/user_storeage.cr

# The ADI::Register annotation tells the DI component how this service should be registered
@[ADI::Register]
class Blog::UserStorage
  # Use a ! property since they'll always be a user defined in our use case.
  #
  # It also provides a `user?` getter in cases where it might not be.
  property! user : Blog::Models::User
end

Since we defined this as a class, it makes it so the same instance is injected into each type, i.e. the user initially set will remain set until the request is finished. A struct on the other hand would cause a copy of the service to be injected.

Be sure to require our new directory in src/blog.cr.

Now lets update our security listener, I omitted the lines that didn't change.

# Define and register a listener to handle authenticating requests.
@[ADI::Register]
struct Blog::Listeners::SecurityListener
  # Define the interface to implement the required methods
  include AED::EventListenerInterface

  # Define our initializer for DI to inject the user storage.
  def initialize(@user_storage : UserStorage); end

  # Define a `#call` method scoped to the `Request` event.
  def call(event : ART::Events::Request, _dispatcher : AED::EventDispatcherInterface) : Nil
    ...

    # Set the user in user storage
    @user_storage.user = Blog::Models::User.find! body[0]["user_id"]
  end
end

By simply annotating the type with @[ADI::Register], Athena handles "wiring" everything up for us. Our required UserStorage dependency is automatically resolved and injected based on type restriction. The listener is also registered automatically since it implements AED::EventListenerInterface, via ADI.auto_configure.

Now that we are authenticating the requests, we can move onto adding the ability to read/update/delete our articles.

Our ArticleController now looks like:

# The `ART::Prefix` annotation will add the given prefix to each route in the controller.
# We also register the controller itself as a service in order to allow injecting our `UserStorage` object.
# NOTE: The controller service must be declared as public.  In the future this will happen behind the scenes
# but for now it cannot be done automatically.
@[ART::Prefix("article")]
@[ADI::Register(public: true)]
class Blog::Controllers::ArticleController < ART::Controller
  # Define our initializer for DI
  def initialize(@user_storage : Blog::UserStorage); end

  @[ART::Post(path: "")]
  @[ART::View(status: :created)]
  @[ART::ParamConverter("article", converter: Blog::Converters::RequestBody, model: Blog::Models::Article)]
  def new_article(article : Blog::Models::Article) : Blog::Models::Article
    # Set the owner of the article to the currently authenticated user
    article.user = @user_storage.user
    article.save
    article
  end

  @[ART::Get(path: "")]
  def get_articles : Array(Blog::Models::Article)
    # We are also using the user in UserStorage as an additional conditional in our query when fetching articles
    # this allows us to only returns articles that belong to the current user.
    Blog::Models::Article.where(:deleted_at, :neq, nil).where(:user_id, :eq, @user_storage.user.id).select
  end

  @[ART::Put(path: "")]
  @[ART::ParamConverter("article", converter: Blog::Converters::RequestBody, model: Blog::Models::Article)]
  def update_article(article : Blog::Models::Article) : Blog::Models::Article
    article.save
    article
  end

  @[ART::Get("/:id")]
  @[ART::ParamConverter("article", converter: Blog::Converters::DB, model: Blog::Models::Article)]
  def get_article(article : Blog::Models::Article) : Blog::Models::Article
    article
  end

  @[ART::Delete("/:id")]
  @[ART::ParamConverter("article", converter: Blog::Converters::DB, model: Blog::Models::Article)]
  def delete_article(article : Blog::Models::Article) : Nil
    article.deleted_at = Time.utc
    article.save
  end
end

These additional methods will allow for:

Listing all the current user's articles
Updating an article
Getting a specific article
Deleting a specific article

The latter two are making use of a new converter; DB. This will do a DB query to find a record of the provided type with the provided id, otherwise returns a 404 error. The code for that is as follows:

@[ADI::Register]
struct Blog::Converters::DB < ART::ParamConverterInterface
  # Define a customer configuration for this converter.
  # This allows us to provide a `model` field within the annotation
  # in order to define _what_ model should be queried for.
  configuration model : Granite::Base.class

  # :inherit:
  #
  # Be sure to handle any possible exceptions here to return more helpful errors to the client.
  def apply(request : HTTP::Request, configuration : Configuration) : Nil
    # Grab the `id` path parameter from the request's attributes
    primary_key = request.attributes.get "id", Int32

    # Raise a 404 if a record with the provided ID does not exist
    raise ART::Exceptions::NotFound.new "An item with the provided ID could not be found" unless model = configuration.model.find primary_key

    # Set the resolved model within the request's attributes
    # with a key matching the name of the argument within the converter annotation
    request.attributes.set configuration.name, model, configuration.model
  end
end

Updating DB models can be a bit tricky in some cases due to how Crystal handles deserialization. In other frameworks it would be required to include ALL the properties of the model in the PUT body, even those managed by the database that should not be editable, such as the id or timestamps. Athena's serializer includes the concept of Object Constructors; which determine how a new object is constructed during deserialization. In our case, we could define a custom constructor that would source the object from the database, making it so we don't need to include the timestamps, or other non-editable properties within our PUT request.

Within request_body_converter.cr add the following code:

# Define a custom `ASR::ObjectConstructorInterface` to allow sourcing the model from the database
# as part of `PUT` requests, and if the type is a `Granite::Base`.
#
# Alias our service to `ASR::ObjectConstructorInterface` so ours gets injected instead.
@[ADI::Register(alias: ASR::ObjectConstructorInterface)]
class DBObjectConstructor
  include Athena::Serializer::ObjectConstructorInterface

  # Inject `ART::RequestStore` in order to have access to the current request.
  # Also inject `ASR::InstantiateObjectConstructor` to act as our fallback constructor.
  def initialize(@request_store : ART::RequestStore, @fallback_constructor : ASR::InstantiateObjectConstructor); end

  # :inherit:
  def construct(navigator : ASR::Navigators::DeserializationNavigatorInterface, properties : Array(ASR::PropertyMetadataBase), data : ASR::Any, type)
    # Fallback on the default object constructor if the type is not a `Granite` model.
    unless type <= Granite::Base
      return @fallback_constructor.construct navigator, properties, data, type
    end

    # Fallback on the default object constructor if the current request is not a `PUT`.
    unless @request_store.request.method == "PUT"
      return @fallback_constructor.construct navigator, properties, data, type
    end

    # Lookup the object from the database; assume the object has an `id` property.
    object = type.find data["id"].as_i

    # Return a `404` error if no record exists with the given ID.
    raise ART::Exceptions::NotFound.new "An item with the provided ID could not be found." unless object

    # Apply the updated properties to the retrieved record
    object.apply navigator, properties, data

    # Return the object
    object
  end
end

# Make the compiler happy when we want to allow any Granite model to be deserializable.
class Granite::Base
  include ASR::Model
end

This type allows us to source the original object from the database, then apply the updated values to it as opposed to creating a whole new object from the request body. The DB logic is only applied to Granite::Base types on PUT requests, everything else uses the default behavior of creating a new object with based on the data within the request body.

We can then update our RequestBody converter to look like:

# Define our converter, register it as a service, inheriting from the base interface struct.
@[ADI::Register]
struct Blog::Converters::RequestBody < ART::ParamConverterInterface
  # Define a custom configuration for this converter.
  # This allows us to provide a `model` field within the annotation
  # in order to define _what_ model should be used on deserialization.
  configuration model : Granite::Base.class

  # Inject the Serializer instance into our converter.
  def initialize(@serializer : ASR::SerializerInterface); end

  # :inherit:
  def apply(request : HTTP::Request, configuration : Configuration) : Nil
    # Be sure to handle any possible exceptions here to return more helpful errors to the client.
    raise ART::Exceptions::BadRequest.new "Request body is empty." unless body = request.body.try &.gets_to_end

    # Deserialize the object, based on the type provided in the annotation.
    object = @serializer.deserialize configuration.model, body, :json

    # Run the validations.
    object.validate!

    # Add the resolved object to the request's attributes.
    request.attributes.set configuration.name, object, configuration.model
  rescue ex : Assert::Exceptions::ValidationError
    # Return a `422` error if the object failed its validations.
    raise ART::Exceptions::UnprocessableEntity.new ex.to_s
  end
end

From here you would want to update ArticleController#update_article with some ACL logic, such as:

  @[ART::Put(path: "")]
  @[ART::ParamConverter("article", converter: Blog::Converters::RequestBody, model: Blog::Models::Article)]
  def update_article(article : Blog::Models::Article) : Blog::Models::Article
    # Ensure that a user cannot edit someone else's article
    raise ART::Exceptions::Forbidden.new "Only the author of the article can edit it." if article.user_id != @user_storage.user.id
    article.save
    article
  end

And we're done! I hope this was a good introduction to Athena and its features. If there is anything in specific you would like to see regarding Granite/Athena, or if I missed something, feel free to leave a comment or join the Athena Gitter channel.

The full code for this tutorial is available on GitHub.

DEV Community: Blacksmoke16

oq - A portable/performant jq wrapper Part 2

OQ Revisited

Methodology

Setup

Results

Simple

Jeopardy.json

YAML => XML/JSON

XML => JSON

Conclusion

Appendix

Simple

jq

yq (go)

yq (python)

oq

Jeopardy.json

jq

yq (go)

yq (python)

oq

YAML => XML/JSON

jq

yq (go)

JSON

yq (python)

JSON

XML

oq

JSON

XML

XML => JSON

jq

yq (go)

yq (python)

oq

Utilizing Macros & Annotations in a Web Framework (Part 2)

Registering Routes

Annotation Data at Runtime

Athena 0.9.0

Athena 0.9.0

DI Overhaul

Action Handling Refactor

Param Converter Refactor

Route Collision Detection

Other Minor Improvements

The Rebirth of Athena

Rebirth

Organizational

Event Dispatcher

Dependency Injection (DI)

ErrorRendererInterface

View Layer

ART::Response

ART::Events::View

Routing

QueryParams

Macro DSL

Access Raw HTTP::Request

The Future

Object Validation with Assert

Assert

Introduction

Features

Usage

Groups

Custom Assertions & Generics

Example Use Cases

Web Framework

ORM Models

Conclusion

Dependency Injection in Crystal

Dependency Injection (DI)

Manager

Problem Statement

Solution

The Code

Why It Matters

Alternate solutions

Access Raw `HTTP::Request`

Also, `Crylog` has been deprecated in favor of the standard libraries' Log module.