DEV Community: Benedikt Deicke

Digging Deeper into Ruby Templating: The Parser

Benedikt Deicke — Tue, 30 Jul 2019 09:29:52 +0000

Today, we continue our journey into Ruby Templating. With the lexer in place, let’s move on to the next step: The parser.

Last time, we looked at string interpolation and subsequently, dived into creating our own templating language. We started by implementing a lexer that reads a template and converts it into a stream of tokens. Today, we’ll implement the accompanying parser. We will also dip our toes into a bit of language theory.

Here we go!

Abstract Syntax Trees

Let’s look back to our simple example template for Welcome to {{name}}. After using the lexer to tokenize the string, we get a list of tokens like this.

Magicbars::Lexer.tokenize("Welcome to {{name}}")
# => [[:CONTENT, "Welcome to "], [:OPEN_EXPRESSION], [:IDENTIFIER, "name"], [:CLOSE]]

Ultimately, we want to evaluate the template and replace the expression with real values. To make things a bit more challenging, we also want to evaluate complex block expressions, allowing for repetition and conditionals.

To do this, we have to generate an abstract syntax tree (AST) that describes the logical structure of the template. The tree consists of nodes that may reference other nodes or store additional data from the tokens.

For our simple example, the desired abstract syntax tree looks like this:

Defining a Grammar

To define the grammar, let's start with the theoretical basis of a language. Like other programming languages, our templating language is a context-free language and therefore can be described by a context-free grammar. (Don’t let the mathematical notations in the detailed Wikipedia descriptions scare you away. The concept is pretty straight forward, and there are more developer-friendly ways to notate a grammar.)

A context-free grammar is a set of rules that describe how all possible strings of a language are constructed. Let’s look at the grammar for our templating language in EBNF notation:

template = statements;
statements = { statement };
statement = CONTENT | expression | block_expression;
expression = OPEN_EXPRESSION, IDENTIFIER, arguments, CLOSE;
block_expression = OPEN_BLOCK, IDENTIFIER, arguments, CLOSE, statements, [ OPEN_INVERSE, CLOSE, statements ], OPEN_END_BLOCK, IDENTIFIER, CLOSE;
arguments = { IDENTIFIER };

Each assignment defines a rule. The rule’s name is on the left and a bunch of other rules (lower case) or tokens (upper case) from our lexer are on the right. Rules and tokens can be concatenated using commas , or alternated using the pipe | symbol. Rules and tokens inside of curly braces { ... } might be repeated several times. When they are inside of brackets [ ... ], they are considered optional.

The above grammar is a concise way to describe that a template consists of statements. A statement is either a CONTENT token, an expression, or a block expression. An expression is an OPEN_EXPRESSION token, followed by an IDENTIFIER token, followed by arguments, followed by a CLOSE token. And a block expression is the perfect example of why it’s better to use a notation like the one above instead of trying to describe it with a natural language.

There are tools that automatically generate parsers from grammar definitions like the one above. But in true Ruby Magic tradition, let's have some fun and build the parser ourselves, hopefully learning a thing or two in the process.

Building the Parser

With the language theory aside, let’s jump into actually building the parser. Let’s start with an even more minimal, but still valid, template: Welcome to Ruby Magic. This template doesn’t have any expressions and the list of tokens consists of just one element. Here’s what it looks like:

[[:CONTENT, "Welcome to Ruby Magic"]]

First, we set up our parser class. It looks like this:

module Magicbars
  class Parser
    def self.parse(tokens)
      new(tokens).parse
    end

    attr_reader :tokens

    def initialize(tokens)
      @tokens = tokens
    end

    def parse
      # Parsing starts here
    end
  end
end

The class takes an array of tokens and stores it. It only has one public method called parse that converts the tokens into an AST.

Looking back at our grammar, the top-most rule is template. That implies that parse, at the start of the parsing process, will return a Template node.

Nodes are simple classes with no behavior of their own. They just connect other nodes or store some values from the tokens. Here’s what the Template node looks like:

module Magicbars
  module Nodes
    class Template
      attr_reader :statements

      def initialize(statements)
        @statements = statements
      end
    end
  end
end

To make our example work, we also need a Content node. It just stores the text content ("Welcome to Ruby Magic") from the token.

module Magicbars
  module Nodes
    class Content
      attr_reader :content

      def initialize(content)
        @content = content
      end
    end
  end
end

Next, let’s implement the parse method to create an instance of Template and an instance of Content and connect them up correctly.

def parse
  Magicbars::Nodes::Template.new(parse_content)
end

def parse_content
  return unless tokens[0][0] == :CONTENT

  Magicbars::Nodes::Content.new(tokens[0][1])
end

When we run the parser, we get the correct result:

Magicbars::Parser.parse(tokens)
# => #<Magicbars::Nodes::Template:0x00007fe90e939410 @statements=#<Magicbars::Nodes::Content:0x00007fe90e939578 @content="Welcome to Ruby Magic">>

Admittedly, this only works for our simple example that only has one content node. Let’s switch to a more complex example that actually includes an expression: Welcome to {{name}}.

Magicbars::Lexer.tokenize("Welcome to {{name}}")
# => [[:CONTENT, "Welcome to "], [:OPEN_EXPRESSION], [:IDENTIFIER, "name"], [:CLOSE]]

For this, we need an Expression node and an Identifier node. The Expression node stores the identifier as well as any arguments (which, according to the grammar, are an array of zero or more Identifier nodes). As with the other nodes, there’s not much to see here.

module Magicbars
  module Nodes
    class Expression
      attr_reader :identifier, :arguments

      def initialize(identifier, arguments)
        @identifier = identifier
        @arguments = arguments
      end
    end
  end
end

module Magicbars
  module Nodes
    class Identifier
      attr_reader :value

      def initialize(value)
        @value = value.to_sym
      end
    end
  end
end

With the new nodes in place, let’s modify the parse method to handle both regular content as well as expressions. We do that by introducing a parse_statements method that just keeps on calling parse_statement as long as it returns a value.

def parse
  Magicbars::Nodes::Template.new(parse_statements)
end

def parse_statements
  results = []

  while result = parse_statement
    results << result
  end

  results
end

parse_statement itself first calls parse_content and if that doesn’t return any value, it calls parse_expression.

def parse_statement
  parse_content || parse_expression
end

Have you noticed that the parse_statement method is starting to look very similar to the statement rule in the grammar? This is where taking the time to explicitly write up the grammar beforehand helps a lot to ensure that we’re on the right path.

Next, let’s modify the parse_content method so that it doesn’t only look at the first token. We do this by introducing an additional @position instance variable in the initializer and use it to fetch the current token.

attr_reader :tokens, :position

def initialize(tokens)
  @tokens = tokens
  @position = 0
end

# ...

def parse_content
  return unless token = tokens[position]
  return unless token[0] == :CONTENT

  @position += 1

  Magicbars::Nodes::Content.new(token[1])
end

The parse_content method now looks at the current token and checks its type. If it’s a CONTENT token, it increments the position (because the current token was successfully parsed) and uses the token’s content to create the Content node. If there is no current token (because we’re at the end of the tokens) or the type doesn’t match, the method exits early and returns nil.

With the improved parse_content method in place, let’s tackle the new parse_expression method.

def parse_expression
  return unless token = tokens[position]
  return unless token[0] == :OPEN_EXPRESSION

  @position += 1

  identifier = parse_identifier
  arguments = parse_arguments

  if !tokens[position] || tokens[position][0] != :CLOSE
    raise "Unexpected token #{tokens[position][0]}. Expected :CLOSE."
  end

  @position += 1

  Magicbars::Nodes::Expression.new(identifier, arguments)
end

First, we check that there is a current token and that its type is OPEN_EXPRESSION. If that’s the case, we advance to the next token and parse the identifier as well as the arguments by calling parse_identifier and parse_arguments, respectively. Both methods will return the respective nodes and advance the current token. When that’s done, we ensure that the current token exists and is a :CLOSE token. If it is not, we raise an error. Otherwise, we advance the position one last time, before returning the newly created Expression node.

At this point, we see some patterns emerge. We’re advancing to the next token several times and we’re also checking that there is a current token and its type. Because the code for that is a bit cumbersome, let’s introduce two helper methods.

def expect(*expected_tokens)
  upcoming = tokens[position, expected_tokens.size]

  if upcoming.map(&:first) == expected_tokens
    advance(expected_tokens.size)
    upcoming
  end
end

def advance(offset = 1)
  @position += offset
end

The expect method takes a variable number of token types and checks them against the next tokens in the token stream. If they all match, it advances past the matching tokens and returns them. The advance method just increments the @position instance variable by the given offset.

For cases where there's no flexibility regarding the next expected token, we also introduce a method that raises a nice error message when the token doesn’t match.

def need(*required_tokens)
  upcoming = tokens[position, required_tokens.size]
  expect(*required_tokens) or raise "Unexpected tokens. Expected #{required_tokens.inspect} but got #{upcoming.inspect}"
end

By using these helper methods, parse_content and parse_expression are now cleaner and more readable.

def parse_content
  if content = expect(:CONTENT)
    Magicbars::Nodes::Content.new(content[0][1])
  end
end

def parse_expression
  return unless expect(:OPEN_EXPRESSION)

  identifier = parse_identifier
  arguments = parse_arguments

  need(:CLOSE)

  Magicbars::Nodes::Expression.new(identifier, arguments)
end

Finally, let’s also look at parse_identifier and parse_arguments. Thanks to the helper methods, the parse_identifier method is as simple as the parse_content method. The only difference is that it returns another node type.

def parse_identifier
  if identifier = expect(:IDENTIFIER)
    Magicbars::Nodes::Identifier.new(identifier[0][1])
  end
end

When implementing the parse_arguments method, we noticed that it’s almost identical to the parse_statements method. The only difference is that it calls parse_identifier instead of parse_statement. We can get rid of the duplicated logic by introducing another helper method.

def repeat(method)
  results = []

  while result = send(method)
    results << result
  end

  results
end

The repeat method uses send to call the given method name until it no longer returns a node. Once that happens, the collected results (or just an empty array) are returned. With this helper in place, both parse_statements and parse_arguments become one-line methods.

def parse_statements
  repeat(:parse_statement)
end

def parse_arguments
  repeat(:parse_identifier)
end

With all these changes in place, let’s try and parse the token stream:

Magicbars::Parser.parse(tokens)
# => #<Magicbars::Nodes::Template:0x00007f91a602f910
#     @statements=
#      [#<Magicbars::Nodes::Content:0x00007f91a58802c8 @content="Welcome to ">,
#       #<Magicbars::Nodes::Expression:0x00007f91a602fcd0
#        @arguments=[],
#        @identifier=
#         #<Magicbars::Nodes::Identifier:0x00007f91a5880138 @value=:name>  >

It’s a bit hard to read but it’s, in fact, the correct abstract syntax tree. The Template node has a Content and an Expression statement. The Content node’s value is "Welcome to " and the Expression node’s identifier is the Identifier node with :name as its value.

Parsing Block Expressions

To complete our parser implementation, we still have to implement parsing of block expressions. As a reminder, here’s the template we want to parse:

Welcome to {{name}}!

{{#if subscribed}}
  Thank you for subscribing to our mailing list.
{{else}}
  Please sign up for our mailing list to be notified about new articles!
{{/if}}

Your friends at {{company_name}}

To do this, let’s first introduce a BlockExpression node. While this node stores a bit more data, it doesn’t do anything else and therefore isn’t very exciting.

module Magicbars
  module Nodes
    class BlockExpression
      attr_reader :identifier, :arguments, :statements, :inverse_statements

      def initialize(identifier, arguments, statements, inverse_statements)
        @identifier = identifier
        @arguments = arguments
        @statements = statements
        @inverse_statements = inverse_statements
      end
    end
  end
end

Like the Expression node, it stores the identifier as well as any arguments. Additionally, it also stores the statements of the block and of the inverse block.

Looking back at the grammar, we notice that to parse block expressions, we have to amend the parse_statements method with a call to parse_block_expression. It now looks just like the rule in the grammar.

def parse_statement
  parse_content || parse_expression || parse_block_expression
end

The parse_block_expression method itself is a bit more complex. But thanks to our helper methods, it’s still quite readable.

def parse_block_expression
  return unless expect(:OPEN_BLOCK)

  identifier = parse_identifier
  arguments = parse_arguments

  need(:CLOSE)

  statements = parse_statements

  if expect(:OPEN_INVERSE, :CLOSE)
    inverse_statements = parse_statements
  end

  need(:OPEN_END_BLOCK)

  if identifier.value != parse_identifier.value
    raise("Error. Identifier in closing expression does not match identifier in opening expression")
  end

  need(:CLOSE)

  Magicbars::Nodes::BlockExpression.new(identifier, arguments, statements, inverse_statements)
end

The first part is very similar to the parse_expression method. It parses the opening block expression with the identifier and the arguments. Afterward, it calls parse_statements to parse the inside of the block.

Once that’s done, we check for an {{else}} expression, identified by an OPEN_INVERSE token followed by a CLOSE token. If both tokens are found, we call parse_statements again to parse the inverse block. Otherwise, we just skip that part entirely.

As a final thing, we ensure that there is an end block expression using the same identifier as the open block expression. If the identifiers don’t match, we raise an error. Otherwise, we create a new BlockExpression node and return it.

Calling the parser with the tokens of the advanced block expression template will return the AST for the template. I’ll not include the example output here, as it’s barely readable. Instead, here’s a visual representation of the generated AST.

Because we’re calling parse_statements inside of parse_block_expression, both the block and the inverse block may include more expressions, block expressions, as well as regular content.

The Journey Continues…

We made decent progress with our journey towards implementing our own templating language. After a short dip into language theory, we defined a grammar for our templating language and used it to implement a parser for it from scratch.

With both the lexer and the parser in place, we’re only missing an interpreter to generate the interpolated string from our template. We’ll cover this part in an upcoming edition of RubyMagic. Subscribe to the Ruby Magic mailinglist, to get alerted when it comes out.

Guest writer Benedikt Deicke is a software engineer and co-founder of Userlist.io. On the side, he’s writing a book about building SaaS applications in Ruby on Rails. You can reach out to Benedikt via Twitter.

Learning by building, a Background Processing System in Ruby

Benedikt Deicke — Wed, 10 Apr 2019 06:52:14 +0000

In today's post, we are going to implement a naive background processing system for fun! We might learn some things along the way as a peek into the internals of popular background processing systems like Sidekiq. The product of this fun is by no means intended for production use.

Let’s imagine we have a task in our application that loads one or more websites and extracts their titles. As we don’t have any influence on the performance of these websites, we’d like to perform the task outside our main thread (or the current request—if we’re building a web application), but in the background.

Encapsulating a Task

Before we get into background processing, let’s build a service object to perform the task at hand. We’ll use OpenURI and Nokogiri to extract the contents of the title tag.

require 'open-uri' 
require 'nokogiri' 

class TitleExtractorService
  def call(url)
    document = Nokogiri::HTML(open(url))
    title = document.css('html > head > title').first.content
    puts title.gsub(/[[:space:]]+/, ' ').strip
  rescue
    puts "Unable to find a title for #{url}" 
  end
end

Calling the service prints the title of the given URL.

TitleExtractorService.new.call('https://appsignal.com')
# AppSignal: Application Performance Monitoring for Ruby on Rails and Elixir

This works as expected, but let’s see if we can improve the syntax a little to make it look and feel a bit more like other background processing systems. By creating a Magique::Worker module, we can add some syntactic sugar to the service object.

module Magique
  module Worker
    def self.included(base)
      base.extend(ClassMethods)
    end

    module ClassMethods
      def perform_now(*args)
        new.perform(*args)
      end
    end

    def perform(*)
      raise NotImplementedError
    end
  end
end

The module adds a perform method to the worker instance and a perform_now method to the worker class to make the invocation a bit better.

Let’s include the module into our service object. While we’re at it, let’s also rename it to TitleExtractorWorker and change the call method to perform.

class TitleExtractorWorker
  include Magique::Worker

  def perform(url)
    document = Nokogiri::HTML(open(url))
    title = document.css('html > head > title').first.content
    puts title.gsub(/[[:space:]]+/, ' ').strip
  rescue
    puts "Unable to find a title for #{url}"
  end
end

The invocation still has the same result, but it’s a bit clearer what's going on.

TitleExtractorWorker.perform_now('https://appsignal.com')
# AppSignal: Application Performance Monitoring for Ruby on Rails and Elixir

Implementing Asynchronous Processing

Now that we have the title extraction working, we can grab all titles from past Ruby Magic articles. To do this, let’s assume we have a RUBYMAGIC constant with a list of all the URLs of past articles.

RUBYMAGIC.each do |url|
  TitleExtractorWorker.perform_now(url)
end

# Unraveling Classes, Instances and Metaclasses in Ruby | AppSignal Blog
# Bindings and Lexical Scope in Ruby | AppSignal Blog
# Building a Ruby C Extension From Scratch | AppSignal Blog
# Closures in Ruby: Blocks, Procs and Lambdas | AppSignal Blog
# ...

We get the titles of past articles, but it takes a while to extract them all. That’s because we wait until each request is completed before moving on to the next one.

Let’s improve that by introducing a perform_async method to our worker module. To speed things up, it creates a new thread for each URL.

module Magique
  module Worker
    module ClassMethods
      def perform_async(*args)
        Thread.new { new.perform(*args) }
      end
    end
  end
end

After changing the invocation to TitleExtractorWorker.perform_async(url), we get all the titles almost at once. However, this also means that we’re opening more than 20 connections to the Ruby Magic blog at once. (Sorry for messing with your blog, folks! 😅)

If you’re following along with your own implementation and testing this outside of a long-running process (like a web server), don’t forget to add something like loop { sleep 1 } to the end of your script to make sure the process doesn’t immediately terminate.

Queueing up Tasks

With the approach of creating a new thread for every invocation, we’ll eventually hit resource limits (both on our side and on the websites we are accessing). As we’d like to be nice citizens, let’s change the implementation to something that is asynchronous but doesn’t feel like a denial-of-service attack.

A common way to solve this problem is to use the producer/consumer pattern. One or more producers push tasks onto a queue while one or more consumers take tasks from the queue and process them.

A queue is basically a list of elements. In theory, a simple array would do the job. However, as we’re dealing with concurrency, we need to make sure that only one producer or consumer can access the queue at a time. If we aren’t careful about this, things will end in chaos—just like two people trying to squeeze through a door at once.

This problem is known as the producer-consumer problem and there are multiple solutions to it. Luckily, it is a very common problem and Ruby ships with a proper Queue implementation that we can use without having to worry about thread synchronization.

To use it, let’s make sure both producers and consumers can access the queue. We do this by adding a class method to our Magique module and assigning an instance of Queue to it.

module Magique
  def self.backend
    @backend
  end

  def self.backend=(backend)
    @backend = backend
  end
end

Magique.backend = Queue.new

Next, we change our perform_async implementation to push a task onto the queue instead of creating its own new thread. A task is represented as a hash including a reference to the worker class as well as the arguments passed to the perform_async method.

module Magique
  module Worker
    module ClassMethods
      def perform_async(*args)
        Magique.backend.push(worker: self, args: args)
      end
    end
  end
end

With that, we’re done with the producer side of things. Next, let’s take a look at the consumer side.

Each consumer is a separate thread that takes tasks from the queue and performs them. Instead of stopping after one task, like the thread, the consumer then takes another task from the queue and performs it, and so on. Here’s a basic implementation of a consumer called Magique::Processor. Each processor creates a new thread that loops infinitely. For every iteration, it tries to grab a new task from the queue, creates a new instance of the worker class, and calls its perform method with the given arguments.

module Magique
  class Processor
    def self.start(concurrency = 1)
      concurrency.times { |n| new("Processor #{n}") }
    end

    def initialize(name)
      thread = Thread.new do
        loop do
          payload = Magique.backend.pop
          worker_class = payload[:worker]
          worker_class.new.perform(*payload[:args])
        end
      end

      thread.name = name
    end
  end
end

In addition to the processing loop, we add a convenience method called Magique::Processor.start. This allows us to spin up multiple processors at once. While naming the thread isn’t really necessary, it will allow us to see if things are actually working as expected.

Let’s adjust the output of our TitleExtractorWorker to include the name of the current thread.

puts "[#{Thread.current.name}] #{title.gsub(/[[:space:]]+/, ' ').strip}"

To test our background processing setup, we first need to spin up a set of processors before enqueueing our tasks.

Magique.backend = Queue.new
Magique::Processor.start(5)

RUBYMAGIC.each do |url|
  TitleExtractorWorker.perform_async(url)
end

# [Processor 3] Bindings and Lexical Scope in Ruby | AppSignal Blog
# [Processor 4] Building a Ruby C Extension From Scratch | AppSignal Blog
# [Processor 1] Unraveling Classes, Instances and Metaclasses in Ruby | AppSignal Blog
# [Processor 0] Ruby's Hidden Gems, StringScanner | AppSignal Blog
# [Processor 2] Fibers and Enumerators in Ruby: Turning Blocks Inside Out | AppSignal Blog
# [Processor 4] Closures in Ruby: Blocks, Procs and Lambdas | AppSignal Blog
# ...

When this is run, we still get the titles of all articles. While it’s not as fast as just using a separate thread for every task, it’s still faster than the initial implementation that had no background processing. Thanks to the added processor names, we can also confirm that all processors are working through the queue. By tweaking the number of concurrent processors, it’s possible to find a balance between processing speed and existing resource limitations.

Expanding to Multiple Processes and Machines

So far, the current implementation of our background processing system works well enough. It’s still limited to the same process, though. Resource-hungry tasks will still affect the performance of the entire process. As a final step, let’s look at distributing the workload across multiple processes and maybe even multiple machines.

The queue is the only connection between producers and consumers. Right now, it’s using an in-memory implementation. Let’s take more inspiration from Sidekiq and implement a queue using Redis.

Redis has support for lists that allow us to push and fetch tasks from. Additionally, the Redis Ruby gem is thread-safe and the Redis commands to modify lists are atomic. These properties make it possible to use it for our asynchronous background processing system without running into synchronization problems.

Let’s create a Redis backed queue that implements the push and shift methods just like the Queue we used previously.

require 'json'
require 'redis'

module Magique
  module Backend
    class Redis
      def initialize(connection = ::Redis.new)
        @connection = connection
      end

      def push(job)
        @connection.lpush('magique:queue', JSON.dump(job))
      end

      def shift
        _queue, job = @connection.brpop('magique:queue')
        payload = JSON.parse(job, symbolize_names: true)
        payload[:worker] = Object.const_get(payload[:worker])
        payload
      end
    end
  end
end

As Redis doesn’t know anything about Ruby objects, we have to serialize our tasks into JSON before storing them in the database using the lpush command that adds an element to the front of the list.

To fetch a task from the queue, we’re using the brpop command, which gets the last element from a list. If the list is empty, it’ll block until a new element is available. This is a nice way to pause our processors when no tasks are available. Finally, after getting a task out of Redis, we have to look up the real Ruby class based on the name of the worker using Object.const_get.

As a final step, let’s split things up into multiple processes. On the producer side of things, the only thing we have to do is change the backend to our newly implemented Redis queue.

# ...

Magique.backend = Magique::Backend::Redis.new

RUBYMAGIC.each do |url|
  TitleExtractorWorker.perform_async(url)
end

On the consumer side of things, we can get away with a few lines like this:

# ...

Magique.backend = Magique::Backend::Redis.new
Magique::Processor.start(5)

loop { sleep 1 }

When executed, the consumer process will wait for new work to arrive in the queue. Once we start the producer process that pushes tasks into the queue, we can see that they get processed immediately.

Enjoy Responsibly and Don’t Use This in Production

While we kept it far from a real world setup you would use in production (so don't!), we took a few steps in building a background processor. We started by making a process run as a background service. Then we made it async and used Queue to solve the producer-consumer problem. Then we expanded the process to multiple processes or machines using Redis rather then an in-memory implementation.

As mentioned before, this is a simplified implementation of a background processing system. There are a lot of things missing and not explicitly dealt with. These include (but are not limited to) error handling, multiple queues, scheduling, connection pooling, and signal handling.

Nonetheless, we had fun writing this and hope you enjoyed a peek under the hood of a background processing system. Perhaps you even took away a thing or two.

Guest writer Benedikt Deicke is a software engineer and CTO of Userlist.io. On the side, he’s writing a book about building SaaS applications in Ruby on Rails. You can reach out to Benedikt via Twitter.

The Magic of Class-level Instance Variables

Benedikt Deicke — Tue, 02 Oct 2018 13:35:44 +0000

In a previous Ruby Magic, we figured out how to reliably inject modules into classes by overwriting its .new method, allowing us to wrap methods with additional behavior.

This time, we're taking it one step further by extracting that behaviour into a module of its own so we can reuse it. We'll build a Wrappable module that handles the class extension for us, and we'll learn all about class-level instance variables along the way. Let's dive right in!

Introducing the `Wrappable` Module

In order to wrap objects with modules when they are initialized, we have to let the class know what wrapping models to use. Let’s start by creating a simple Wrappable module that provides a wrap method which pushes the given module into an array defined as a class attribute. Additionally, we redefine the new method as discussed in the previous post.

module Wrappable 
  @@wrappers = []

  def wrap(mod)
    @@wrappers << mod 
  end 

  def new(*arguments, &block)
    instance = allocate
    @@wrappers.each { |mod| instance.singleton_class.include(mod) } 
    instance.send(:initialize, *arguments, &block)
    instance
  end 
end

To add the new behavior to a class, we use extend. The extend method adds the given module to the class. The methods then become class methods. To add a module to wrap instances of this class with, we can now call the wrap method.

module Logging 
  def make_noise 
    puts "Started making noise"
    super 
    puts "Finished making noise"
  end
end

class Bird
  extend Wrappable 

  wrap Logging

  def make_noise
    puts "Chirp, chirp!"
  end
end

Let’s give this a try by creating a new instance of Bird and calling the make_noise method.

bird = Bird.new 
bird.make_noise
# Started making noise
# Chirp, chirp!
# Finished making noise

Great! It works as expected. However, things start to behave a bit strange once we extend a second class with the Wrappable module.

module Powered
  def make_noise 
    puts "Powering up"
    super 
    puts "Shutting down" 
  end 
end 

class Machine 
  extend Wrappable 

  wrap Powered

  def make_noise 
    puts "Buzzzzzz" 
  end 
end

machine = Machine.new 
machine.make_noise
# Powering up
# Started making noise
# Buzzzzzz
# Finished making noise
# Shutting down

bird = Bird.new 
bird.make_noise
# Powering up
# Started making noise
# Chirp, chirp!
# Finished making noise
# Shutting down

Even though Machine hasn't been wrapped with the Logging module, it still outputs logging information. What’s worse - even the bird is now powering up and down. That can’t be right, can it?

The root of this problem lies in the way we are storing the modules. The class variable @@wrappables is defined on the Wrappable module and used whenever we add a new module, regardless of the class that wrap is used in.

This get’s more obvious when looking at the class variables defined on the Wrappable module and the Bird and Machine classes. While Wrappable has a class method defined, the two classes don't.

Wrappable.class_variables # => [:@@wrappers]
Bird.class_variables # => []
Machine.class_variables # => []

To fix this, we have to modify the implementation so that it uses instance variables. However, these aren't variables on the instances of Bird or Machine, but instance variables on the classes themselves.

In Ruby, classes are just objects

This is definitely a bit mind boggling at first, but still a very important concept to understand. Classes are instances of Class and writing class Bird; end is equivalent to writing Bird = Class.new. To make things even more confusing Class inherits from Module which inherits from Object. As a result, classes and modules have the same methods as any other object. Most of the methods we use on classes (like the attr_accessor macro) are actually instance methods of Module.

Using Instance Variables on Classes

Let’s change the Wrappable implementation to use instance variables. To keep things a bit cleaner, we introduce a wrappers method that either sets up the array or returns the existing one when the instance variable already exists. We also modify the wrap and new methods so that they utilize that new method.

module Wrappable
  def wrap(mod)
    wrappers << mod
  end

  def wrappers
    @wrappers ||= []
  end

  def new(*arguments, &block)
    instance = allocate
    wrappers.each { |mod| instance.singleton_class.include(mod) }
    instance.send(:initialize, *arguments, &block)
    instance
  end
end

When we check the instance variables on the module and on the two classes, we can see that both Bird and Machine now maintain their own collection of wrapping modules.

Wrappable.instance_variables #=> []
Bird.instance_variables #=> [:@wrappers]
Machine.instance_variables #=> [:@wrappers]

Not surprisingly, this also fixes the problem we observed earlier - now, both classes are wrapped with their own individual modules.

bird = Bird.new 
bird.make_noise
# Started making noise
# Chirp, chirp!
# Finished making noise

machine = Machine.new 
machine.make_noise
# Powering up
# Buzzzzzz
# Shutting down

Supporting Inheritance

This all works great until inheritance is introduced. We would expect that classes would inherit the wrapping modules from the superclass. Let’s check if that's the case.

module Flying
  def make_noise
    super
    puts "Is flying away"
  end
end

class Pigeon < Bird
  wrap Flying

  def make_noise
    puts "Coo!"
  end
end

pigeon = Pigeon.new
pigeon.make_noise
# Coo!
# Is flying away

As you can see, it doesn’t work as expected, because Pigeon is also maintaining its own collection of wrapping modules. While it makes sense that wrapping modules defined for Pigeon aren’t defined on Bird, it’s not exactly what we want. Let’s figure out a way to get all wrappers from the entire inheritance chain.

Lucky for us, Ruby provides the Module#ancestors method to list all the classes and modules a class (or module) inherits from.

Pigeon.ancestors # => [Pigeon, Bird, Object, Kernel, BasicObject]

By adding a grep call, we can pick the ones that are actually extended with Wrappable. As we want to wrap the instances with wrappers from higher up the chain first, we call .reverse to flip the order.

Pigeon.ancestors.grep(Wrappable).reverse # => [Bird, Pigeon]

Ruby’s #=== method

Some of Ruby’s magic comes down to the #=== (or case equality) method. By default, it behaves just like the #== (or equality) method. However, several classes override the #=== method to provide different behavior in case statements. This is how you can use regular expressions (#=== is equivalent to #match?), or classes (#=== is equivalent to #kind_of?) in those statements. Methods like Enumerable#grep, Enumerable#all?, or Enumerable#any? also rely on the case equality method.

Now we can call flat_map(&:wrappers) to get a list of all wrappers defined in the inheritance chain as a single array.

Pigeon.ancestors.grep(Wrappable).reverse.flat_map(&:wrappers) # => [Logging]

All that's left is packing that into an inherited_wrappers module and slightly modifying the new method so that it uses that instead of the wrappers method.

module Wrappable 
  def inherited_wrappers
    ancestors
      .grep(Wrappable)
      .reverse
      .flat_map(&:wrappers)
  end

  def new(*arguments, &block)
    instance = allocate
    inherited_wrappers.each { |mod|instance.singleton_class.include(mod) }
    instance.send(:initialize, *arguments, &block)
    instance
  end
end

A final test run confirms that everything is now working as expected. The wrapping modules are only applied to the class (and its subclasses) they are applied on.

bird = Bird.new
bird.make_noise
# Started making noise
# Chirp, chirp!
# Finished making noise

machine = Machine.new
machine.make_noise
# Powering up
# Buzzzzz
# Shutting down

pigeon = Pigeon.new
pigeon.make_noise
# Started making noise
# Coo!
# Finished making noise
# Is flying away

That's a wrap!

Admittedly, these noisy birds are a bit of a theoretic example (tweet, tweet). But inheritable class instance variables are not just cool to understand how classes work. They are a great example that classes are just objects in Ruby.

And we'll admit that inheritable class instance variables might even be quite useful in real life. For example, think about defining attributes and relationships on a model with the ability to introspect them later. For us the magic is to play around with this and get a better understanding of how things work. And open your mind for a next level of solutions. 🧙🏼‍♀️

As always, we’re looking forward to hearing what you build using this or similar patterns. Any ideas? Please leave a comment!.

Benedikt Deicke is a software engineer and CTO of Userlist.io. On the side, he’s writing a book about building SaaS applications in Ruby on Rails. You can reach out to Benedikt via Twitter.

Changing the Way Ruby Creates Objects

Benedikt Deicke — Tue, 07 Aug 2018 13:12:14 +0000

One of the things that makes Ruby great is that we can customize almost anything to our needs. This is both useful and dangerous. It's easy to shoot ourselves in the foot, but when used carefully, this can result in pretty powerful solutions.

At Ruby Magic, we think useful and dangerous is an excellent combination. Let's look at how Ruby creates and initializes objects and how we can modify the default behavior.

The Basics of Creating New Objects from Classes

To get started, let's see how to create objects in Ruby. To create a new object (or instance), we call new on the class. Unlike other languages, new isn't a keyword of the language itself, but a method that gets called just like any other.

class Dog
end

object = Dog.new

In order to customize the newly created object, it is possible to pass arguments to the new method. Whatever is passed as arguments, will get passed to the initializer.

class Dog
  def initialize(name)
    @name = name
  end
end

object = Dog.new('Good boy')

Again, unlike other languages, the initializer in Ruby is also just a method instead of some special syntax or keyword.

With that in mind, shouldn't it be possible to mess around with those methods, just like it is possible with any other Ruby method? Of course it is!

Modifying the Behavior of a Single Object

Let's say we want to ensure that all objects of a particular class will always print log statements, even if the method is overridden in subclasses. One way to do this is to add a module to the object's singleton class.

module Logging
  def make_noise
    puts "Started making noise"
    super
    puts "Finished making noise"
  end
end

class Bird
  def make_noise
    puts "Chirp, chirp!"
  end
end

object = Bird.new
object.singleton_class.include(Logging)
object.make_noise
# Started making noise
# Chirp, chirp!
# Finished making noise

In this example, a Bird object is created using Bird.new, and the Logging module is included in the resulting object using its singleton class.

What's a Singleton Class?
Ruby allows methods that are unique to a single object. To support this, Ruby adds an anonymous class between the object and its actual class. When methods are called, the ones defined on the singleton class get precedence over the methods in the actual class. These singleton classes are unique to every object, so adding methods to them doesn't affect any other objects of the actual class. Learn more about classes and objects in the Programming Ruby guide.

It's a bit cumbersome to modify the singleton class of each object whenever it is created. So let's move the inclusion of the Logging class to the initializer to add it for every created object.

module Logging
  def make_noise
    puts "Started making noise"
    super
    puts "Finished making noise"
  end
end

class Bird
  def initialize
    singleton_class.include(Logging)
  end

  def make_noise
    puts "Chirp, chirp!"
  end
end

object = Bird.new
object.make_noise
# Started making noise
# Chirp, chirp!
# Finished making noise

While this works well, if we create a subclass of Bird, like Duck, its initializer needs to call super to retain the Logging behavior. While one can argue that it's always a good idea to properly call super whenever a method is overridden, let's try to find a way that doesn't require it.

If we don't call super from the subclass, we lose the inclusion of the Logger class:

class Duck < Bird
  def initialize(name)
    @name = name
  end

  def make_noise
    puts "#{@name}: Quack, quack!"
  end
end

object = Duck.new('Felix')
object.make_noise
# Felix: Quack, quack!

Instead, Let's override Bird.new. As mentioned before, new is just a method implemented on classes. So we can override it, call super, and modify the newly created object to our needs.

class Bird
  def self.new(*arguments, &block)
    instance = super
    instance.singleton_class.include(Logging)
    instance
  end
end

object = Duck.new('Felix')
object.make_noise
# Started making noise
# Felix: Quack, quack!
# Finished making noise

But, what happens when we call make_noise in the initializer? Unfortunately, because the singleton class doesn't include the Logging module yet, we won't get the desired output.

Luckily, there's a solution: It's possible to create the default .new behavior from scratch by calling allocate.

class Bird
  def self.new(*arguments, &block)
    instance = allocate
    instance.singleton_class.include(Logging)
    instance.send(:initialize, *arguments, &block)
    instance
  end
end

Calling allocate returns a new, uninitialized object of the class. So afterward, we can include the additional behavior and only then, call the initialize method on that object. (Because initialize is private by default, we have to resort to using send for this).

The Truth About Class#allocate
Unlike other methods, it's not possible to override allocate. Ruby doesn't use the conventional way of dispatching methods for allocate internally. As a result, just overriding allocate without also overriding new doesn't work. However, if we're calling allocate directly, Ruby will call the redefined method. Learn more about Class#new and Class#allocate in Ruby's documentation.

Why Would We Do This?

As with a lot of things, modifying the way Ruby creates objects from classes can be dangerous and things might break in unexpected ways.

Nonetheless, there are valid use cases for changing the object creation. For instance, ActiveRecord uses allocate with a different init_from_db method to change the initialization process when creating objects from the database as opposed to building unsaved objects. It also uses allocate to convert records between different single-table inheritance types with becomes.

Most important, by playing around with object creation, you get a deeper insight into how it works in Ruby and open your mind to different solutions. We hope you enjoyed the article.

We'd love to hear about the things you implemented by changing Ruby's default way of creating objects. Please don't hesitate to leave a comment.

Benedikt Deicke is a software engineer and CTO of Userlist.io. On the side, he's writing a book about building SaaS applications in Ruby on Rails. You can reach out to Benedikt via Twitter.