The magic of metaprogramming : Examining the has_many association using Active Record's code base - Part 1

It's Magic

When working with Rails, its capacity for metaprogramming and adherence to convention over configuration can make programming basically feel like this:

This is okay! In fact, it's wonderful to see just how little boilerplate coding is necessary for each new model and its associations. For example, let's look at the following code:

class Teacher < ApplicationRecord
has_many :books
end

class Book < ApplicationRecord
belongs_to :teacher
end

These few lines of code allow you to do many things based on the methods that are extended to the Teacher and Book class by using ActiveRecord's Base module. Perhaps most powerfully though, is how it lets one take any instance of the Teacher class and access all instances of the Book class associated with that teacher instance by invoking the .books accessor method. But how?

No - it's Code

So - a lot is clearly being generated under the hood in the above example. To figure out how passing has_many :books could make the associations and build out the methods for our Teacher class, I started by looking at the documentation before diving into the GitHub repo for Active Record to look at the source code to see what I could uncover. This turned out to be a somewhat ambitious project so I ultimately decided to split this up into two separate blog posts. In order to understand this, we'll need to spend some time uncovering what reflections are in ActiveRecord in this blog post before moving on to what the define_accessors method is doing, and what mixins are in Ruby in the next. So let's get this duology off the ground!

While my initial forays have not enabled me to grasp every part of the code base - and there will be gaps in my explanation - I hope it will at least provide some understanding and provoke the desire for additional exploration for you!

The starting point was to look at the ActiveRecord docs for additional information on the has_many assocation.

The use of this macro, :has many, will call a method that takes the following parameters:

(name, scope = nil, **options, &extension) 

So we'll get the name of association, the scope which will default to nil if nothing is passed, an options parameter, and the possibility for passing in a block as an extension. But how are these parameters passed in? It's not very clear from the above example how the method receives these options.

Fortunately Rails is well documented and the way to accomplish these tasks are just below the explanation of possible methods that can be utilized. For the scope: "You can pass a second argument scope as a callable (i.e. proc or lambda) to retrieve a specific set of records or customize the generated query when you access the associated collection." So an example for us might be to pass something like the following to restrict the books to only books published within the last ten years.

has_many :books, -> { where(book_age > 10) }

Options has a lot of possible ways that you can specify how you're getting your data or how your data is coming back to you. Perhaps the most common example would be the :through option, which allows you to use an intermediary class that operates as a join table between two other classes so that you can access pertinent information -through- a connected class. Other possible options include :source if you need to specify a different class name to prevent overlapping accessor methods and :dependent if you wanted to set a way to destroy multiple associations when a #destroy method is called. The syntax for creating an option mirrors how to pass in a scope parameter.

Lastly, there are extensions which are passed as blocks and define methods within body of the block. These custom methods allow you to create unique ways of interacting with your model that could enable you to generate additional accessor, creator, or associate the instances in other ways with one another.

While this is great information - and we could certainly stop here if we just want to know how to do something - it doesn't really answer the question of how these things are actually being created under the hood. So - off to the Rails Github repo!

We're Off

After stumbling around the labyrinth of ActiveRecord files, my searching lead me to this ruby file which handles all macro definitions that are created by our association method call. Every has_many, belongs_to, and belongs_to_one you have written have effectively been passed to this block of twelve lines of code twelve lines of code:

def self.build(model, name, scope, options, &block)
      if model.dangerous_attribute_method?(name)
        raise ArgumentError, "You tried to define an association named #{name} on the model #{model.name}, but " \
                             "this will conflict with a method #{name} already defined by Active Record. " \
                             "Please choose a different association name."
      end

      reflection = create_reflection(model, name, scope, options, &block)
      define_accessors model, reflection
      define_callbacks model, reflection
      define_validations model, reflection
      reflection
    end

While twelve lines may not seem like a lot, it will take a lot more time and code parsing to try to understand what is happening here.

The first half of the #build method is fairly intuitive - it is a class level method which is called and is passed five arguments - the model, the name, the scope, the options and an optional block statement. So while we haven't seen the fact that the model is called in the documentation - when we go under the hood - we can see how the particular model is brought into the method that will create the association. The model will refer to our class model - so in our example above - it would be the Teacher. The name parameter refers to the name of the association. In our case, the :books symbol is being passed in here. The scope and options will line up with our corresponding understanding from starting in the ruby docs and and would be defined as nil in our above Teacher class. As a side note, as we work through the code you'll see examples of the options so let's put a pin it that for a moment for when we come back it. Finally, there is an optional block that can be passed in via the &block parameter - which we know would refer to any custom extensions we might want to add.

Next, there is some error handling happening. This particular error seems to be handling for this exception listed in the Active Record Docs:

"Don’t create associations that have the same name as instance methods ... of ActiveRecord::Base. Since the association adds a method with that name to its model, using an association with the same name as one provided by ActiveRecord::Base will override the method inherited through ActiveRecord::Base and will break things. For instance, 'attributes' and 'connection' would be bad choices for association names, because those names already exist in the list of ActiveRecord::Base instance methods." In short, if there is a conflict with the association name that you passed in and it falls under a declared method name from within ActiveRecord, it will politely tell you, "Hey friend - that's already in use. Have you thought about trying something else?"

Our :books should have no trouble clearing this error handling so we can move forward!

These two lines hide multitudes

   reflection = create_reflection(model, name, scope, options, &block)
      define_accessors model, reflection

The following becomes a bit more dense so this post will focus on tracking down just these the first line to the best that our rookie Ruby skills can manage. Part two will focus on the second line.

Line One - Reflections and Options and Extensions, Oh My!

For the first line, we don't have to go travel far as the method create_reflection is defined further down the page.

    def self.create_reflection(model, name, scope, options, &block)
      raise ArgumentError, "association names must be a Symbol" unless name.kind_of?(Symbol)

      validate_options(options)

      extension = define_extensions(model, name, &block)
      options[:extend] = [*options[:extend], extension] if extension

      scope = build_scope(scope)

      ActiveRecord::Reflection.create(macro, name, scope, options, model)
    end

Here are passing in the same arguments from our build method, and having an additional step for error handling. The association cleared the fact that it wasn't dangerous - but was it written as a symbol? In our case - yes - :books is a symbol. Validate options is declared further down the file and it handles error handling for any passed in options. We haven't passed any that we need to concern ourselves with - so I'll step over that to focus on the extension creation. Even though we haven't passed any extensions, it's worth taking a look at this method to see a little bit behind the curtain.

The #define_extension method points to this location in collection_assocation.rb file

   def self.define_extensions(model, name, &block)
      if block_given?
        extension_module_name = "#{name.to_s.camelize}AssociationExtension"
        extension = Module.new(&block)
        model.const_set(extension_module_name, extension)
      end
    end

This checks to see if a block was given. If so, it's here that it generates an association extension that has a CamelCase name. We can note where some of Rail's convention logic comes into play. If there is not a class called Book that our Teacher class can associate to, this association will not connect to anything. This is why a class called book won't work because it doesn't adhere to the camel cased creation happening throughout the codebase. This helps create our extension which is generated with a new Module class using our passed in block. If we were adding anything, our model would receive the extension model via the #const_set. Instead, as we won't meet the conditional requirement, we will simply jump down to the bottom of the method and hit our end to move move back and look at the options.

  options[:extend] = [*options[:extend], extension] if extension

      scope = build_scope(scope)

Our options[:extend] key is updated to include all of the components previously inside of our :extend and our newly minted extension (if we had one) would be added and now made to be part of an array. This then moves us towards building our
scope variable which is created by passing our scope to the build_scope method.

   def self.build_scope(scope)
      if scope && scope.arity == 0
        proc { instance_exec(&scope) }
      else
        scope
      end
    end

Within this method, our scope is then checked as part of a conditional. If it exists AND when #arity method is invoked on the scope variable (#arity is part of Ruby and returns, per the docs, "an indication of the number of arguments accepted by a method") it return that we have 0 arguments being accepted, then it uses a proc method to create a new variable which which, in turn, creates a hash that stores what the scope will be and returns it. While I do not entirely understand this, I believe what it does is if our scope variable has no arguments when it is passed in (like ours would in this case), this line creates the properties necessary to prevent any errors as scope is passed around later during the creation process. My reasoning is because if the scope is passed in and it clears this hurdle - that is - it possesses more than 0 arguments, it will instead simply be returned fully intact. So if the scope exists - leave it be. If it doesn't - let's 'create' it here - hence the build_scope method name.

So now that scope has been passed back we're left with:

 ActiveRecord::Reflection.create(macro, name, scope, options, model)
    end

So now we're almost to the end of this method - but before we get there .. we have to make a little trip to find our Reflection module and see what is being created here. It looks like this little method is the culprit:

 def create(macro, name, scope, options, ar)
        reflection = reflection_class_for(macro).new(name, scope, options, ar)
        options[:through] ? ThroughReflection.new(reflection) : reflection
      end

So it creates a reflection, which, according to the docs is what "enables the ability to examine the associations and aggregations of Active Record classes and objects. This information, for example, can be used in a form builder that takes an Active Record object and creates input fields for all of the attributes depending on their type and displays the associations to other objects." It looks like we've stumbled right into what will help with the heavy lifting of setting associations.

Here we can also see another place where an option is possible. Since we did not pass in the option of :through and define a way that teachers might connect to another model through the :books accessor, when we finish with the reflection_class_for, we will simply return the reflection rather than creating a new ThroughReflection instance when we eventually hit the ternary statement. But - we're not there yet so let's not get too ahead of ourselves! Instead we now have a new definition to track down - the reflection_class_for method. Well let's scroll down through this file and see ..

        def reflection_class_for(macro)
          case macro
          when :composed_of
            AggregateReflection
          when :has_many
            HasManyReflection
          when :has_one
            HasOneReflection
          when :belongs_to
            BelongsToReflection
          else
            raise "Unsupported Macro: #{macro}"
          end
        end
    end

Well look at this - it's our good buddy switch case which is checking the symbol associated with our macro and passing a class back. Our little :has_many that started its journey oh so long ago is finally looking back at us! We won't hit that error at the end because we typed it correctly so let's scroll down the file to find the HasManyReflection below ...

  class HasManyReflection < AssociationReflection # :nodoc:
      def macro; :has_many; end

      def collection?; true; end

      def association_class
        if options[:through]
          Associations::HasManyThroughAssociation
        else
          Associations::HasManyAssociation
        end
      end
    end

Okay - seems fairly straight forward. The HasManyReflection inherits from a class called AssocationReflection, and has a few short definitions built in - including if you've set the option for there to be a has_many, through: relationship - it will return the HasManyThroughAssocations module as opposed to the HasManyAssocation module. Otherwise it will define the method of #macro as has_many and create a method #collection? which will be used later to let the builder know if this macro will require a collection (most likely a reference to an array, as we will see later) during creation. Neat.

Okay, so - we can jump back. We've got our macro back - which is a class instance of HasManyAssocations and .new is called on it and we pass in our arguments to get back our particular HasManyAssociation instance. To see more about the HasManyAssociation class itself, you can examine the class here. It will handle a great deal of the functionality that our custom class will ultimately need to make use of when we are working with it in a Rails app. Specifically there are methods that handle when #destroy is called on an object, how the teacher instance will be able access the array that contains its books, how to see the length of the specified association array, and how these pieces of information are updated 'behind the scenes' when books are added or removed from having an association with the teacher.

HasManyAssocations is also an extension of the of the CollectionAssocation class. This class will actually do the work of creating the array that will hold our associations mentioned above. Remember def collection?; true; end awhile ago? If this was false, there would be no need to create a collection. It also creates the setter and getter methods for collection IDs associated - in our case - to specific book foreign ID keys. We'll look at those methods in part two as they will more cleanly line up with the generation of our accessor methods.

After that, our :books macro has been created - but it isn't wired up to all of our specifications quite yet as most of the methods that handle the behavior mentioned above are private methods that, as a user, we're not able to directly access. So, the macro will be passed back as 'reflection' to our starting method and we can move on to the next line of code in the Association class. Do you remember that location?

We're now back in our original method call and we've actually come to the next line. Hoo - it's amazing what one method can do when you have a lot of pure functions, huh?

Take a moment and when you're ready, you can follow me as I endeavor to track down our next line and finish out the method in part two!

As an aside, if there are more experienced programmers who can see errors in my explanation and who have somehow made it this far without clawing your eyes out and banging your head against a wall - please chime in and explain any and all mistakes in the comments!