Extracting Rails Engine by example - Vikings social media

This post will describe step by step extraction process of a Rails engine from a monolithic application. This is a follow-up to my previous piece, where I explained the pros and cons of Rails engines as a solution for modularizing big monolithic applications. The full source code of application used for this post can be found here

Table Of Contents

• Introduction
• Engine preparation
  • Engine initilization
  • RSpec initialization
  • Databse connection configuration
• Parent app preparation
  • Request test creation
  • Engine instalation
• Code transfer
  • Migrations
  • Models
  • Factories
  • Rest of Rails parts
  • Not standard Rails components
  • External dependencies

Introduction

To set a scene, let's imagine that we are earls (aka chieftains) of Viking communities. As a forward-minded leader, we've already created a simple social media website for our people. It allows townsmen to enlist as warriors or farmers, and browse lists of each profession, to find companions or suppliers for upcoming raids. It also has a separate section where special e-seer gives its daily weather prophecies. It all works together, but now we've decided that it is time to improve architecture, and maybe share some parts of our systems with other Viking communities as an open source. With no further ado let's get to work, and extract seer module into the engine.

Engine preparation

Before We can move any of the existing code from the parent app into the engine, we need to initialize it, set up a unit test framework and configure development and test database connections.

Initialize engine

To initialize engine We can use Rails generator with this command.

 $ rails plugin new seer --mountable --dummy-path=spec/dummy --skip-test --databse=postgresql --skip-git  

It accepts a number of flags that adjust the generated engine scaffold to our needs. Most important ones are:

• --mountable, it instructs the generator that we want a 'mountable' engine, and it should be namespace isolated from the host application. This prevents constants name collision, between the code of the host app and the engine.
• --dummy-path, sets the location of 'dummy' app generated by Rails for unit test purposes.
• --skip-test, instructs generator to not scaffold tests, it is useful if you prefer to use another test framework over Minitest. For me preferred one is Rspec (which will be configured later on).
• --database, configures database adapter which will be used. It is the best to set it, to the same as the one used by the application from which we want to extract an engine
• --skip-git, prevents new git repository initialization. I prefer to do that at the very end of a whole process when I'm moving the engine from a subdirectory of the parent app into a separate repository. Before that, I work on a separate branch of the main repository and compare changes in the merge (aka pull) request.

After the command is executed, the generator scaffolds the requested engine in a subdirectory of the parent app. As the generator is working we can see typical output upon the terminal. Among standard parts like controllers, models, we can also notice a few additional files has been generated

create  lib/seer.rb
create  lib/seer/engine.rb
create  seer.gemspec
create  lib/seer/version.rb
vendor_app  spec/dummy

Those files are really interesting for two reasons, one they clearly indicate that engine is a specialized gem, and two each of them has a special role:

• lib/seer.rb is an entry point to the engine, this file will be required when the host application of the engine loads its dependencies. It can be used to make engines external dependencies configurable and loads lib/seer/engine.rb
• lib/seer/engine.rb informs the parent application that this given gem is an engine. It also can be used to configure engine parameters (similar config/application.rb file is used in a standard Rails app). In fact, the generator already configured one parameter for us: isolate_namespace Seer, it declares that the whole engine is isolated in the namespace named after the engine.
• seer.gemspec contains metadata about engine and gems upon which it depends.
• lib/seer/version.rb this file declares the current version of the engine
• spec/dummy is previously mentioned 'dummy' Rails app which will be used in unit tests of the engine

Rspec initialization

After the engine is initialized, we should configure a unit test framework. Although there is no limitation on which one we can use, it is the most convenient to choose the same one, as it is used in the parent app. That way, we can simply reuse most of the already existing unit test code. At first, we need to add development dependencies into the gemspec file.

Gem::Specification.new do |spec|  
  ....
  spec.test_files = Dir["spec/**/*"]            

  spec.add_development_dependency 'rspec-rails'
  spec.add_development_dependency 'factory_bot_rails'
  spec.add_development_dependency 'dotenv-rails'
end

Then we can proceed lib/seer/engine.rb and set up generators.

module Seer
  class Engine < ::Rails::Engine
    isolate_namespace Seer


    config.generators do |generators|
      generators.test_framework :rspec
      generators.fixture_replacement :factory_bot
      generators.factory_bot dir: 'spec/factories'
    end
  end
end

And at least we should install new dependencies and RSpec

developer@macbook engines-directory $ bundle install
developer@macbook engines-directory $ rails g rspec:install
  create  .rspec
  exist  spec
  create  spec/spec_helper.rb
  create  spec/rails_helper.rb

After rails_helper.rb file gets created, we have to adjust the environment path in it, to point onto the 'dummy' app.

require File.expand_path('../dummy/config/environment.rb', __FILE__)

And now we are ready for the next step.

Database connection configuration

A first step to configure 'dummy' app database connection is to adjust database.yml file in 'dummy' db directory, for that most of the database.yml file from the parent app can be reused, except for databases names.

default: &default
adapter: postgresql
host: <%= ENV.fetch('POSTGRES_HOST') %>
username: <%= ENV.fetch('POSTGRES_USER') %>
password: <%= ENV.fetch('POSTGRES_PASSWORD') %>
port: <%= ENV.fetch('POSTGRES_PORT') %>

development:
  <<: *default
  database: <%= ENV.fetch('POSTGRES_DB') %>_development

test:
  <<: *default
  database: <%= ENV.fetch('POSTGRES_DB') %>_test

I've decided to store sensitive data outside the git repository, inside environment variables. In this case, we should prepare a tool to load locally environment variables for us. I've chosen Dotenv, so I've created a .env file in 'dummy' app directory and required dotenv before dummy apps boots.

#seer/spec/dummy/config/application.rb
Bundler.require(*Rails.groups)
require "seer"

require 'dotenv-rails'

module Dummy
  class Application < Rails::Application
...

At least we should execute two Rails database-related tasks.

developer@macbook engines-directory $ rails db:setup
developer@macbook engines-directory $ rails db:migrate

If all steps are correctly performed inside the 'dummy' app db catalog, schema file should get created.

After that, we are ready to take the next step.

Parent app preparation

With the initialized engine next, we need to plug it into the parent app, so the changes we are going to make Can be iteratively verified as we proceeded with code transfer.

Request test creation

Before we extract any piece of code, it will be very handy to ensure, that the whole system from an end-user perspective has not changed after the process is completed. To assure that, I prefer to add requests test for the extracted module.

require 'rails_helper'

RSpec.describe 'WeatherProphecies', type: :request do
  let(:villager) { create(:villager) }

  before { sign_in villager }

  describe 'GET /weather_prophecies' do
    it 'returns current prophecy' do
      get weather_prophecies_path

      expect(response.body).to include('rain')
    end
  end
end

Engine installation

To use newly created engine we need to include it in Gemfile of thr parent app

source 'https://rubygems.org'
...

# engines
gem 'seer', path: 'seer'

And mount it in the routes.

Rails.application.routes.draw do
  ...
  mount Seer::Engine, at: '/seer'  
  ...
end

Code transfer

With the parent app and the engine ready, we can start moving pieces of code from one to another. I prefer to start from the database and continue up the layers until I got to the controllers. That way we have less external dependencies to handle at the time.

Migrations

In the end, the engine and the parent app will use the same database. However for test and development purposes, and also to make the engine truly mountable, even to other apps, we have to copy all migrations related to transferred database tables into the engine. While coping migrations from the parent, we should leave migrations names intact, so when the engine is installed those migrations will not get duplicated.

By default all tables of the engine are prefixed with the name of the engine (seer in this case) since that could not be the case before the extraction when all existing migrations are copied to the engine, we must add new ones which will rename transferred tables with the correct prefix.

class RenameWeatherPropheciesToSeerWeatherProphecies < ActiveRecord::Migration[6.0]
  def change
    rename_table :weather_prophecies, :seer_weather_prophecies
  end
end

To copy and apply the migrations of the engine to the parent app, we can use the following commands.

developer@macbook host-app-directory $ rails seer:install:migrations
NOTE: Migration 20170809130629_create_weather_prophecies.rb from seer has been skipped. Migration with the same name already exists.
developer@macbook host-app-directory $ rails db:migrate

Models

To transfer models from the parent into the engine, we can simply run the bash command, or use the cut and paste method.

developer@macbook host-app-directory $ mv app/models/weather_prophecy.rb seer/app/models/seer/weather_prophecy.rb

We should be careful to place each file in the correct directory, which represents the namespace of the engine (eg: seer/app/models/seer) otherwise, an autoloader will ignore them and we will get an error NameError: uninitialized constant.

After each model with the corresponding spec is transferred, we need to wrap their code in the module of the engine.

module Seer
  class WeatherProphecy < ApplicationRecord
  end
end

Factories

When models and their specs are moved, the one thing we lack to make test back to the green state are factories.

To use FactoryBot at first we need to adjust rails_helper.rb file and point set a path to a directory which will contain factories definitions

require 'factory_bot'
FactoryBot.definition_file_paths = [File.join(File.dirname(__FILE__), 'factories')]
FactoryBot.find_definitions

RSpec.configure do |config|
  config.include FactoryBot::Syntax::Methods
  ...
end

Then we can move factory definitions just like we've done with the model files.

Rest of Rails parts

The rest of the Rails parts like controllers, jobs, channels can be moved the same way as we've transferred models.

As for the routes, they should be cut out from the routes.rb file of the parent app and placed in the one that belongs to the engine.

Not standard Rails components

In this scenario I've created an additional services layer, there is one gotcha with such an approach. To ensure that autoloader loads these parts of the code as well we can do one of two things, either we extend autoloader paths, or we place these additional layers in the app directory of the engine. I've chosen the latter for this scenario.

Besides that, we can move them like the rest.

External dependencies

The last finishing touch is to make external dependencies configurable.
Although while developing the engine, we could refer to parents app modules and classes, by using their qualified names (prefixed with ::).

module Seer
  class ApplicationController < ::ApplicationController
  end
end

Unfortunately, this introduces thigh coupling between the engine and the parent app. To address that, and make our engine more flexible we should allow others to configure external dependencies upon engine initialization.

Because of that, we should add a module accessor which would store such dependencies.

# lib/seer.rb
module Seer
  mattr_accessor :application_controller_class

  def self.application_controller_class
    @@application_controller_class.constantize
  end
end

To configure them, we need to create an initializer in the parent app.

# parent_app/config/initializers/seer.rb

Seer.application_controller_class = 'ApplicationController'

Inside the engine, we can use them in the following manner.

  class ApplicationController < Seer.application_controller_class

After this last step, our engine is ready to be used. We could leave it as a subdirectory of parents app, or move it to stand alone repository and use it freely in a number of apps.

As we've just accomplished this challenging task our system is better structuralized and easier to navigate, a test suite is running faster, it is easier to work in parallel on different parts of the system without worry about merge conflicts. We are now ready to celebrate our glory on a Viking feast.

Comments

[Amit Patel]

Nicely written. Thanks Mikolaj!

What if there is an exact opposite scenario where you have a full-fledged rails application and want to use it as an engine to another app?

Like I have github.com/chatwoot/chatwoot and I want to build a marketing website on top of it without touching(or modify with minimal changes) the source code of that app(chatwoot)

So the webapp essentially will have Marketing stuff + Chatwoot

[Mikus]

Thanks, used as good source for better understanding of Rails engines and as good starting point.

[David Sanchez]

Awesome! A great example of how we can moduliraze our application, thanks for sharing! 🤟