DEV Community: Mikołaj Wawrzyniak

Extracting Rails Engine by example - Vikings social media

Mikołaj Wawrzyniak — Mon, 01 Jul 2019 17:19:26 +0000

This post will describe step by step extraction process of a Rails engine from a monolithic application. This is a follow-up to my previous piece, where I explained the pros and cons of Rails engines as a solution for modularizing big monolithic applications. The full source code of application used for this post can be found here

Introduction
Engine preparation
- Engine initilization
- RSpec initialization
- Databse connection configuration
Parent app preparation
- Request test creation
- Engine instalation
Code transfer
- Migrations
- Models
- Factories
- Rest of Rails parts
- Not standard Rails components
- External dependencies

Introduction

To set a scene, let's imagine that we are earls (aka chieftains) of Viking communities. As a forward-minded leader, we've already created a simple social media website for our people. It allows townsmen to enlist as warriors or farmers, and browse lists of each profession, to find companions or suppliers for upcoming raids. It also has a separate section where special e-seer gives its daily weather prophecies. It all works together, but now we've decided that it is time to improve architecture, and maybe share some parts of our systems with other Viking communities as an open source. With no further ado let's get to work, and extract seer module into the engine.

Engine preparation

Before We can move any of the existing code from the parent app into the engine, we need to initialize it, set up a unit test framework and configure development and test database connections.

Initialize engine

To initialize engine We can use Rails generator with this command.



 $ rails plugin new seer --mountable --dummy-path=spec/dummy --skip-test --databse=postgresql --skip-git

It accepts a number of flags that adjust the generated engine scaffold to our needs. Most important ones are:

--mountable, it instructs the generator that we want a 'mountable' engine, and it should be namespace isolated from the host application. This prevents constants name collision, between the code of the host app and the engine.
--dummy-path, sets the location of 'dummy' app generated by Rails for unit test purposes.
--skip-test, instructs generator to not scaffold tests, it is useful if you prefer to use another test framework over Minitest. For me preferred one is Rspec (which will be configured later on).
--database, configures database adapter which will be used. It is the best to set it, to the same as the one used by the application from which we want to extract an engine
--skip-git, prevents new git repository initialization. I prefer to do that at the very end of a whole process when I'm moving the engine from a subdirectory of the parent app into a separate repository. Before that, I work on a separate branch of the main repository and compare changes in the merge (aka pull) request.

After the command is executed, the generator scaffolds the requested engine in a subdirectory of the parent app. As the generator is working we can see typical output upon the terminal. Among standard parts like controllers, models, we can also notice a few additional files has been generated



create  lib/seer.rb
create  lib/seer/engine.rb
create  seer.gemspec
create  lib/seer/version.rb
vendor_app  spec/dummy

Those files are really interesting for two reasons, one they clearly indicate that engine is a specialized gem, and two each of them has a special role:

lib/seer.rb is an entry point to the engine, this file will be required when the host application of the engine loads its dependencies. It can be used to make engines external dependencies configurable and loads lib/seer/engine.rb
lib/seer/engine.rb informs the parent application that this given gem is an engine. It also can be used to configure engine parameters (similar config/application.rb file is used in a standard Rails app). In fact, the generator already configured one parameter for us: isolate_namespace Seer, it declares that the whole engine is isolated in the namespace named after the engine.
seer.gemspec contains metadata about engine and gems upon which it depends.
lib/seer/version.rb this file declares the current version of the engine
spec/dummy is previously mentioned 'dummy' Rails app which will be used in unit tests of the engine

Rspec initialization

After the engine is initialized, we should configure a unit test framework. Although there is no limitation on which one we can use, it is the most convenient to choose the same one, as it is used in the parent app. That way, we can simply reuse most of the already existing unit test code. At first, we need to add development dependencies into the gemspec file.



Gem::Specification.new do |spec|  
  ....
  spec.test_files = Dir["spec/**/*"]            

  spec.add_development_dependency 'rspec-rails'
  spec.add_development_dependency 'factory_bot_rails'
  spec.add_development_dependency 'dotenv-rails'
end

Then we can proceed lib/seer/engine.rb and set up generators.



module Seer
  class Engine < ::Rails::Engine
    isolate_namespace Seer


    config.generators do |generators|
      generators.test_framework :rspec
      generators.fixture_replacement :factory_bot
      generators.factory_bot dir: 'spec/factories'
    end
  end
end

And at least we should install new dependencies and RSpec



developer@macbook engines-directory $ bundle install
developer@macbook engines-directory $ rails g rspec:install
  create  .rspec
  exist  spec
  create  spec/spec_helper.rb
  create  spec/rails_helper.rb

After rails_helper.rb file gets created, we have to adjust the environment path in it, to point onto the 'dummy' app.



require File.expand_path('../dummy/config/environment.rb', __FILE__)

And now we are ready for the next step.

Database connection configuration

A first step to configure 'dummy' app database connection is to adjust database.yml file in 'dummy' db directory, for that most of the database.yml file from the parent app can be reused, except for databases names.



default: &default
adapter: postgresql
host: <%= ENV.fetch('POSTGRES_HOST') %>
username: <%= ENV.fetch('POSTGRES_USER') %>
password: <%= ENV.fetch('POSTGRES_PASSWORD') %>
port: <%= ENV.fetch('POSTGRES_PORT') %>

development:
  <<: *default
  database: <%= ENV.fetch('POSTGRES_DB') %>_development

test:
  <<: *default
  database: <%= ENV.fetch('POSTGRES_DB') %>_test

I've decided to store sensitive data outside the git repository, inside environment variables. In this case, we should prepare a tool to load locally environment variables for us. I've chosen Dotenv, so I've created a .env file in 'dummy' app directory and required dotenv before dummy apps boots.



#seer/spec/dummy/config/application.rb
Bundler.require(*Rails.groups)
require "seer"

require 'dotenv-rails'

module Dummy
  class Application < Rails::Application
...

At least we should execute two Rails database-related tasks.



developer@macbook engines-directory $ rails db:setup
developer@macbook engines-directory $ rails db:migrate

If all steps are correctly performed inside the 'dummy' app db catalog, schema file should get created.

After that, we are ready to take the next step.

Parent app preparation

With the initialized engine next, we need to plug it into the parent app, so the changes we are going to make Can be iteratively verified as we proceeded with code transfer.

Request test creation

Before we extract any piece of code, it will be very handy to ensure, that the whole system from an end-user perspective has not changed after the process is completed. To assure that, I prefer to add requests test for the extracted module.



require 'rails_helper'

RSpec.describe 'WeatherProphecies', type: :request do
  let(:villager) { create(:villager) }

  before { sign_in villager }

  describe 'GET /weather_prophecies' do
    it 'returns current prophecy' do
      get weather_prophecies_path

      expect(response.body).to include('rain')
    end
  end
end

Engine installation

To use newly created engine we need to include it in Gemfile of thr parent app



source 'https://rubygems.org'
...

# engines
gem 'seer', path: 'seer'

And mount it in the routes.



Rails.application.routes.draw do
  ...
  mount Seer::Engine, at: '/seer'  
  ...
end

Code transfer

With the parent app and the engine ready, we can start moving pieces of code from one to another. I prefer to start from the database and continue up the layers until I got to the controllers. That way we have less external dependencies to handle at the time.

Migrations

In the end, the engine and the parent app will use the same database. However for test and development purposes, and also to make the engine truly mountable, even to other apps, we have to copy all migrations related to transferred database tables into the engine. While coping migrations from the parent, we should leave migrations names intact, so when the engine is installed those migrations will not get duplicated.

By default all tables of the engine are prefixed with the name of the engine (seer in this case) since that could not be the case before the extraction when all existing migrations are copied to the engine, we must add new ones which will rename transferred tables with the correct prefix.



class RenameWeatherPropheciesToSeerWeatherProphecies < ActiveRecord::Migration[6.0]
  def change
    rename_table :weather_prophecies, :seer_weather_prophecies
  end
end

To copy and apply the migrations of the engine to the parent app, we can use the following commands.



developer@macbook host-app-directory $ rails seer:install:migrations
NOTE: Migration 20170809130629_create_weather_prophecies.rb from seer has been skipped. Migration with the same name already exists.
developer@macbook host-app-directory $ rails db:migrate

Models

To transfer models from the parent into the engine, we can simply run the bash command, or use the cut and paste method.



developer@macbook host-app-directory $ mv app/models/weather_prophecy.rb seer/app/models/seer/weather_prophecy.rb

We should be careful to place each file in the correct directory, which represents the namespace of the engine (eg: seer/app/models/seer) otherwise, an autoloader will ignore them and we will get an error NameError: uninitialized constant.

After each model with the corresponding spec is transferred, we need to wrap their code in the module of the engine.



module Seer
  class WeatherProphecy < ApplicationRecord
  end
end

Factories

When models and their specs are moved, the one thing we lack to make test back to the green state are factories.

To use FactoryBot at first we need to adjust rails_helper.rb file and point set a path to a directory which will contain factories definitions



require 'factory_bot'
FactoryBot.definition_file_paths = [File.join(File.dirname(__FILE__), 'factories')]
FactoryBot.find_definitions

RSpec.configure do |config|
  config.include FactoryBot::Syntax::Methods
  ...
end

Then we can move factory definitions just like we've done with the model files.

Rest of Rails parts

The rest of the Rails parts like controllers, jobs, channels can be moved the same way as we've transferred models.

As for the routes, they should be cut out from the routes.rb file of the parent app and placed in the one that belongs to the engine.

Not standard Rails components

In this scenario I've created an additional services layer, there is one gotcha with such an approach. To ensure that autoloader loads these parts of the code as well we can do one of two things, either we extend autoloader paths, or we place these additional layers in the app directory of the engine. I've chosen the latter for this scenario.

Besides that, we can move them like the rest.

External dependencies

The last finishing touch is to make external dependencies configurable.
Although while developing the engine, we could refer to parents app modules and classes, by using their qualified names (prefixed with ::).



module Seer
  class ApplicationController < ::ApplicationController
  end
end

Unfortunately, this introduces thigh coupling between the engine and the parent app. To address that, and make our engine more flexible we should allow others to configure external dependencies upon engine initialization.

Because of that, we should add a module accessor which would store such dependencies.



# lib/seer.rb
module Seer
  mattr_accessor :application_controller_class

  def self.application_controller_class
    @@application_controller_class.constantize
  end
end

To configure them, we need to create an initializer in the parent app.



# parent_app/config/initializers/seer.rb

Seer.application_controller_class = 'ApplicationController'

Inside the engine, we can use them in the following manner.



  class ApplicationController < Seer.application_controller_class

After this last step, our engine is ready to be used. We could leave it as a subdirectory of parents app, or move it to stand alone repository and use it freely in a number of apps.

As we've just accomplished this challenging task our system is better structuralized and easier to navigate, a test suite is running faster, it is easier to work in parallel on different parts of the system without worry about merge conflicts. We are now ready to celebrate our glory on a Viking feast.

Modularizing old monolith - Rails Engines to the rescue!

Mikołaj Wawrzyniak — Tue, 21 May 2019 12:08:25 +0000

TLDR

This post is a shout out to the amazing features of Rails that allow developers to extract code fairly easily from monolith application into stand-alone modules. It aims to point out the advantages of Rails engines over microservices and acknowledges the cons of engines solution.

About Me

I’m a back-end developer, and I got hooked up on Rails over 4.5 years ago. Since then, I’ve spent almost three years as a professional working on a single product initially developed in 2011. This is my first post, and I hope you like it.

Introduction

A few weeks ago, I started researching ways to improve the architecture of my company’s main application, which is fairly old and fairly big (81,980 commits and counting). Over time, many developers have tried to keep it in the best possible shape. Despite going beyond the standard MVC model and adding layers to the application, it is no longer sufficient. We have encountered typical problems related to big monolith systems: very large cognitive load required to maintain the mental model of a system, slow test suite, dead code, etc.
In such a situation, the first idea that pops out is to transition such a system into a shiny microservice world. As tempting it sounds, such as transition is not always easy or even possible for a number of reasons:

It moves application codebase complexity into the system infrastructure architecture
It introduces new kinds of problems, mostly with communication, synchronization, and integration.
It requires a significant amount of base work to prepare the system for such change before even a single business module can be moved out of a monolith
It makes the local development environment setup more complex

Teams choosing a microservices approach should have a number of skilled software architects along with dev ops specialists, and a fair amount of time, to accomplish such goal.
As for my case, microservices are off the table. I have continued looking, and I have noticed a convenient middle way, as you will read on . . . .

Rails Engines

Engines are similar to a standard Rails application but in miniaturized form. Their structure is very similar to a typical Rails project; one can find inside all familiar MVC parts of Rails architecture, such as models, views, controllers, jobs, and channels. There are a few minor differences, but what is most important, and which makes them such a useful tool, is that instead of being run as separate applications, Rails engines are mounted inside larger systems. This feature also implements a convenient mechanism that prevents name collisions between host application and engine code, while still allowing for cross-referencing when needed.
With that in mind, one of the coolest use cases of engines is the ability to extract code grouped in namespaces inside monolith host application into engines that are mounted in the parent application. This solution delivers most of the promises of microservices, such as system modularity, reduced cognitive load required to work on a single service and decreased execution time of test suite.
Moreover, Rails engines shine in a few particular areas:

Because they are mounted inside the host application, they are not affected by communication and integration issues related to independent microservices.
There is an easy setup; in fact, Rails engines could be initiated with a single Rails command.
The smoothness of extracting bits of code, as an engine, at first could be merely a subdirectory of a host application. Files could be moved between them using drag and drop.
The ability to be extracted to a stand-alone repository, in the form of a gem. And be reused by the number of applications.
Different versions could be included in different apps, based on the used gem version number, which makes developing new features and A/B testing simpler.

Because there is no silver bullet in software development, this approach also has its cons:

The host application, besides being composed with many independent modules, is still a single system that could be affected by dependencies conflicts among multiple engines.
Rails engines do not offer the technological freedom of microservices.
The host application still requires a significant amount of hardware resources, as a monolith would.
Despite its convenience, the ability to cross-reference the host application and the mounted engine also makes it easier to produce unwanted coupling between them.

Summary

Based on the above analysis, the pros outweigh the cons by far, particularly for smaller developer teams that do not have many resources to perform a transition to microservices. Even in terms of such transition, Rails engines could be used as a middle step, allowing a series of smaller changes rather than one big leap. Each previously extracted engine could be mounted later on in a smaller Rails app and used as microservices itself. What also should be mentioned is that engines are well ‘battle tested’ as some very popular solutions adopt them (e.g. devise and spree gems),
I hope you’ve enjoyed this post. In my next one, I’ll provide a tutorial for ‘Hello World’ in the Rails engines world.

Learning from backend to full stack dev

Mikołaj Wawrzyniak — Thu, 28 Feb 2019 09:09:33 +0000

Hi fellow devs!

I've been working as RoR developer since couple of years now. Recently I've decided to widen my skills with some frontend knowledge.

Since then I'm looking around web, to find best jump start resources for person with my type of background. Beside strong backend and programming knowledge. I'm also familiar with fairly basics of javascript, html, and css (but all of this is surely heavily outdated and unorganized). I want to follow current frontend roadmap. Could You suggest any books and courses which are worth of interesting in?

DEV Community: Mikołaj Wawrzyniak

Extracting Rails Engine by example - Vikings social media

Table Of Contents

Introduction

Engine preparation

Initialize engine

Rspec initialization

Database connection configuration

Parent app preparation

Request test creation

Engine installation

Code transfer

Migrations

Models

Factories

Rest of Rails parts

Not standard Rails components

External dependencies

Modularizing old monolith - Rails Engines to the rescue!

TLDR

About Me

Introduction

Rails Engines

Summary

Learning from backend to full stack dev