DEV Community: Sergey Tsvetkov

Payload and parameters validation in Rails

Sergey Tsvetkov — Sat, 09 Mar 2024 19:08:10 +0000

Overview

Here we go again! A lot of text and code in this post 😄 We are going to continue where we stopped last time with focus on input parameters validation. In my opinion, this is the weakest and almost missing part of the standard Rails setup which can be easily improved, polished and extended. We are going to show how with a few tricks and tips one can solve validations once and for all.

In order to read and understand this article you don't necessary need to go through the previous one, but if you have not read first part you may notice some strange details which are not standard for Rails and may look awkward. If you find yourself a bit confused try to go read first part and come back. It explains a lot of basics we use here.

Consider subscribing to my Dev.to blog to to stay in touch! 😉 And feel free to ask questions in the comments section below.

You can also find me here:

Telegram: https://t.me/kgcodes
Twitter: https://twitter.com/kimrgrey

See you!

What do we have?

My professional life developed in such a way that most of the time I was working on different kinds of mobile applications mostly in transportation area: taxi, delivery, bike and car sharing, public transport. When you are building a common API based mobile app a lot of your every day work can be boiled down to the following sequence:

Take input from mobile
Validate this input
Store something in database
Make it possible to fetch records you created

Of course, you have some business logic and algorithms around that which creates actual value and essence of your application. This logic is what makes it worth money in the end of the day. It is often complicated and requires a lot of thinking to cook and deliver. That is why it is so important to offload most of the routine to your framework. And Rails really shines in such an environment.

Convention over configuration principle saves you from living in the world of making stack traces out of XML files, as we were joking back in the Java days 😄

Framework takes care of a database connection for you. It provides you with very convenient way of creating, validating and accessing your records via ActiveRecord.

You can render and send an e-mail in a few lines of code thanks to ActionMailer. Which I actually rarely use for my projects, but that is a topic for another day 😉

And, of course, asynchronous jobs are taken care of by ActiveJobs.

And so on and so far.

In many aspects Rails decides and does a lot of heavy lifting so you can think about important things. It saves a lot of arguing time as well. I remember good old days of battles about the best way to layout your data objects or structure your configuration files. Such a waste of time!

It is really good to live in a world where boilerplate is solved and be efficient, isn't it?

What do we want?

With time though I realized that there are a few caveats which spoil the party. One of the things almost neglected by Rails is input parameters validation.

Let's say we are developing delivery application for a dark store or dark kitchen. Shortly speaking, we have an app which allows user to pick products, put them into a cart, estimate a price, order instant or scheduled delivery and pay for it.

If you want to create such an app it is necessary to take care of many questions. How do we represent and calculate the price? Should we provide some promo codes? How do we allow service fee or handle surges? How user can sign up and login into the system? How do we manage and keep track of stocks? How picking and packing processes are organized? What should we do if something ordered is actually missing? All of that creates value. All of that is what matters the most. All of that pays your bills.

In addition to hard questions we also should provide our procurement team with ability to manage assortment. Our admins should be able to create products, categorize them, hide and show them in store, fill in descriptions, specify prices, tags and so on. Doesn't matter how big and cool your application is anyway you have to solve simple things too. It all grows up from the back office.

Let's start from creating very simplistic table and model for our products:

# db/migrate/20240309154032_create_products.rb

class CreateProducts < ActiveRecord::Migration[7.1]
  def change
    create_table :products do |t|
      t.string :name, null: false
      t.string :description, null: true
      t.timestamps
    end
  end
end

# app/models/product.rb

class Product < ApplicationRecord
  validates :name, presence: true
end

Now we should make it possible to create products through API requests. Again, obviously oversimplified code which does the job may look like this:

# app/controllers/products_controller.rb

class ProductsController < ApplicationController
  def create
    product = Product.create!(create_params)

    render json: product.as_json(only: [:name, :description])
  end

  private def create_params
    params.permit(:name, :description)
  end
end

Should we try to use it?

curl --location '127.0.0.1:3000/products' \
--header 'Content-Type: application/json' \
--data '{
    "name": "Lindt Excellence Dark 70% Cacao Bar",
    "description": "Extra fine dark chocolate Lindt EXCELLENCE Dark 70% Bar 100g. A finely balanced taste sensation, finished with a hint of fine vanilla."
}'

Response is looking good:

{
    "name": "Lindt Excellence Dark 70% Cacao Bar",
    "description": "Extra fine dark chocolate Lindt EXCELLENCE Dark 70% Bar 100g. A finely balanced taste sensation, finished with a hint of fine vanilla."
}

We should also have at least ability to see the list with pagination, update existing products and delete them. Let's keep it very simple as well:

# app/controllers/products_controller.rb

class ProductsController < ApplicationController
  DEFAULT_LIMIT = 25
  DEFAULT_OFFSET = 0

  def index
    products = Product.order(id: :desc)
      .limit(params[:limit] || DEFAULT_LIMIT)
      .offset(params[:offset] || DEFAULT_OFFSET)

    render json: {
      products: products.as_json(only: [:id, :name, :description])
    }
  end

  def create
    product = Product.create!(create_params)

    render json: product.as_json(only: [:id, :name, :description])
  end

  def show
    product = Product.find(params[:product_id])

    render json: product.as_json(only: [:id, :name, :description])
  end

  def update
    product = Product.find(params[:product_id])
    product.update!(update_params)

    render json: product.as_json(only: [:id, :name, :description])
  end

  def destroy
    product = Product.find(params[:product_id])
    product.destroy!

    render json: {}
  end

  private def create_params
    params.permit(:name, :description)
  end

  private def update_params
    params.permit(:name, :description)
  end
end

We have entire set of standard CRUD operations for our products. But even from this little example it is easy to see capabilities which Rails lacks out of the box.

First, it mixes all actions together. Which is not such big of a problem, but we have to do some naming exercise to differentiate create_params from update_params, for example. Plus constants DEFAULT_LIMIT and DEFAULT_OFFSET which are only relevant for index are present for every action. Core models, such as product, tend to have a lot of methods: update this, update that, toggle, activate, hide, show, etc. More actions you have in one controller harder it is going to find relevant stuff. And more tempting it will be to re-use some parts and bits. Which is not always a great idea.

Second, we have URL parameters, such as product_id for some methods. For others we have payload parameters, such as name or description. And sometimes we have both. Also name is required while description is completely optional.

Unless you can read and understand Ruby in general and Rails models particularly there is no way to know all of that. Code is going to grow and we will have more parameters and rules. Mobile and back office teams would like to know what endpoints do we have, what they should send and in which form. We can't ask them to use source code to find this out, because it is not obvious from the first glance at all. So, in order to provide some insights on our API we gonna need to spend time on documenting it.

Can we improve it a bit and save some time on documenting internal API-s without going too far?

Let's move things around, shall we?

What if for the beginning we isolate every action into separated class. That is not classic Rails approach, but taking into account our module friendly project structure from the previous article it should be possible to do it quite easily and the result should look quite nicely.

We call such classes "handlers". Their role is to take some set of parameters, payload, handle the request and respond with some data. It is important to remember, that our goal here is to improve developer's experience, coding speed and readability without going too far away from vanilla Rails common sense, so that our code won't surprise new people too much.

Here we define a base class for our handlers to share common logic:

# app/application_handler.rb

class ApplicationHandler
  attr_reader :controller

  def initialize(controller)
    @controller = controller
  end

  protected def render(**args)
    controller.render(**args)
  end

  protected def params
    controller.params
  end

  protected def headers
    controller.headers
  end
end

After that we can move our first action out from controller to a handler:

# app/products/handlers/create.rb

module Products
  module Handlers
    class Create < ApplicationHandler
      def handle
        product = Product.create!(payload)

        render json: product.as_json(only: [:id, :name, :description])
      end

      private def payload
        params.permit(:name, :description)
      end
    end
  end
end

As you can see, this file is very narrow focused. It covers one specific flow - product creation. It shares zero space with other flows. All methods, names and parameters are scoped. In the same moment, it contains all information required about this specific endpoint. In order to work with product creation flow you need this file and this file only.

I think it is quite obvious how other handlers will look like, so I'll share one more example here just to make a point clear:

# app/products/handlers/index.rb

module Products
  module Handlers
    class Index < ApplicationHandler
      DEFAULT_LIMIT = 25
      DEFAULT_OFFSET = 0

      def handle
        products = Product.order(id: :desc)
          .limit(params[:limit] || DEFAULT_LIMIT)
          .offset(params[:offset] || DEFAULT_OFFSET)

        render json: {
          products: products.as_json(only: [:id, :name, :description])
        }
      end
    end
  end
end

Once we moved all of our actions to handlers we should change the controller which, frankly speaking, doesn't make a lot of work now:

# app/controllers/products_controller.rb

class ProductsController < ApplicationController
  def index
    handler = Products::Handlers::Index.new(self)
    handler.handle
  end

  def create
    handler = Products::Handlers::Create.new(self)
    handler.handle
  end

  def show
    handler = Products::Handlers::Show.new(self)
    handler.handle
  end

  def update
    handler = Products::Handlers::Update.new(self)
    handler.handle
  end

  def destroy
    handler = Products::Handlers::Destroy.new(self)
    handler.handle
  end
end

We could make one extra step here and introduce some DSL or convention, so that calling of handlers is done for us "automagically". But based on my experience it is much better idea to keep a little boilerplate, because it makes whole thing understandable for any new developer joining the team. What we are looking at are just plain simple Ruby classes and methods. Nothing too fancy, too scary or too controversial.

Good job, but this move achieves only one thing. It gives us some namespace for programming and isolates one handler from another. But there is very little improvement in other areas.

Can we make things better? And what about validations?

Finally, validations!

As it was mentioned before, Rails does very good job in solving almost every common web development problem. But somehow it is not true for the most frequent one - input validation.

The best tool we have in Rails for validating parameters is params.require(:product).permit(:name, :description). Which is at very least limited, if not to say "primitive". Without going out of the box it is not easy to make any next step.

Luckily, there is a large pool of community wisdom around and outside of Rails which may help us a lot here. Instead of inventing our own wheel for now we will use one invented before us by others. Pretty much sure you have seen this magic used outside of Hogwarts before: https://dry-rb.org/gems/dry-validation.

As we agreed, each handler represents one endpoint with some set of parameters and it's payload. But while logic is right there, in the handle method, other stuff stays implicit. We can't immediately tell from code what is expected as an input. The idea is to make our contract with client explicit and very much official by using dry-schema right in our handlers.

First, let's add dry-validation into our Gemfile:

gem "dry-validation", "~> 1.10"

Now I'll add a few more methods into AppplicationHandler:

# app/application_handler.rb

def self.payload(&block)
  if block_given?
    klass = Class.new(ApplicationContract) do
      json(&block)
    end

    @payload_contract = klass.new
  end

  @payload_contract
end

def self.params(&block)
  if block_given?
    klass = Class.new(ApplicationContract) do
      params(&block)
    end

    @params_contract = klass.new
  end

  @params_contract
end

Method payload helps us to define a contract for the payload we expect to come in request. It uses helper method json which takes care of type coercion for us. And method params does the same for URL parameters. It uses another helper conveniently called params which coerces data coming from URL query. You can read more about it here and here.

Now we can go to one of the handlers and define what parameters and payload it expects:

# app/products/handlers/update.rb

module Products
  module Handlers
    class Update < ApplicationHandler
      params do
        required(:product_id).filled(:integer, gt?: 0)
      end

      payload do
        required(:name).filled(:string)
        optional(:description).maybe(:string)
      end

      def handle
        product = Product.find(params[:product_id])
        product.update!(payload)

        render json: product.as_json(only: [:id, :name, :description])
      end

      private def payload
        params.permit(:name, :description)
      end
    end
  end
end

That helps with documentation issues, but sadly contracts are not applied and enforced, so it is easy to break what you agreed on. Let's add some more code into ApplicationHandler to actually use our definitions:

# app/application_handler.rb

attr_reader :controller
attr_reader :params
attr_reader :payload
attr_reader :errors

def initialize(controller)
  @errors = []

  if self.class.params.present?
    result = self.class.params.call(controller.params.to_h)

    if result.failure?
      @errors << result.errors(full: true).to_h
    end

    @params = result.to_h
  end

  if self.class.payload.present?
    result = self.class.payload.call(controller.params.to_h)

    if result.failure?
      @errors << result.errors(full: true).to_h
    end

    @payload = result.to_h
  end

  @controller = controller
end

protected def render(**args)
  controller.render(**args)
end

protected def headers
  controller.headers
end

In the constructor we check if our class has contracts for params and payload defined. Both contracts if present are applied and the result is stored in an appropriate instance variable available as an attribute. In case of validation errors we store relevant messages in another instance variable which we can use later on. In addition we had to remove method params from ApplicationHandler. It is used to call controller.params but now it was replaced by the attribute.

There is one more problems to solve. By default Rails doesn't allow you to call to_h on params because it expects you to explicitly permit specific keys. But in our case contracts are taking care of validations and they are way-way more powerful than build-in tools. So, let's just get rid of that check:

# app/application_controller.rb

class ApplicationController < ActionController::API
  before_action { params.permit! }
end

Handler can now be changed to utilize all of what has been done:

# app/products/handlers/update.rb

module Products
  module Handlers
    class Update < ApplicationHandler
      params do
        required(:product_id).filled(:integer, gt?: 0)
      end

      payload do
        required(:name).filled(:string)
        optional(:description).maybe(:string)
      end

      def handle
        if errors.present?
          render json: { errors: errors }, status: 422
          return
        end

        product = Product.find(params[:product_id])
        product.update!(payload)

        render json: product.as_json(only: [:id, :name, :description])
      end
    end
  end
end

Let's hit our endpoint with some corrupted payload just to test it and see what happens:

curl --location --request PATCH '127.0.0.1:3000/products/7' \
--header 'Content-Type: application/json' \
--data '{
    "name": 100
}'

Here is what response with status 422 Unprocessable Entity looks like:

{
    "errors": [
        {
            "name": [
                "name must be a string"
            ]
        }
    ]
}

That's great! Contracts we defined in handler are not just there, they are actually being used and enforced. Checking errors in every handler is a bit noisy though, so let's move this part out to ApplicationHandler as well:

# app/application_handler.rb

def handle!
  if errors.present?
    render json: { errors: errors }, status: 422
    return
  end

  handle
end

Now instead of calling directly handle method from specific handler controller will call handle! defined in ApplicationHandler. First it checks validation errors and if everything is fine it call down to the handler logic itself.

# app/controllers/products_controller.rb

def update
  handler = Products::Handlers::Update.new(self)
  handler.handle!
end

The result is pretty much the same, but we don't need to remember about checking validation errors every time now. We just have to define a contract, everything else will work by itself:

# app/products/handlers/update.rb

module Products
  module Handlers
    class Update < ApplicationHandler
      params do
        required(:product_id).filled(:integer, gt?: 0)
      end

      payload do
        required(:name).filled(:string)
        optional(:description).maybe(:string)
      end

      def handle
        product = Product.find(params[:product_id])
        product.update!(payload)

        render json: product.as_json(only: [:id, :name, :description])
      end
    end
  end
end

Here is another handler after adapting it to the latest changes:

# app/products/handlers/index.rb

module Products
  module Handlers
    class Index < ApplicationHandler
      DEFAULT_LIMIT = 25
      DEFAULT_OFFSET = 0

      params do
        optional(:limit).filled(:integer, gt?: 0)
        optional(:offset).filled(:integer, gteq?: 0)
      end

      def handle
        products = Product.order(id: :desc)
          .limit(limit)
          .offset(offset)

        render json: {
          products: products.as_json(only: [:id, :name, :description])
        }
      end

      private def limit
        params.fetch(:limit, DEFAULT_LIMIT)
      end

      private def offset
        params.fetch(:offset, DEFAULT_OFFSET)
      end
    end
  end
end

Conclusion

With just one gem added and a few lines of code written on top of it we managed to solve one of the most annoying problems in Rails applications. Important part here is that we have not moved too far away from default Rails path, so everything looks and feels familiar and can be easily traced through. No magic tricks, just simple code which one can read and understand in ~ 15 minutes.

Next topic we are going to talk about is authentication and authorization. In the next article I'm going to show how we can authenticate users and admins and check their permissions in handlers using quite simple but very powerful approach. Please, subscribe to stay tuned!

Tuning Rails application structure

Sergey Tsvetkov — Thu, 15 Feb 2024 14:20:06 +0000

Overview

This post is quite big because I wanted to cover basic setup of the app in one go, so here is an overview:

First, we are going to talk about our needs and goals
Second, we will configure reasonable defaults
Third, we install some missing libraries
And last, we configure nice directory structure

Consider subscribing to my Dev.to blog to to stay in touch! 😉 And feel free to ask questions in the comments section below.

You can also find me here:

Telegram: https://t.me/kgcodes
Twitter: https://twitter.com/kimrgrey

See you!

What do we want?

What do you think is an actual reason for Ruby-on-Rails to become one of the most successful frameworks out there?

For a long period of time I was under impression that is is all about it's agility and freedom. Dynamically typed Ruby together with meta-programming magic allowed us to make incredible things with very little effort. And rubygems.org back than contained answers to all of the questions you could possibly have. Go crazy all alone and yet stay productive, nothing will stop you!

With time principles behind development of Rails apps have changed though. More enterprise level software, established businesses, large scale solutions, high load. More stability, more types, stricter rules. Less gems because every gem is a dependency to keep track of and support. Less end-to-end solutions. More custom development. More testing. And so on.

Yet framework itself is still feels quite good and lightweight, productive and pleasurable to work with. So, what is the root cause of this feeling?

As far as I see it, the main point here is that Rails simply helps you to solve real day-to-day tasks fast and easy. That's it. As easy as it sounds. Handle and parse user's request. Route it. Go to the database. Create or update records there. Validate them. Respond. All in a standard way known and agreed on by many people involved into the community. You don't have to explain how it works every time someone joins. You can pick up a project you left for a while on the shelf and everything is still clear.

Of course, it is not ideal. Do I think that ActiveRecord is the best choice in any situation you may find yourself in? Nope. But is works quite good for many cases. If you have only models and controllers out of the box, it means logic will stick to one place or another, right? Is that good? Well, depends on a size of your application. And, what's even more important, on a size of your team. Should we trust web sockets handling to Rails? Up until recently Ruby was not particularly good in handling long connections. That's correct. But if your scale is not that big, well, it could still work pretty well. And once you reached the boundaries you can always switch to AnyCable and use Go to handle such troubling connections if you wanted to. As a result a lot of small projects could have their access to real time features out of the box without even knowing about HTTP upgrade and other relevant stuff.

In other words, my feeling is that we hear a lot of critics from large corporate Ruby-on-Rails users on how hard it is to scale it's basic principles behind certain size after 10 years of work. But the truth is that we should focus more on those who can only be dreaming to have such a scale and 10 years of development behind them and who's task at the moment is to work efficiently and move forward fast to survive and strive.

It happened that for the last few years quite frequently I participate in 2 types of projects. Launching of something or re-launching of something. When you work in a small teams with significant constraints in your resources this ability to solve typical problems fast and in a standard way which Rails grants you is just a "must have". With time I collected some tips and tricks on tuning Rails setup from the get go which, first of all, helps you to bootstrap even faster and, secondly, allows you to go trough "middle size" stages of the project and "grow up" without too much pain.

Reasonable defaults

Let's start from an obvious thing: standard set of parameters for rails new configured once and more or less forever. The thing is that due to the nature of our projects (mobile apps with some parts of the system available via web as well) we usually drop the whole "view" level of the framework, using mainly API with JSON. Database is PostgreSQL. Testing with rspec. We do not use any code preloading gems such as spring. We send e-mails using separated service written in Node.JS with templates done in React (a topic for another conversation probably). We don't need WYSIWYG-editor or ability to receive and process incoming emails. All of that are parts you can opt in or out in Rails. Here is how it looks like:

➜ rails new --database=postgresql --api \
  --skip-spring \
  --skip-test \
  --skip-active-storage \
  --skip-action-text \
  --skip-jbuilder \
  --skip-action-mailer \
  --skip-action-mailbox \
  my_new_project

Quite hard to remember, isn't it? And easy to forget about something specific when you create a new project. And then it takes some time to carve it out. Luckily, Rails has a feature which allows you to configure it once using configuration file and forget about it, using selected keys as a default. And if happens that you need something from the list of disabled stuff, well, you can always override the default and enable it through CLI arguments.

The file is called .railsrc and is it assumed that it lives in your home directory. Here is how it is documented: click.

In our case .railsrc contains this:

➜ cat ~/.railsrc
--skip-spring
--skip-test
--api
--skip-active-storage
--skip-action-text
--skip-jbuilder
--skip-action-mailer
--skip-action-mailbox
--database=postgresql

Thanks to it creation of a new project boils down to the simple command rails new <PROJECT NAME>. Default arguments will be taken from the file, so you don't have to remember about them anymore:

➜ rails new my-awesome-project
➜ cd my-awesome-project && ls
Gemfile      README.md    app          config       db           log          tmp
Gemfile.lock Rakefile     bin          config.ru    lib          public       vendor

Latest Rails

Our next step is much less obvious, but it turns out to be quite useful from practical point of view with just a few downsides. Let's go to our Gemfile and change fixed version of Rails there to the main branch of Rails:

gem "rails", github: "rails/rails", branch: "main"

I almost can hear you screaming: "What's the hell? Are you out of your mind? Why one would do that?". Well, look... One of the biggest challenges in any Rails project is an upgrade of Rails itself. Even if backward compatibility of Rails itself is there for you still every time you upgrade some gems won't make it though the process. Plus you never have time to actually look into it. So, this process takes time. Sometimes it takes years. In many cases it never actually happens.

Frequently Rails upgrade means also Ruby upgrade which automatically means infrastructure upgrade. Nowadays you most likely have to change at least Dockerfile. There is a probability that this part is done by other people and they have their priorities. So, the complexity of simple Rails upgrade goes through the roof.

It was not that rare for me to see some projects stuck with ~> 3.0, ~> 4.0, ~> 5.0 and so on, depending on when the project was started initially. Development teams just had not enough courage to just go for an upgrade whatever it means. And with time this issue grew up to be a real deal problem. And, of course, quite frequent outcome in such case is to drop it all, abandon this old dirty outdated "monolith" and go for shiny new "micro-services" written in Go or Node. Seen it before? Me too. Sometimes it ends up good. But it takes resources, time and money. So, to avoid the trouble just use the latest available version all the time! Spread the effort thin.

What advantages does it give to you:

Main branch of Rails is quite stable thing. Bugs are being fixed. New issues happen sometimes, but critical stuff is not that frequent and most likely you won't even notice anything up until certain scale. But, of course, you should trust it completely. One day everything can go off rails 😄 What does it mean? Well, you just know: without good test coverage you won't survive. So, tests are must have. All in. Skin in the game. Specs are your life insurance. With good test coverage something still can go wrong. For example, you may experience some performance issues and memory leaks. So, monitoring, green-blue deployments and canary roll-outs are welcomed into your life as well. Configure them early and enjoy your life every since!
There are a lot of changes in Rails which impact it's performance. It means, that most of the time you have the fastest possible version of the framework in production. It is true for Ruby as well. Every next release gives 20-30% to speed. If you adopt early use of Ruby / Rails builds you don't need to compromise on performance anymore. As a bonus you have all of the latest features. For example, for a long time encrypted database fields support in ActiveRecord was available in master but not in any released version of the framework before Rails 7.
The reality is such that not every gem works with the latest version. Some of them are "stuck" on some specific major release and just won't install. Well, on the one hand it limits your choice to only libraries with very good community support. On another hand, if there is this one little thing you really need but it is not available for the latest Rails, why won't you clone it, fix it and make a PR to give back to the community, huh? To busy for that?

Well, let's say I convinced you and version of Rails has been changed to main. If the risk is too high then at least track every single Rails release and make it your priority to do an upgrade early on. That will save you a lot of time and pain in the future, believe me.

Useful gems

What's next? There are some libraries in the Gemfile we almost 100% are going to use but their are commented out by default. Examples are redis, bcrypt, rack-cors. Let's put them in!

Once we are done with default gems, should we look into something we usually use? That's jwt because we need session tokens for our API. Next comes our one and only sidekiq. For a long period of time it was the best in town solution for background jobs. Now we could also consider solid_queue or good_job. In development and testing groups we need rspec-rails, factory_bot_rails and ffaker. Dealing with money? Start doing it properly from the beginning! Do not forget to install money-rails. Once everything is added to the Gemfile do not forget to trigger bundle install.

Next you should generate required folders and configs for rspec using bin/rails generate rspec:install, as it is said in it's README. In addition it is useful to do bundle binstub rspec-core to generate bin/rspec. I'm not going to dive into rspec configuration itself here, because it works quite well out of the box and you'll learn what you need to know as you go.

Now let's take care of factories by creating a directory for them using mkdir rspec/factories.

What about sidekiq? If you decided to stick to it then go to config/application.rb and configure ActiveJob adapter there: config.active_job.queue_adapter = :sidekiq. Another option would be to use sidekiq directly and just forget about ActiveJob which would save you a few dives in the documentation and Google probably.

Taking into account we already opened application configuration, let's change our default time zone to UTC. Based on our experience, it simplifies your life a lot, because you don't have to be worried about location of your physical infrastructure and what time is it there. So config.time_zone = "UTC".

Modules and directories

Up until now we were doing more or less classical changes in our project. It's time to go a bit sideways. Here is a trick. By default, loader in Rails is configured in such a way that it loads all files from from any directory inside of the app folder, assuming that all of them will define classes from the main module. Easiest example here is a models folder. There you can find all database models, defined in the app, right? It's useful that way. When you need a User model you just drop a file models/user.rb into the project and use it without any module prefix: User.new. Loaders knows how to find it.

This approach is less cool when you would like to have some modules in your code. Which is not such a terrible idea in the project growing fast. Let's say you would like to have a class which defines JSON presentation for your user. It sounds reasonable to put it into users/presenters/show.rb. Like this:

module Users
  module Presenters
    class Show
      def initialize(user)
        @user = user
      end

      def as_json
        {
          id: @user.id,
          email: @user.email
        }
      end
    end
  end
end

Well, Zeitwerk won't support it:

➜ bin/rails c
irb(main):001:0> Users::Presenters::Show
Traceback (most recent call last):
        1: from (irb):1
NameError (uninitialized constant Users)

Because users folder belongs to app directory Rails will treat that the same way it treats models folder from our example above. You should not do Models::User there, right? But it is not very helpful in case of our presenter because Users::Presenters::Show is not mapped to users/presenters/show.rb. Layout assumed by Rails offers you to go for something like presenters/users/show_presenter.rb and, respectively, Users::ShowPresenter. Same as you do for, let's say, controllers. But this approach has some downsides. First, module is replaced by the suffix. Second, parts of the system connected to each other are located far away from file system point of view. As a result, you'll frequently find yourself changing multiple directories related to the same "domain" here and there.

Can we fix that without breaking the whole convention? Let's see first who and where adds folders from app directory into the loader as if they were root? After some digging you'll find this file:

paths = Rails::Paths::Root.new(@root)

paths.add "app", eager_load: true, glob: "{*,*/concerns}", exclude: ["assets", javascript_path]
paths.add "app/assets",          glob: "*"
paths.add "app/controllers",     eager_load: true
paths.add "app/channels",        eager_load: true, glob: "**/*_channel.rb"
paths.add "app/helpers",         eager_load: true
paths.add "app/models",          eager_load: true
paths.add "app/mailers",         eager_load: true
paths.add "app/views"

paths.add "lib",                 load_path: true
paths.add "lib/assets",          glob: "*"
paths.add "lib/tasks",           glob: "**/*.rake"

paths.add "config"
paths.add "config/environments", glob: -"#{Rails.env}.rb"
paths.add "config/initializers", glob: "**/*.rb"
paths.add "config/locales",      glob: "**/*.{rb,yml}"
paths.add "config/routes.rb"
paths.add "config/routes",       glob: "**/*.rb"

paths.add "db"
paths.add "db/migrate"
paths.add "db/seeds.rb"

paths.add "vendor",              load_path: true
paths.add "vendor/assets",       glob: "*"``

As you can see standard folders are being added into the paths here. In particular, we are interested in glob: "{*,*/concerns}". That's exactly the line where it says that everything in the app folder itself should be loaded (that's * part) and also files from concerns too (that's */concerns, such as models/concerns or controllers/concerns).

We found where it is defined, but it turns out that this configuration is used much-much later on. So, we can actually change it when our application is loaded. For example, we could do it in config/application.rb by adding following line:

# makes zeitwerk do not autoload subfolders of app folder by default
config.paths["app"].glob = nil

Here is how config/application.rb looks like after the change:

require_relative "boot"

require "rails"
require "active_model/railtie"
require "active_job/railtie"
require "active_record/railtie"
require "action_controller/railtie"
require "action_view/railtie"
require "action_cable/engine"

Bundler.require(*Rails.groups)

module MyAwesomeProject
  class Application < Rails::Application
    config.load_defaults 6.1
    config.time_zone = "UTC"
    config.api_only = true
    config.active_job.queue_adapter = :sidekiq

    # makes zeitwerk do not autoload subfolders of app folder by default
    config.paths["app"].glob = nil
  end
end

Let's try it out:

➜  my-awesome-project git:(master) ✗ bin/rails c
irb(main):001:0> Users::Presenters::Show
=> Users::Presenters::Show
irb(main):002:0>

File is being automatically loaded by Zeitwerk! But what about all the standard paths we had? Well, because they are explicitly configured in the file we discussed above no damage is done:

➜ bin/rails c
irb(main):001:0> User
=> User (call 'User.connection' to establish a connection)

Another nice side effect I usually exploit is that standard basic classes now can be also moved to the app folder:

➜ mv app/models/application_record.rb app
➜ mv app/controllers/application_controller.rb app
➜ mv app/jobs/application_job.rb app

Other stuff from the same category, like application_presenter.rb, can go there as well. It is not required and you can leave them where they were, but it sounds pretty reasonable to me, that basic classes can be found in the app directory with no additional effort spent to find them.

Conclusion

Well, that's basically it for now. I think it is enough for the beginning 😄 In the next series we are going to talk about standard tasks you may find in almost any application which have no nice solution provided by Rails: incoming request payload and params validation, session management, phone verification and so on.

Rust is taking off?

Sergey Tsvetkov — Wed, 14 Feb 2024 11:50:06 +0000

It is very interesting to observe the raise of Rust. Just a few years ago it was a niche language used by Mozilla and couple of other big tech companies as a modern replacement for C.

Linux kernel followed as well (see https://www.kernel.org/doc/html/next/rust/index.html). But still it was far away from main stream development most of the programmers doing every day.

Then Rust revolutionized JS tooling because of esbuild, JavaScript bundler and minifier created in 2020, which was order of magnitude faster than everything on the market at the moment.

This way Rust has become a huge part of everyday experience for many-many people building web sites using modern tooling. Not sure, if there is one new project done with webpack instead of esbuild now 🙂

Next big appearance happened when Elon Musk announced his new X.AI project with LLM called Grok and written from scratch in Rust. See "Engineering" section on https://x.ai/

After that Rust has come to the desktop environment with Zed, modern text editor with GPU rendering support from authors of Atom. See https://zed.dev/ for details.

When developers adopt something for the projects they do during their free time or for their own side projects I take it like a proper take off of the language or a framework. Looks like Rust has finally achieved it.

The Skunk Works Rules

Sergey Tsvetkov — Tue, 13 Feb 2024 23:29:14 +0000

Most likely you have heard about Lockheed or, later on, Lockheed Martin and everything they have created and built. Shortly speaking, you can see them as a prototype for "Stark Industries" from the "Iron Man" 🙂

And if you are into the history of the Cold War or the Silicon Valley history (which in many ways are the same stories especially early days) there is also a probability that you know a department called "Skunk Works". It is a team of engineers inside of Lockheed Martin who designed, built and tested most of those amazing fighter and reconnaissance airplanes they are known for.

The "Skunk Works" was founded by the guy known as Clarence Leonard "Kelly" Johnson. Absolutely legendary man. And he had rules which they applied to organize and operate the "Skunk Works" from Day 1.

Here are a few of them I like the most:

Rule №1: The "Skunk Works" manager must be delegated practically complete control of his program in all aspects. He should report to a division president or higher.
Rule №3: The number of people having any connection with the project must be restricted in an almost vicious manner. Use a small number of good people (10% to 25% compared to the so-called normal systems).
Rule №5: There must be a minimum number of reports required, but important work must be recorded thoroughly.

When I found and read those rules for the first time it was just so exciting thing to see! And since than every time I see this sarcastic "so-called normal systems" line it makes me smile. Yup, I agree, old man, and I know what are you talking about 🙂 Amazing, isn't it?

Here you can find all of the rules still published on Lockheed Martin web site: https://www.lockheedmartin.com/content/dam/lockheed-martin/aero/photo/skunkworks/kellys-14-rules.pdf

Here you can read more about the "Skunk Works": https://en.wikipedia.org/wiki/Skunk_Works

Here is a personal history of Kelly Johnson: https://en.wikipedia.org/wiki/Kelly_Johnson_(engineer)

There is also a book on the topic: https://www.amazon.com/Skunk-Works-Personal-Memoir-Lockheed/dp/0316743003

Can I use has_secure_token in ActiveRecord?

Sergey Tsvetkov — Tue, 05 May 2020 07:57:21 +0000

Recently reviewing few different projects I’ve noticed that method has_secure_token provided by ActiveRecord is used a lot here and there for different kind of situations where it’s actually not very good idea to use it. Specifically, different scenarios of email / role confirmation or even some user authentication flows.

TL;DR. Please, do not use this method for generating reset password tokens or anything like this. That’s it. Now if you would like to know more about why not and how you could replace it then everything written below is for you.

My guess is that a lot of misunderstanding around has_secure_token is coming from another helper method available in ActiveRecord models and called very-very much similar - has_secure_password.

As you may assume has_secure_password provides basically complete flow of password based authentication for the model. Here is the usage example of this method you may typically see:

class User < ApplicationRecord
  validates :email, presence: true

  has_secure_password
end

user = User.create!(email: "test@example.com", password: "qwerty")

if user.authenticate("qwerty")
  # the password is good
end

The best thing about has_secure_password is that it takes care about security concerns you really would like to address for storing user’s passwords in 2020. Specifically, instead of saving actual password this method under the hood is transforming it into the digest generated by bcrypt and stores only this hash. So even if somebody will steal your entire database they will see only digests and won’t be able to reverse passwords out of hashes.

Good, back to has_secure_token. Here is the basic example you can meet in a real project:

class User < ApplicationRecord
  validates :email, presence: true

  has_secure_password
  has_secure_token :reset_password_token
end

user = User.create!(email: "test@example.com", password: "qwerty")
user.reset_password_token # some random 24 chars long token

Now open reset password implementation flow and you’ll see there something like:

if user.reset_password_token == params[:reset_password_token]
  user.reset_password_token = nil
  user.password = params[:password]
  user.password_confirmation = params[:password_confirmation]
  user.save!
end

The problem here is that in comparison to has_secure_password method has_secure_token does pretty much nothing to protect developers from simple mistakes. It stores generated token simply as it is. In plain text. I would call it has_unsecure_token instead to be honest!

What’s amazing here is that if you ask somebody around is it good to openly store passwords in database the answer most likely will be “no”. In the same moment by some reason same people assume it’s still good enough way of storing the token. Which actually allows to reset our securely digested password. And, in many cases, to get active session out of it. Doesn’t make any sense for me. Doubt it’s true? Then just google has_secure_token and take a look on examples people are using to demonstrate it.

Here are first 2 links from the top of my google response: here, here. Sure, those texts are written just to highlight the fact that such method exists in ActiveRecord API and they are using resetting password for the sake of example. Any experienced developer would have noticed it without a doubt. But it may hurt a lot of beginners pretty badly!

It’s reasonable to say that such method can be acceptable if there is some few minutes long invalidation period. And in some cases I would agree. But the problem is that has_secure_token is not taking any responsibility about it as well. In comparison to has_secure_password this helper is far away from complete production ready implementation. As a result, knowing the facts about has_secure_token, it’s pretty hard to imagine the situation when it’s actually safe to use it. The only one case I imagine after some thinking is promo code generation or something similar.

Another hard to understand thing for me is why it was decided to go with such poor implementation. Is it hard to write something more or less secure out of the box? What can you do to generate pretty much secure token and validate it?

Well, let’s see. I’ll start from the class which wraps and hides all scary hashing logic from us:

class Token
  def self.generate(cost: BCrypt::Engine.cost, valid_till: Time.now.utc + 24.hours)
    Secrets::Token.new(SecureRandom.base58(24), valid_till: valid_till, cost: cost)
  end

  attr_reader :valid_till

  def initialize(token, valid_till: nil, cost: BCrypt::Engine.cost)
    @token = token
    @valid_till = valid_till
    @cost = cost
  end

  def valid?(value)
    (@valid_till.nil? || @valid_till.utc > Time.now.utc) && BCrypt::Password.new(value) == @token
  end

  def to_s
    @token.to_s
  end

  def digest
    BCrypt::Password.create(@token.to_s, cost: @cost)
  end
end

Now let’s generate our token once user has started forgot password flow:

# /users/password/forgot

user = User.find_by(email: params[:email])

if user.nil?
  raise ActiveRecord::RecordNotFound
end

reset_password_token = Token.generate

user.reset_password_token = reset_password_token.digest
user.reset_password_token_valid_till = reset_password_token.valid_till

Important thing to notice here is that I’m storing the digest while validating it against the real token I may have sent in the email as part of the reset password form URL. Basically in the same way has_secure_password doing it with password_digest and authenticate. Nothing new here.

That’s pretty much it. It’s not a lot of code so I’m publishing it in a form of the snippet instead of packing it into gem. You are free to take it and own it because knowing for sure what happens in such a critical part of you system is very important. Stay of the safe side! See you!