DEV Community: Morten Trolle

Don't use instance variables in Rails Validators

Morten Trolle — Sun, 13 Apr 2025 09:45:45 +0000

Take a look at this code:

class Comment < ApplicationRecord
  validates_with Validators::CommentContentValidator
end

module Validators
  class CommentContentValidator
    def validate(comment)
      @comment = comment

      if contains_bad_language?
        @comment.errors.add(:comment, :bad_language)
      end

      if contains_unwanted_urls?
        @comment.errors.add(:comment, :bad_url)
      end
    end

    def contains_bad_language?
      # Imagine this takes a few seconds
      BadWords.find_each do |bad_word|
        break true if bad_word.has_a_bard_word?(@comment.message)
      end
    end

    def contains_bad_urls?
      @comment.message.match?(/.../)
    end
  end
end

A first glance it looks harmless right? And you might not see any issues for a long time.

But Rails actually initializes your Validators::CommentContentValidator at the time it loads the Comment model and the result is your instance variable @comment is not thread safe in your controllers.
If you get many requests saving comments at the same time you'll end up mixing up the @comment between requests.

E.g. imagine these two requests happens in the same split second:

request_a: Comment.create!(...) # contains a bad url
request_b: Comment.create!(...) # is valid
request_a: Calls Validators::CommentContentValidator#validate(comment)
request_b: Calls Validators::CommentContentValidator#validate(comment)

Request A starts out validating bad words, but before it's done request B's validation has started and now @comment in Validators::CommentContentValidator is overwritten by request B's Comment.

When request A gets to contains_bad_urls? it is now using the comment of request B and thereby making request A valid.

The solution
If you are using custom validators, then make sure to pass the variable along in all method calls:

module Validators
  class CommentContentValidator
    def validate(comment)
      if contains_bad_language?(comment)
        comment.errors.add(:comment, :bad_language)
      end

      if contains_unwanted_urls?(comment)
        comment.errors.add(:comment, :bad_url)
      end
    end

    def contains_bad_language?(comment)
      # Imagine this takes a few seconds
      BadWords.find_each do |bad_word|
        break true if bad_word.has_a_bard_word?(comment.message)
      end
    end

    def contains_bad_urls?(comment)
      comment.message.match?(/.../)
    end
  end
end

Running StimulusJS & ImportMap with asset pipeline / Rails 6

Morten Trolle — Thu, 23 Mar 2023 17:47:39 +0000

I've never been a fan of Webpacker. So when starting a new project back in the Rails 5.1 days I avoided Webpacker and stuck to asset pipeline.

We have a ton of Coffeescript. And later on we upgraded to Rails 6.1, but stuck to our beloved asset pipeline.

Now Rails 7 is in front of us, but we are not quite ready to move to Rails 7 yet. But we wanna start taking advantage of StimulusJS and ImportMap and start slowly migrating our old Coffeescripts to ES6 and StimulusJS.

From the get go I tried converting to the full turbo suite, also upgrading Turbolinks 5 to the new Turbo. By the looks of it, it worked out great. But once I started to submit forms I realized we are far from ready to use Turbo.
With Turbo by default, Rails / Turbo will commit all forms with Ajax and Turbo will require your controllers to render http bad request on error and redirect with http status 303 (see other). But Rails renders default 302 (found) and then Turbo will try to handle that response instead of actually redirecting.

So we quickly gave up on Turbo. For now. That requires much more effort in our controllers than we were ready to do.

But back to the topic. Getting StimulusJS to work. I didn't care about ImportMap honestly, but StimulusJS does care. Either it's ImportMap or Webpacker and with that selection I had no doubt we needed ImportMap.

TL;DR

I updated my Gemfile to add:

# Hotwire's modest JavaScript framework [https://stimulus.hotwired.dev]
gem "stimulus-rails", '~> 1.2.1'
# Use JavaScript with ESM import maps [https://github.com/rails/importmap-rails]
gem "importmap-rails"

I ran my bundle install and then installed ImportMap rails importmap:install
Finally I could run rails stimulus:install

Now in app/assets/config/manifest.js I added the lines:

//= link_tree ../../javascript .js
//= link_tree ../../../vendor/javascript .js

Also very import, I renamed my app/assets/javascripts/application.js to app/assets/javascripts/gems.js to avoid naming conflicts with the new Stimulus created app/javascripts/application.js.

And finally in app/views/layouts/application.html.erb I removed

<%= javascript_include_tag 'application', 'data-turbolinks-track': 'reload' %>

in favor of:

<%= javascript_include_tag 'gems', 'data-turbolinks-track': 'reload' %>
<%= javascript_importmap_tags %>

Done.

Now I could run the mandetory test of creating my hello_controller.js and in a view add <div data-controller="hello"></div> and see my connect() method could write to the div innerText. Success!

Creating and restoring manual AWS Elasticsearch snapshots in Ruby

Morten Trolle — Fri, 02 Apr 2021 13:42:27 +0000

AWS makes regular automated snapshots of your hosted Elasticsearch services. But if you for whatever reason wanna create a manual snapshot, you can only do this scripted. At the time of writing this there's no way of doing this from the AWS console.

This Ruby script will help you creating manual snapshots of an AWS hosted Elasticsearch service using signed requests.

I had to create a manual snapshot of my ES service to migrate to a new VPC. You cannot change the VPC for an existing service, so I had no other option than creating a new ES service, manually snapshotting my old service and restoring this snapshot on the new service. And btw this is the documented way to do it by AWS.

Before you begin, start by reading the guide from AWS to familiarize yourself with the process:
Creating Amazon ES Index Snapshots

These are the steps you need to take to prepare yourself to create the manual snapshots and restoring them.

Start by creating an AWS S3 bucket for your snapshots.
Next go to the AWS IAM console and create a new policy. The content of the policy is also listed in the above link:

{
  "Version": "2012-10-17",
  "Statement": [{
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::s3-bucket-name"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::s3-bucket-name/*"
      ]
    }
  ]
}

Now create an AWS IAM Role - this will be used in the script to permit the snapshot.

The console will ask for a use case. Select DMS for now - we'll change this later as Elasticsearch is not listed.
Click next to set the role permissions. Here you can search for the policy you just created.
Name your role something meaningful and create it. You'll have to reference it in the script.
Once created find your role and open it in the console.
Click on the Trust Relationsships tab and click Edit trust relationship
Replace the existing policy with the following:

{
 "Version": "2012-10-17",
 "Statement": [{
   "Sid": "",
   "Effect": "Allow",
   "Principal": {
     "Service": "es.amazonaws.com"
   },
   "Action": "sts:AssumeRole"
 }]
}

The script

Now we are ready to configure the script.
I'm using the aws-sdk-elasticsearchservice gem - also listed in the script. Start by installing it:

gem install aws-sdk-elasticsearchservice

You need to do some small configuration in the script. The AwsEsSnapshot class defines a number of constants that you should replace with your own values:

AWS bucket name and region (notice you can only migrate Elasticsearch to a new service in same region)
Role arn for your freshly created role. It's listed in the top of the summary screen in the AWS console.
Finally you'll have to name a repository and snapshot. For a one time migration these values are not that important, but if you wanna continue to create manual snapshots you might wanna set some meaningful names.

Finally I'm fetching access key id and secret from environment variables. You can run your script like this:
AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=xxx ruby miration.rb

gem 'aws-sdk-elasticsearchservice'
require 'aws-sdk-elasticsearchservice'

class AwsEsSnapshot
  attr_accessor :logger
  REPO = 'my-repository-name'
  SNAPSHOT_NAME = 'my-snapshot-name'
  ROLE_ARN = 'arn:aws:iam::123456789012:role/TheSnapshotRole'
  BUCKET = 's3-bucket-name'
  REGION = 'eu-west-1'
  ACCESS_KEY_ID = ENV['AWS_ACCESS_KEY_ID']
  SECRET_ACCESS_KEY = ENV['AWS_SECRET_ACCESS_KEY']

  def initialize(logger: Logger.new(STDOUT))
    @logger = logger
    @logger.info('AwsEsSnapshot') { <<~HEREDOC
        Initialized with:
        Repository: #{REPO}
        Snapshot name: #{SNAPSHOT_NAME}
        Role ARN: #{ROLE_ARN}
        S3 bucket: #{BUCKET}
        AWS Region: #{REGION}
      HEREDOC
    }
  end

  # Creating a signed request
  def signer
    @signer ||= Aws::Sigv4::Signer.new(
      service: 'es',
      region: REGION,
      access_key_id: ACCESS_KEY_ID,
      secret_access_key: SECRET_ACCESS_KEY
    )
  end

  def signature(method:, url:, payload: nil)
    signer.sign_request(
      http_method: method.to_s.upcase,
      url: url,
      body: payload ? payload.to_json : nil
    )
  end

  def request(method: :put, host: nil, path:, payload: nil)
    url = host + '/' + path
    signature = signature(method: method, url: url, payload: payload)
    uri = URI(url)

    logger.debug("AwsEsSnapshot#request") { "#{method.to_s.upcase}ing #{payload} to #{url}"}

    Net::HTTP.start(uri.host, uri.port, :use_ssl => true) do |http|
      request = send(method, uri)
      request.body = payload.to_json if payload
      request['Host'] = signature.headers['host']
      request['X-Amz-Date'] = signature.headers['x-amz-date']
      request['X-Amz-Security-Token'] = signature.headers['x-amz-security-token']
      request['X-Amz-Content-Sha256']= signature.headers['x-amz-content-sha256']
      request['Authorization'] = signature.headers['authorization']
      request['Content-Type'] = 'application/json'
      response = http.request request
      if response.code.to_i == 200
        logger.debug("AwsEsSnapshot#request") { response.body }
      else
        logger.fatal("AwsEsSnapshot#request") { response.body }
        raise "Received unexpected status code #{response.code} in response."
      end
      response
    end
  end

  def post(uri)
    Net::HTTP::Post.new uri
  end

  def put(uri)
    Net::HTTP::Put.new uri
  end

  def get(uri)
    Net::HTTP::Get.new uri
  end

  def register(host:)
    payload = {
      type: 's3',
      settings: {
        bucket: BUCKET,
        region: REGION,
        role_arn: ROLE_ARN
      }
    }

    request host: host, path: "_snapshot/#{REPO}", payload: payload
    logger.debug("AwsEsSnapshot#request") { "Registered S3 bucket on #{host}" }
  end

  def create_snapshot(host:)
    # First we'll ensure our S3 bucket is registered
    register(host: host)

    request host: host, path: "_snapshot/#{REPO}/#{SNAPSHOT_NAME}"
    logger.debug("AwsEsSnapshot#request") { "Completed snapshot for #{SNAPSHOT_NAME}" }
  end

  def restore(host:)
    # First we'll ensure our S3 bucket is registered
    register(host: host)

    payload = {
      indices: "-.kibana*,-.opendistro_security",
      include_global_state: false,
      ignore_unavailable: true,
    }

    request method: :post, host: host, path: "_snapshot/#{REPO}/#{SNAPSHOT_NAME}/_restore", payload: payload
    logger.debug("AwsEsSnapshot#request") { "Restored snapshot for #{SNAPSHOT_NAME}" }
  end

  def status(host:)
    request method: :get, host: host, path: "_snapshot/_status"
  end
end

Now you are ready to use the script.
By default it's logging to STDOUT, but you can initialize with a custom logger:

es = AwsEsMigration.new(logger: MyCustomLogger.new)

There are tree methods of interest.
AwsEsMigration#create_snapshot
Expects a host: argument. This is your existing Elasticsearch service endpoint URL from where you wanna create a snapshot.

es.create_snapshot(host: 'https://my-legacy-endpoint-url.eu-west-1.es.amazonaws.com')

AwsEsMigration#status
will tell you the current snapshot progress. It's my experience the snapshot is completed fairly quickly.
Like litereally in minutes for a 10gb ES storage.

puts es.status(host: 'https://my-legacy-endpoint-url.eu-west-1.es.amazonaws.com').body

AwsEsMigration#restore
Finally if you like me need to restore a snapshot:

es.restore(host: 'https://my-new-endpoint-url.eu-west-1.es.amazonaws.com')

I hope this will help somebody out there :)

Don't name your Rails model Image

Morten Trolle — Tue, 23 Mar 2021 11:27:58 +0000

I named my model Image and I'm now realizing that's a bad idea.

I've never used my model with a form previously as it belongs to another model. But now I had to build a form to allow the user to update some of the attributes of my images.

<%= form_with model: image do |f| %>

But suddenly I got errors from the asset pipeline

The asset "[Image" is not present in the asset pipeline.

I even tried providing my form the image path directly:

<%= form_with model: image, url: image_path(image) do |f| %>

But got the exact same error. Okay - so now I realized that the asset pipeline is actually responding to my image_path, even though I created my images routes in config/routes.rb with the usual resources :images syntax.

So long story short. Don't name your model Image. It's always easier to use Rails conversions instead of having to work around them.

Should you do it anyways, then help your self by naming your routes customary. In config/routes.rb you can do it with the as option to your resources statement:

resources :images, as: :img

which gives you the routes helpers:

new_img_path # new
img_path(image) # show
edit_img_path(image) # edit
img_index_path # index - notice it requires index to be specified in method name

and in the form_for you need to tell Rails to use your custom routes:

<%= form_with model: image, url: img_path(image), method: :put do |f| %>

It's not perfect. But when the damage is done it's a lot easier to use custom route names than renaming your model.

ActiveStorage: Attaching file inline in Model.create / update

Morten Trolle — Mon, 08 Mar 2021 14:23:22 +0000

In the Rails guides you learn you can attach a file to a model using the attach method on you attachment:

@message.image.attach(
  io: File.open('/path/to/file'),
  filename: 'file.pdf'
)

Super useful when your record is already created. But if you need to create a new record you can also attach your file directly in the create command:

Message.create(
  foo: 'baa', 
  image: {
    io: File.open(Rails.root.join('image.png')),
    filename: 'image.png'
  }
)

Migrating from Refile to ActiveStorage

Morten Trolle — Thu, 04 Mar 2021 16:03:24 +0000

Historically I've been using the Refile gem for attaching files to my Rails models. But the gem hasn't been maintained for a long time and since Rails 5.2 ActiveStorage has been my preferred way to go.

These days I've been working on upgrading some of our older Rails 5.1 applications to Rails 6.0 and now I started seeing deprecation warnings on Refile, so It's time to say goodbye to Refile and hello to ActiveStorage.

Here I'll show you how I migrated my users avatar images from Refile to ActiveStorage.

The basics

On my User model I've replaced Refile's attachment :avatar with ActiveStorage's has_one_attached :avatar

In my Users#form view I've replaced the Refile field:

<%= f.attachment_field :avatar %>

with a regular file_field for ActiveStorage:

<%= f.file_field :image %>

Finally in my views when I wanna show the avatar image I moved from Refile's:

<%= image_tag attachment_url(user, :avatar, :fill, 75, 75) %>

to ActiveStorage:

<%= image_tag user.avatar.variant(resize_to_limit: [75, 75]) %>

You should check out the Rails guides for more options on ActiveStorage.

The migration

Great, with my frontend ready to handle ActiveStorage let's move on to the actual file migrations.
I've been using AWS S3 for hosting my Refile files.

Refile relies on the AWS SDK v2. Before I ran this migration I've removed the Refile gems including the AWS SDK v2 with the following gems for ActiveStorage in my Gemfile:

gem "aws-sdk-s3", require: false
gem 'image_processing'

Historically my User model just had a single Refile database field: avatar_id (string).
I wish I had used Refile's content_type detection as well, but I didn't, so in my migration I decided to use ImageMagick to detect the file type of my users images.

I created the following migration file in db/migrate/moving_from_refile_to_active_storage.rb:

require 'mini_magick' # included by the image_processing gem
require 'aws-sdk-s3' # included by the aws-sdk-s3 gem

class User < ActiveRecord::Base
  has_one_attached :avatar
end

class MovingFromRefileToActiveStorage < ActiveRecord::Migration[6.0]
  def up
    puts 'Connecting to AWS S3'
    s3_client = Aws::S3::Client.new(
      access_key_id: ENV['AWS_S3_ACCESS_KEY'],
      secret_access_key: ENV['AWS_S3_SECRET'],
      region: ENV['AWS_S3_REGION']
    )

    puts 'Migrating user avatar images from Refile to ActiveStorage'
    User.where.not(avatar_id: nil).find_each do |user|
      tmp_file = Tempfile.new

      # Read S3 object to our tmp_file
      s3_client.get_object(
        response_target: tmp_file.path,
        bucket: ENV['AWS_S3_BUCKET'],
        key: "store/#{user.avatar_id}"
      )

      # Find content_type of S3 file using ImageMagick
      # If you've been smart enough to save :avatar_content_type with Refile, you can use this value instead
      content_type = MiniMagick::Image.new(tmp_file.path).mime_type

      # Attach tmp file to our User as an ActiveStorage attachment
      user.avatar.attach(
        io: tmp_file,
        filename: "avatar.#{content_type.split('/').last}",
        content_type: content_type
      )

      if user.avatar.attached?
        user.save # Save our changes to the user
        puts "- migrated #{user.try(:name)}'s avatar image."
      else
        puts "- \e[31mFailed to migrate the avatar image for user ##{user.id} with Refile id #{user.avatar_id}\e[0m"
      end

      tmp_file.close
    end

    # Now remove the actual Refile column
    remove_column :users, :avatar_id, :string
    # If you've created other Refile fields like *_content_type, you can safely remove those as well
    # remove_column :users, :avatar_content_type, :string
  end

  def down
    raise ActiveRecord::IrreversibleMigration
  end
end

Just to be safe I chose not to delete Refile files from my S3 bucket in this migration file. I could of course have done so, but I choose to delete the storage and cache folders in my bucket manually instead.

That's it. After having run this migration all my users avatar images are now handled through ActiveStorage.

Good bye Refile.

Prevent Whenever scheduled cron jobs to run simultaneously in Rails.

Morten Trolle — Mon, 15 Feb 2021 10:23:42 +0000

In Simply.TV we have a number of cronjobs running to surveil some of our internal processes.

These jobs are scheduled to run every single minute using the whenever gem and its runner job type.

We have a job scheduled to run every single minute:

every 1.minute do
  runner 'Parse.detect_crashed_jobs'
end

It's using the rails runner command to execute the method Parse.detect_crashed_jobs and is doing that every minute.

Unfortunately some times our load rises on the server for various reasons and there's a chance that one of these jobs ends up running for more than a minute. This can result in the same job being scheduled to run simultaneously as the crontab will just keep processing the job every minute.

Enter flock. flock is shipped with most Linux distributions, at least this is tested on Linux Ubuntu 16.4 and upwards.

I've introduced a new job type and template to the whenever config/schedule.rb file:

job_type :runner,  "cd :path && bin/rails runner -e :environment \":task\" :output"
set :job_template, "bash -l -c 'flock -n '\\''/tmp/:task-runner.lockfile'\\'' -c '\\'':job'\\'''"

The job_template is using the bash command exactly like whenever defaults, but I've wrapped the actual job execution in a flock command. flock writes a temp file (using the task as filename) and uses this temp file to track if the job is running already. If it's already running it's skipping execution and exits.
At the same time I'm also changing the job_type for runner slightly by wrapping the task in double quotes instead of single quotes which is whenevers default value. This is to allow the job_template to wrap the flock command in single quotes. It just makes everything easier.

Now whenever a job is scheduled using whenever via the runner method, the job will be executed through flock who'll ensure the same job isn't running twice.