DEV Community: Harut Margaryan

How to prevent race conditions in Rails apps

Harut Margaryan — Wed, 09 Oct 2024 18:25:53 +0000

Imagine a case where you have a model with callback when 2 requests are send at the same time, and a new record is created in that model with callbacks. Your goal is to trigger the callback 1 time to make sure no race conditions happens.

Lets have an example to go more deeper into that.

We have a courses and students table and each course can have many students. We wanna trigger an email when the specific course got its first student. Here if 2 or more students try to enter the course at the same time we can get several emails but our goal is to be just informed via 1 email.

class Course extend ActiveRecord
  has_many :students
  after_create :notify_via_email


  def notify_via_email
    if students.count == 1
      # send email to the administration
    end
  end
end

Here we have in general and there is no way to tell whether this is the second student or the first because of race condition.

There is a good solution which makes possible to prevent race conditions using redis or other in memory storage to store, have a sidekiq worker and make it to work as a queue so that one callback can be processed at any given time.

class EmailExecutorJob < ApplicationJob
  def perform(student)
    lock_key = "EmailExecutorJob:#{student.id}"

    if redis.get(lock_key)
      self.class.set(wait: 1.second).perform_later(student)
    else
      redis.set(lock_key, student.id, 1.second)
      if students.count == 1
        # send email to the administration
      end
      redis.delete(lock_key)
    end
  end
end

And in the students model we can have this code

class Course extend ActiveRecord
  has_many :students
  after_create :notify_via_email


  def notify_via_email
    EmailExecutorJob.perform_later(self)
  end
end

As you can see we converted model callback to trigger EmailExecutorJob which checks whether other job is running and delays the current one. We are storing current student id in redis with prefixed key and that way we clearly can tell whether there is another job for the same name is running or not.
In this way we can prevent any race conditions.

P.S.
I haven't tested this and give just an example, but if you get stuck anywhere feel free to contact and I will be glad to help with such scenarios.

How to get several data count in a single query by applying conditions

Harut Margaryan — Sun, 22 Sep 2024 15:42:56 +0000

Here I am gonna explain how it is possible to obtain multiple counts in one SQL query for speeding up purposes

Let's assume we have students table which looks like this

students
id
name
grade
course_id

Grade field can have an integer between 40 - 100 or null if there is no grade yet.

We wanna get count of students who

received grades which is above 80.
received grades which is below or equal to 80.
that haven't received any.

In generic situations we may create and do 3 requests, but this approach is costly as any separate attempt to query from database adss up some timing.

So the look of the query looks like this

SELECT 
    COUNT(CASE WHEN grade > 80 THEN students.id END) as group_1,
    COUNT(CASE WHEN grade <= 80 THEN students.id END) as group_2,
    COUNT(CASE WHEN grade is null then students.id END) as group_3
FROM students

Rather than generating separate queries, the conditions are inserted inside counts and everything is done inside SQL code.

P.S.
I used Postgresql database for testing the query

Import the database from the Heroku dump

Harut Margaryan — Wed, 10 Jul 2024 07:33:54 +0000

Occasionally, as developers, we have to import staging or production databases and restore them to our local system in order to recreate bugs or maintain the most recent version, among other reasons.

Manual work takes a lot of time and can result in mistakes.

I propose creating a shell script that will do everything automatically, giving us more time to focus on our work.

Here's an example

File name can be import_remote_db.sh for example

#!/bin/bash

# stop on first error
set -e

for ARGUMENT in "$@"
do
   KEY=$(echo $ARGUMENT | cut -f1 -d=)

   KEY_LENGTH=${#KEY}
   VALUE="${ARGUMENT:$KEY_LENGTH+1}"

   export "$KEY"="$VALUE"
done

command -v heroku >/dev/null 2>&1 || { echo >&2 "heroku command is required"; exit 1; }
command -v docker-compose >/dev/null 2>&1 || { echo >&2 "docker-compose command is required"; exit 1; }

DUMPFILE="tmp/db_dump.dump"

if [ -e $DUMPFILE ]
then
    echo "backup db found trying to restore it"
else
    echo "backup db not found downloading from heroku"
    heroku pg:backups:download --output=$DUMPFILE --app=${app:-heroku-app-name}
fi

# This will usually generate some warnings, due to differences between your Heroku database
# and a local database, but they are generally safe to ignore.
docker-compose run db pg_restore --verbose --clean --no-acl --no-owner -h db -p 5433 -U u_name -d db_name $DUMPFILE

Here, it first verifies that the packages docker-compose and heroku are installed.
Next, it attempts to retrieve the database backup file; if this fails, it attempts to fetch data from heroku. Keep in mind that we must first complete the heroku login instruction.

Using the docker-compose console, it adds to a local database after the backup is done. If you do not need to use docker images at all, you can simply replace this with your local PostgreSql server.

To run the script use
sh import_remote_db.sh
You can also specify app name via
sh import_remote_db.sh app=heroku_app_name

Note: Make sure you backup your data before running the script if the Heroku app does not have automated daily or hourly backups. If not, you risk running out of backup data.

I saved a ton of time and effort during my work with this straightforward script.

Using Async in Ruby on Rails for CSV export

Harut Margaryan — Tue, 09 Jul 2024 11:10:14 +0000

In this article, we'll go over the methods used to achieve async CSV export with Ruby on Rails.

Problem: When a large amount of data is exported and the server is unable to process it quickly after receiving client requests, a timeout error may occasionally occur with CSV exports.

The solution is to move the CSV export to a worker thread, process it there so that it doesn't slow down the main thread, and notify the client when the desired CSV file is prepared.

We implemented a general solution using the Command design pattern, which will enable us to run the same code for many types of CSV format outputs.

Technology used

Heroku
Ruby On Rails
Sidekiq as a worker thread
Redis for keeping the state of csv export processing
Filestack or any other cloud storage service

Here, two enums with the following structure are used.

  EXPORTABLE_REDIS_STATUSES = {
    processing: 'Processing',
    complete: 'Processed',
  }.freeze

  EXPORTABLE_REDIS_KEYS = {
    members_csv: 'MEMBERS_CSV_GENERATORS',
    all_tasks_csv: 'ALL_TASKS_CSV',
    my_tasks_csv: 'MY_TASKS_CSV',
  }.freeze

Here, we have two states: processing and processed. One is used to launch a worker thread, and the other is used to produce a CSV file once the task is completed.

The second enum is only used as a filter to prevent csv exports that aren't registered with the application.

This is how the command invoker class, which acts as a sidekiq worker, looks.

class Exports::ExportableCommandJob < ApplicationJob
  after_enqueue do |job|
    uuid = job.arguments.first[:uuid]
    redis_key = redis_collection_key(job.arguments.first[:redis_key])

    REDIS.hset(
      redis_key,
      uuid,
      { status: Constants::EXPORTABLE_REDIS_STATUSES[:processing] }.to_json
    )
  end

  def perform(uuid:, redis_key:, command:, params: {}, cleanup_interval: nil)
    params = JSON.parse(params).symbolize_keys unless params.is_a?(Hash)
    command = command.constantize.new(params)
    redis_key = redis_collection_key(redis_key)

    file_data = command.call

    tmp_file = Tempfile.new('upload', encoding: 'ascii-8bit')
    tmp_file << file_data
    tmp_file.flush
    tmp_file.rewind

    file_name = command.file_name
    uploaded_file = UploadFileService::UploadableFile.new(file: tmp_file, filename: file_name)
    details = UploadFileService.upload_file(uploaded_file)
    tmp_file.unlink

    file_path = details.metadata[:fileurl]

    generator = JSON.parse(REDIS.hget(redis_key, uuid))
    generator['status'] = Constants::EXPORTABLE_REDIS_STATUSES[:complete]
    generator['exportable'] = file_path
    REDIS.hset(redis_key, uuid, generator.to_json)
  end

  after_perform do |job|
    uuid = job.arguments.first[:uuid]
    redis_key = redis_collection_key(job.arguments.first[:redis_key])

    ExportableCleanup.set(wait: job.arguments.first[:cleanup_interval] || 1.hour)
                     .perform_later(uuid: uuid, redis_key: redis_key)
  end

  private

  def redis_collection_key(key)
    redis_key = key.to_sym
    Constants::EXPORTABLE_REDIS_KEYS[redis_key] || key
  end
end

We used the uuid and redis_key to set the job's status to processing after it was queuing, allowing us to monitor its progress at any time.

We accept a command class name and its arguments via params in the perform method, allowing us to invoke a function and expect the presence of CSV data. After that, we store the data in a temporary file and upload it using a fileStack service or any cloud storage service.
We obtain the file's url after putting it on the cloud, and we use it to set Redis to change the task state from processing to processed. The client can now request to get updated when the CSV export is completed and to receive generated URL for downloading.

For filtering purposes, the private method redis_collection_key has been used here.

In the end after_perform schedules a cleanup task, as shown in this example.

class ExportableCleanup < ApplicationJob
  def perform(uuid:, redis_key:)
    exportable_json = REDIS.hget(redis_key, uuid)
    unless exportable_json.nil?
      generator = JSON.parse(exportable_json)
      file_url = generator['exportable']
      UploadFileService.remove_file(file_url) unless file_url.blank?
    end

    REDIS.hdel(redis_key, uuid)
  end
end

Here it just removes data from Redis and file from cloud storage.

The invoker class call looks like this

def export_csv_async(args, redis_key)
  uuid = SecureRandom.uuid

  Exports::ExportableCommandJob.perform_later(
    uuid: uuid,
    redis_key: redis_key,
    command: 'CsvExportDataGenerator',
    params: args.to_h.to_json,
  )

  uuid
end

which returns the uuid and will return it to the client so it can make the state checking calls as described above.

This is a simple action which checks process state in Redis

class ExportableGeneratorsController < ActionController::API
    include HttpErrorHandling

    before_action :load_resource

    def show
      render json: { status: @generator['status'], fileUrl: @generator['exportable'] }
    end

    private

    def load_resource
      @exportable_key = Constants::EXPORTABLE_REDIS_KEYS[params[:key].to_sym]
      gen = REDIS.hget(@exportable_key, params[:uuid])
      return not_found('Process not found') if gen.nil?

      @generator = JSON.parse(gen)
    end
  end