DEV Community: Andrey Novikov

Fullstaq Ruby: First impressions, and how to migrate your Docker/Kubernetes Ruby apps today

Andrey Novikov — Wed, 07 Aug 2019 14:15:13 +0000

What is Fullstaq Ruby?

Fullstaq Ruby is a custom build of standard MRI Ruby interpreter with memory allocator replaced, security patches applied, and more goodies on the way.

If some old timers are here, they can remember REE–Ruby Enterprise Edition–from ancient times of Ruby 1.8.7 and Ruby on Rails 2.2 (almost ten years ago!). Ah, good ol’ times! You could install it via RVM or Rbenv, and some legacy applications are still running on it or have been just recently migrated. REE has a dozen of different patches on top of Ruby 1.8.7 to improve performance, reduce memory consumption, adjust obsolete security settings, and so on.

In MRI 1.9.x most of these problems were solved, and, as it gained adoption, REE became obsolete. But even modern “vanilla” MRI still has quirks that can be fixed relatively easy. The most annoying of them is memory bloat due to memory fragmentation.

So it is not at all surprising that Hongli Lai, the creator of REE, have released Fullstaq Ruby.

REE is dead, long live Fullstaq Ruby!

Why do we need it?

At one of our projects at Evil Martians we were experiencing severe memory bloat. Our application does a lot of IO, and we have a lot of Sidekiq processes with high concurrency setting (20 threads per process). This setting is optimal from the performance point of view because workers are mostly making requests to different remote APIs, our own database, and caches. But such a high level of concurrency also leads to high memory fragmentation. Our Sidekiq processes eat several gigabytes of RAM each.

Read more about choosing Sidekiq concurrency setting in the Sidekiq in Practice part 1 by Nate Berkopec.

We have decided to replace our MRI 2.6.3 to Fullstaq Ruby 2.6.3 with jemalloc to see how it will perform.

Now that's the difference!

We tried Fullstaq Ruby on a commercial application that runs in production and serves requests from paying clients around the clock.

First of all: nothing broke. Zero downtime!

Now, take a look at these monitoring graphs. Memory bloat of long-running processes has practically gone!

Web application processes have become very stable in memory consumption (4 times less memory!). Bloat still occurs sporadically, but still the readings show that about 50% less memory is consumed during spikes.
Background job workers (we are using Sidekiq) also lost two thirds of their weight. From 1.5-2 GB before to 500-700 MB after the migration to Fullstaq Ruby.
There is no noticeable difference in memory consumption for short processes (e.g., cron jobs)
We didn't notice any changes in response times or CPU utilization.

The graphs above prove that memory fragmentation was the reason for high memory consumption.

And that’s it—quite an improvement for swapping one ruby binary for another, isn’t it?

Alternatives?

If jemalloc isn't an option for you or you cannot afford to replace MRI with something else, you can try MALLOC_ARENA_MAX=2 spell to adjust MRI's standard glibc malloc behavior. Results will be close enough to Fullstaq Ruby to treat them almost as equal.

In our case, Ruby with limited number of malloc arenas (on the right) consumed about 50-100 MB more memory than Ruby with jemalloc (on the left).

Read more and see benchmarks of MALLOC_ARENA_MAX=2 in this post:

Cables vs. malloc_trim, or yet another Ruby memory usage benchmark

Vladimir Dementyev ・ Mar 19 '19 ・ 6 min read

#ruby #benchmarks #rails

We decided to stick with Fullstaq Ruby.

How to install?

At the moment the only way to install Fullstaq Ruby is to use deb or rpm packages (either installing directly or via repositories). But we deploy our app to Kubernetes cluster, so we need a Docker image. As there is no "container edition" available on the official website yet, so let’s build our own image–actually, it is not that hard!

Let's use Debian 9, as this is the Linux distribution being used by official Ruby Docker image, and define the Ruby version:

FROM debian:stretch-slim

ARG RUBY_VERSION=2.6.3-jemalloc

And then install prerequisites, add Fullstaq Ruby APT repository, install Ruby itself and cleanup apt caches—all in a single command to reduce the Docker layer size:

RUN apt-get update -q \
    && apt-get dist-upgrade --assume-yes \
    && apt-get install --assume-yes -q --no-install-recommends curl gnupg apt-transport-https ca-certificates \
    && curl -SLf https://raw.githubusercontent.com/fullstaq-labs/fullstaq-ruby-server-edition/master/fullstaq-ruby.asc | apt-key add - \
    && echo "deb https://apt.fullstaqruby.org debian-9 main" > /etc/apt/sources.list.d/fullstaq-ruby.list \
    && apt-get update -q \
    && apt-get install --assume-yes -q --no-install-recommends fullstaq-ruby-${RUBY_VERSION} \
    && apt-get autoremove --assume-yes \
    && rm -fr /var/cache/apt

Fullstaq Ruby also installs Rbenv as dependency but we don't need it in Docker, so let's add ruby and gems binaries to system $PATH in the same way that official Docker image for Ruby does:

ENV GEM_HOME /usr/local/bundle
ENV BUNDLE_PATH="$GEM_HOME" \
    BUNDLE_SILENCE_ROOT_WARNING=1 \
    BUNDLE_APP_CONFIG="$GEM_HOME" \
    RUBY_VERSION=$RUBY_VERSION \
    LANG=C.UTF-8 LC_ALL=C.UTF-8

# path recommendation: https://github.com/bundler/bundler/pull/6469#issuecomment-383235438
ENV PATH $GEM_HOME/bin:$BUNDLE_PATH/gems/bin:/usr/lib/fullstaq-ruby/versions/${RUBY_VERSION}/bin:$PATH

CMD [ "irb" ]

And that's it!

We’ve already built and published this image. You can pull it from our repository at quay.io:

docker pull quay.io/evl.ms/fullstaq-ruby:2.6.3-jemalloc-stretch-slim

Dockerfiles are available on GitHub: https://github.com/evilmartians/fullstaq-ruby-docker

Now we can just replace base image in our application Dockerfile:

-ARG RUBY_VERSION=2.6.3
+ARG RUBY_VERSION=2.6.3-jemalloc

-FROM ruby:${RUBY_VERSION}-stretch-slim
+FROM quay.io/evl.ms/fullstaq-ruby:${RUBY_VERSION}-stretch-slim

And deploy it to staging and then to production.

Feel free to do the same!

Recap

Migration is smooth. Just reinstall Ruby and gems, and everything should just work.
Application servers and background jobs worker processes should reduce memory consumption drastically.
There is no noticeable difference in memory consumption for short processes (like cron jobs or scripts).
Performance may slightly improve, but it depends on your workload profile.

Reporting non-nullable violations in graphql-ruby properly

Andrey Novikov — Fri, 19 Jul 2019 07:37:08 +0000

GraphQL encourages you to decouple your frontend from your backend. You can think more about your types’ “correctness” and less about particular view layout. To achieve that GraphQL provides strong type system. GraphQL doesn’t allow to return String instead of Integer and also doesn’t allow return nulls where API declares that type is non-nullable.

But sometimes your data will accidentally contain empty values where even you don’t expect them. Some external APIs will return incorrect or missing data, even if its contract disallows that. Sometimes it will be your own bug. But in any case, it is your responsibility to track for such failures and correct them.

If you use Ruby for developing GraphQL API, the chances are high that you use graphql-ruby. It is standard de-facto in Ruby because it is a very mature and feature-rich gem with a whole ecosystem of plugins. But “out of the box” there it doesn’t inform you about violations of these “non-nullable type” rules and just silently returns a partial response with error messages to the client.

But we want to know about non-null violations because it is our responsibility to develop bullet-proof applications. And according to the graphql-ruby’s documentation about type errors, there is a special hook to catch such errors. Let’s use it!

We’re using Honeybadger, so we’ll report this offense there:

class YourAppSchema < GraphQL::Schema
  def self.type_error(exception, query_context)
    if exception.is_a?(::GraphQL::InvalidNullError)
      Honeybadger.notify exception, context: { query: query_context.query.query_string }
    end
    super
  end
end

So far, so good. Soon we’ll able to see our first error in error tracker UI.

But wait, what’s it? There are errors about non-null violations in multiple different fields mixed in one error field. Aren’t they different bugs? What’s wrong?

It turns out that by default Honeybadger groups errors together by exception class, component (controller and action), and backtrace fragment. Please note that the exception message doesn’t count. In the case of GraphQL, it is always a single controller and action, so exceptions for absolutely different queries are getting grouped together. Let’s fix that!

According to the same documentation page, we need to calculate unique fingerprint string for every different error. So, what we need to include to this fingerprint for non-null violation?

Exception class? Yes, we need to distinguish GraphQL::InvalidNullError from other types of errors
Controller and action? No, they’re always the same. And even if an error will occur during triggering subscription, will it make any difference?
Backtrace? Hmm… I’m not sure. What do you think?
Type and field names, of course! We need to separate errors in different fields.

Let’s use an exception class name, GraphQL type name, and field name:

class ApplicationSchema < GraphQL::Schema
  class << self
    def type_error(exception, query_context)
      fingerprint = honeybadger_fingerprint(exception)
      Honeybadger.notify exception, fingerprint: fingerprint, context: {
        query:   query_context.query.query_string,
        user_id: query_context[:user]&.id,
      }
      super
    end

    private

    def honeybadger_fingerprint(error)
      case error
      when GraphQL::InvalidNullError
        "#{error.class}:#{error.parent_type.name}.#{error.field.name}"
      end
    end
  end
end

And that’s it: from now on exceptions for different fields lives in different Honeybadger errors.

Thank you for your attention! Let’s build reliable APIs!

Lefthook, Crystalball, and git magic for smooth development experience

Andrey Novikov — Wed, 10 Jul 2019 11:51:39 +0000

From this step-by-step tutorial, you will learn how to setup Lefthook git hooks manager, Crystalball test selection library, and also how to automatically install missing gems and migrate your database when you're switching to a feature branch and back to the master.

Every project should have a CI installed. But CI builds sometimes can queue up, you need to wait for the notification. And anyway, is there a method to reduce the time for "implement-test-fix" cycle, save keystrokes and mouse clicks, but don't let broken code to get into the project repository? Yes, it exists long ago and is well-known: git hooks. Plain executable scripts in your local .git/hooks/ directory. But it is so bothersome to set them up, to update them. and to sync with your collaborators because you can't commit them to the repository itself.

Setting up Lefthook

Lefthook is our own git hook manager written in Go. Single dependency-free binary. Fast, reliable, feature-rich, language-agnostic.

Arkweid / lefthook

Fast and powerful Git hooks manager for any type of projects.

Lefthook

The fastest polyglot Git hooks manager out there

Fast and powerful Git hooks manager for Node.js, Ruby or any other type of projects.

Fast. It is written in Go. Can run commands in parallel.
Powerful. With a few lines in the config you can check only the changed files on pre-push hook.
Simple. It is single dependency-free binary which can work in any environment.

📖 Read the introduction post

# On `git push` lefthook will run spelling and links check for all of the changed files
pre-push
  parallel: true
  commands
    spelling
      files: git diff --name-only HEAD @{push}
      glob: "*.md"
      run: npx yaspeller {files}
    check-links:
      files: git diff --name-only HEAD @{push}
      glob: "*.md"
      run: npx markdown-link-check {files}

Usage

Choose your environment:

Then you can find all Lefthook features in the full…

View on GitHub

Install it with gem, npm, or from source or your OS package manager.

Define your hooks in config file lefthook.yml

pre-push:
  parallel: true
  commands:
    rubocop:
      tags: backend
      run: bundle exec rubocop
    rspec:
      tags: rspec backend
      run: bundle exec rspec --fail-fast

And run lefthook install.

And voila, starting from now rspec and rubocop will be run in parallel on every git push and push will be aborted if they would find any issues.

But why you've chosen pre-push over pre-commit?

First of all, sometimes, during refactorings, you want to make a lot of small commits locally. Most of them won't pass linters. Why execute all this machinery if you know that it won't pass? Afterward, you will squash and reorder them with git rebase --interactive and push clean code to the repo.

More importantly, some things are not executing fast. Wait a minute on every git commit is meh. You lose speed. So why not to move long operations to the much more rare push event?

However, running whole test suite may take a too long time (and we have CI exactly for that anyway), so we need some method to run only specs that we probably might break by our changes.

Setting up Crystalball

Crystalball is a Ruby library by Toptal which implements Regression Test Selection mechanism. Its main purpose is to select a minimal subset of your test suite, which should be run to ensure your changes didn't break anything. It is a tricky problem in Ruby applications in general and especially in Rails applications because of Ruby on Rails constant autoloading mechanism.

Crystalball solves this problem by tracing code execution and tracking dependencies between files when your test suite is running. Using this profiling data, it could tell which files affect which. Take a look at slides about Crystalball from the talk at RubyKaigi 2019.

toptal / crystalball

Regression Test Selection library for your RSpec test suite

Crystalball

Crystalball is a Ruby library which implements Regression Test Selection mechanism originally published by Aaron Patterson Its main purpose is to select a minimal subset of your test suite which should be run to ensure your changes didn't break anything.

Installation

Add this line to your application's Gemfile:

group :test do
  gem 'crystalball'
end

And then execute:

$ bundle

Or install it yourself as:

$ gem install crystalball

Usage

Please see our official documentation.

Versioning

We use semantic versioning for our releases.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/toptal/crystalball This project is intended…

View on GitHub

Install it

   # Gemfile
   gem "crystalball"

Configure for our pre-push case (by default crystalball is configured to be used in pre-commit hooks)

   # config/crystalball.yml
   ---
   map_expiration_period: 604800 # 1 week
   diff_from: origin/master

Setup your test suite to collect code coverage information:

   # spec/spec_helper.rb
   if ENV['CRYSTALBALL'] == 'true'
     require 'crystalball'
     require 'crystalball/rails'

     Crystalball::MapGenerator.start! do |config|
       config.register Crystalball::MapGenerator::CoverageStrategy.new
       config.register Crystalball::Rails::MapGenerator::I18nStrategy.new
       config.register Crystalball::MapGenerator::DescribedClassStrategy.new
     end
   end

Generate code execution maps:

   CRYSTALBALL=true bundle exec rspec

Replace RSpec with crystalball in lefthook.yml:

   -      run: bundle exec rspec --fail-fast
   +      run: bundle exec crystalball --fail-fast

And from now every push will be accelerated dramatically if your changes are small.

But crystalball needs up-to-date code execution maps to work correctly. Can we automate these maps refreshing, too? Sure, we can!

Keeping Crystalball up-to-date

For that sake git's post-checkout hook fits very well. We can run code with updating crystalball data logic. “Logic” implies complexity as there is no single command for that. To cover such cases, Lefthook allows having separate executable script files. We can put our logic to .lefthook/post-checkout/crystalball-update file, make it executable, and declare in lefthook configuration like this:

# lefthook.yml

post-checkout:
  scripts:
    crystalball-update:
      tags: rspec backend

And there, in crystalball-update script, we need a bit of magic.

First of all, we don't need to do anything when a developer uses git checkout -- path command to reject some changes from working tree. Because this is not switching between commits and repository is possibly "dirty." Yes, Git CLI sometimes can feel weird as checkout command is used for two different tasks.

Per git docs, git will always pass to post-checkout hook three arguments: previous HEAD commit identifier (SHA1 sum), current HEAD commit identifier and flag whether it was checkout between branches (1) or file checkout to the state of another commit (0). Lefthook will catch these arguments and will carefully pass them to every managed script.

#!/usr/bin/env ruby

_prev_head, _curr_head, branch_change, * = ARGV

exit if branch_change == "0" # Don't run on file checkouts

Next, we want to update crystalball profiling data only on the master branch as recommended in the Crystalball docs. To do so, we need to ask git what branch we've checked out:

# Rails.root if we look from .lefthook/post-checkout dir
app_dir = File.expand_path("../..", __dir__)
ENV["BUNDLE_GEMFILE"] ||= File.join(app_dir, "Gemfile")
require "bundler/setup"

require "git"
exit unless Git.open(app_dir).current_branch == "master"

git gem is a dependency of Crystalball, so we don't have to install it.

And finally we need to do most heavy part: ask Crystalball, “Are your profiling data up-to-date?”

require "crystalball"
config = Crystalball::RSpec::Runner.config
prediction_builder = Crystalball::RSpec::Runner.prediction_builder

exit if File.exists?(config["execution_map_path"]) && !prediction_builder.expired_map?

And if it is not fresh we need to run the whole test suite with special environment variable set:

puts "Crystalball Ruby code execution maps are out of date. Performing full test suite to update them…"

ENV["CRYSTALBALL"] = "true"
RSpec::Core::Runner.run([app_dir])

And we're done. Are we?

Automate other routine tasks

But running specs require that we have:

a. installed gems and
b. actual database state.

And in actively developing application gems are frequently updated, added, and removed, database schema sometimes can be changed several times a day in different branches. It is so typical to pull fresh master at morning and get updated gems and new database migrations. In that case, RSpec would fail, and Crystalball execution path maps won't be complete. So we need to ensure that our specs always can run beforehand.

Install missing gems on a `git checkout`

This task is quite simple and can be achieved by a simple bash script. Most of it will consist of checks to avoid calling bundler when it's not needed. Bundler is quite heavy as it runs by noticeable time.

Two first of these checks are same, but just rewritten to shell: is this branch checkout? Did we actually move between commits?

#!/bin/bash

BRANCH_CHANGE=$3
[[ $BRANCH_CHANGE -eq 0 ]] && exit

PREV_HEAD=$1
CURR_HEAD=$2
[ $PREV_HEAD == $CURR_HEAD ] && exit

Next one is more tricky:

# Don't run bundler if there were no changes in gems
git diff --quiet --exit-code $PREV_HEAD $CURR_HEAD -- Gemfile.lock && exit;

We're asking here, “Does a set of required gems changed between commits?” If Gemfile.lock was changed, we need to check do we have all of the gems installed by invoking bundler.

bundle check || bundle install

Again, if you have up-to-date gems (and in most checkouts, it will be so), only bundle check will be executed.

Automatically rollback and apply database migrations

Next task is much more interesting.

When we're switching from branch A to branch B, we need to ensure that database schema actually is compatible with our specs. To do so, we must rollback every migration that exists in branch A but not in branch B and then to apply every migration that exists in B and still isn't applied. Rollback part is required because migrations that remove or rename columns and tables are not backward compatible.

The problem here is that there is no pre-checkout hook in git, only post-checkout one. And after checkout, there are no more migration files left that existed only in the branch we're switched from. How to rollback them?

But this is git! The files are out there. Why not just take them from git itself?

To do so programmatically let's use gem git to access our git repository. Crystalball already uses it under the hood so there will be no new dependency, but it is a good idea to add it to the Gemfile explicitly.

Let's start from the check that we really have any migrations to run (either up or down):

require "git"

# Rails.root if we look from .lefthook/post-checkout dir
app_dir = File.expand_path("../..", __dir__)

git = Git.open(app_dir)

# Don't run if there were no database changes between revisions
diff = git.diff(prev_head, curr_head).path("db/migrate")
exit if diff.size.zero?

Then, to be able to use migrations, we need to load our rails application and connect to the database:

require File.expand_path("config/boot", app_dir)
require File.expand_path("config/application", app_dir)
require "rake"
Rails.application.load_tasks

Rake::Task["db:load_config"].invoke

Then we can take files and save them somewhere:

# migrations added in prev_head (deleted in curr_head) that we need to rollback
rollback_migration_files = diff.select { |file| file.type == "deleted" }

if rollback_migration_files.any?
  require "tmpdir"
  MigrationFilenameRegexp = ActiveRecord::Migration::MigrationFilenameRegexp
  versions = []

  Dir.mktmpdir do |directory|
    rollback_migration_files.each do |diff_file|
      filename = File.basename(diff_file.path)
      contents = git.gblob("#{prev_head}:#{diff_file.path}").contents
      File.write(File.join(directory, filename), contents)
      version = filename.scan(MigrationFilenameRegexp).first&.first
      versions.push(version) if version
    end

And now, when we have files for migrations that need to be rolled back we can rollback them:

begin
  old_migration_paths = ActiveRecord::Migrator.migrations_paths
  ActiveRecord::Migrator.migrations_paths.push(directory)

  versions.sort.reverse_each do |version|
    ENV["VERSION"] = version
    Rake::Task["db:migrate:down"].execute
  end
ensure
  ENV.delete("VERSION")
  ActiveRecord::Migrator.migrations_paths = old_migration_paths
end

Here we're adding our temporary directory with another branch migrations to the ActiveRecord's migrations_paths. This setting is available since Rails 5, but not widely known. Now ActiveRecord can see our ghost migration files, and we can simply invoke rake db:migrate:down VERSION=number for every migration to rollback it.

And after that we can just migrate not yet applied migrations:

Rake::Task["db:migrate"].invoke

And that's it!

Composing it together

Now we only need to invoke these scripts in the right order: install gems, run migrations and run specs (if required). To do so we need to name files in alphabetical order, place them in .lefthook/post-checkout directory and declare them in lefthook.yml:

post-checkout:
  piped: true
  scripts:
    01-bundle-checkinstall:
      tags: backend
    02-db-migrate:
      tags: backend
    03-crystalball-update:
      tags: rspec backend

The piped option will abort the rest of the commands if the preceding command fails. For example, if you forget to launch a database server, the second step will fail, and lefthook will skip the third step altogether.

Frontend devs, not interested in running RSpec can exclude rspec tag in their lefthook.local.yml, and they will only get always installed gems and migrated database. Automagically.

Any gotchas?

From now on you always have to write reversible migrations. Look at example application to learn how to do it.

Conclusion

Now we not only have checks that will prevent us from pushing broken code to the repository (and will check this really fast) but also, as a side-effect, we always will have installed gems, and our database will be in migrated state. You will forget about bundle install (unless you are the one who update gems). And no more “Okay, now I need to rollback X and Y first, and checkout back to master only then.”

Experiment with example application published on GitHub: https://github.com/Envek/lefthook-crystalball-example

Envek / lefthook-crystalball-example

Learn how to make git hooks to do most routine tasks for you: install gems, migrate the database, run tests, and linters.

Example application for Lefthook + Crystalball guide

Please read “Lefthook, Crystalball, and git magic for smooth development experience” blog post first.

Installation

Ensure that you have Ruby 2.6.3 and SQLite installed (with development libraries)

Fork and clone this repository locally:

git clone https://github.com/[YOUR-NAME]/lefthook-crystalball-example

Go to directory with it:
```
cd lefthook-crystalball-example
```
Install lefthook from Rubygems explicitly:
```
gem install lefthook
```
Install missing hooks to your local .git/hooks/ directory:
```
lefthook install
```
Every developer need to do this only once. After that Lefthook will sync hooks after changes in lefthook.yml automatically.

Install other dependencies:

bundle install
yarn install --check-files

Prepare database
```
rails db:setup
```
Generate code execution maps for Crystalball:
```
CRYSTALBALL=true bundle exec rspec
```
This need to be done only once too. post-checkout hook will run this automatically once a week.

Usage

Checkout to feature/multiaccount branch, see how migrations are applied:
```
git checkout feature/multiaccount
```
Checkout to feature/variations branch, see how new gems…

View on GitHub

Happy coding!

The silence of the Ruby exceptions: a Rails/PostgreSQL database transaction thriller

Andrey Novikov — Tue, 24 Jul 2018 08:46:23 +0000

Here comes a tale on why you should never silence errors inside database transactions. Learn how to use transactions properly and what to do when using them is not an option. Spoiler: use PostgreSQL advisory locks!

I was working on a project where users could import a bunch of heavy entities (let's call them products) from an external service into our application. Each product brought in even more associated data from external APIs. It is not uncommon for a user to import hundreds of products, and each product has dependencies that need to be fetched too, so you can imagine that the whole process takes a while (30 to 60 seconds for each product). As a user may get tired of waiting and hit the "Cancel" button at any moment, the application should still be usable with only those records that made it.

Here's how our "interruptable import” was implemented: at the start of an import a temporary record in a separate database table was created for each enqueued product. Then for each of those records, a background job was spawned that pulled in all external information, persisted it in the right places (by creating associations, if necessary), and finally deleted a temporary record. If by the time the background job started the record was not found (when a user cancels the import, all temporary records are deleted)—the job just did nothing and exited silently.

Interrupted import or not—the absence of temporary records meant we were done.

Our design seemed simple and reliable, but it did not always work exactly as planned. A common bug description stated: "After canceling an import, a user is presented with a list of imported records. However, the next time the page is refreshed, the list has more records than shown initially".

The reason for that was clear: as background jobs took up to a minute to finish, even a canceled import had an "afterglow".

Nothing wrong with the design, but it led to confusing user experience, so we needed to address it in two possible ways: either somehow identify and cancel jobs already in progress, or wait for the last imports to finish before confirming that the whole process is in fact "canceled". I chose the latter.

Transaction locks to the rescue!

For anyone dealing with relational databases often, the answer is clear: use transactions! According to a description in Rails docs, they are "protective blocks where SQL statements are only permanent if they can all succeed as one atomic action". As documentation states, "you should use transaction blocks whenever you have a number of statements that must be executed together or not at all."

It is good to keep in mind that in most RDBMS records being updated inside a transaction will be locked and unmodifiable by other processes until the transaction is finished. The same is true when you fetch the records with SELECT FOR UPDATE.

Exactly our case! I used a single database transaction to wrap complex import tasks for individual products and to lock task records from being changed or deleted:

ActiveRecord::Base.transaction do
  task = Import::Task.lock.find_by(id: id) # SELECT … FOR UPDATE means “I want to change it later, keep it for me”
  return unless task # it was already deleted so we're free
  # do some heavy lifting
  task.destroy
end

Now whenever the whole import is canceled, this code should wait for all individual tasks to finish:

user.import_tasks.delete_all # waits while all imports will be finished

Simple and elegant! I've run everything locally and in the staging environment, found no problems and deployed to production.

Not so fast...

Satisfied with my work, I was surprised to wake up to the myriad of log errors and complaints from colleagues. Many products weren't being imported at all. In some cases, only a single product would appear after the import is canceled.

Error messages did not make much sense: PG::InFailedSqlTransaction from code that executed innocent SELECTs. What's going on?

After an entire day of debugging, I've identified three root causes of problems:

Concurrent inserts of conflicting records into a database.
Automatic transaction rollback in PostgreSQL.
Silenced Ruby exceptions.

First problem: Concurrent inserts of conflicting records
As each product's import job takes up to a minute, and there are multiple jobs per user, we run them in parallel to save time. Dependent records that are created when those jobs run can overlap, meaning different products can have the same dependency, first created, then reused.

There are checks for these cases on the application level, but now that we use transactions they are rendered useless: if transaction A has created a dependent record, but has not committed yet, transaction B won't be aware of it and will try to create a duplicate.

Second problem: Transaction rollback on a DB-level error
We can prevent creation of the duplicate records at the database level, with the following DDL:

ALTER TABLE deps ADD UNIQUE (user_id, chars);

Now if transaction A had inserted a new record and the transaction B, running in parallel, tries to insert the same record—we will get an error:

BEGIN;
INSERT INTO deps (user_id, chars) VALUES (1, '{"same": "value"}');
-- Now it will block until first transaction will be finished
ERROR:  duplicate key value violates unique constraint "deps_user_id_chars_key"
DETAIL:  Key (user_id, chars)=(1, {"same": "value"}) already exists.
-- And will throw an error when first transaction have commited and it is become clear that we have a conflict

One point that is commonly overlooked is that the transaction B, upon encountering an error, will roll-back entirely and the sequence of steps leading up to the error will be wasted. An invalid transaction, however, will still be open until all the code inside runs its course. Every attempt to use records in question will result in another error:

SELECT * FROM products;
ERROR:  current transaction is aborted, commands ignored until end of transaction block

It goes without saying that no information (doesn't matter if you dealt with it before or after the error) will be persisted at all:

COMMIT;  -- If we even try to save what we've done
ROLLBACK -- RDBMS will reject our attempt

Third problem: Silenced exceptions
At that point, it became clear that adding such a common and robust concept as database transaction to the application logic effectively broke it. I had no choice but to dig into other people's code. This is when I started seeing snippets such as this one:

def process_stuff(data)
  # Magic, lots of magic
rescue StandardError
  nil # Happy debugging
end

What the author is trying to say here is "We tried something that failed, but we don't care, so we keep doing stuff". While the reasons behind that approach may be justified (there are some things you can’t validate for on the application level), it makes transaction-based logic unimplementable: a silenced exception will never bubble up to a transaction block and will not trigger a rollback (because ActiveRecord catches all exceptions, rolls back a transaction and re-raises them).

A perfect storm

Here's how those three factors came together to create a perfect ~~storm~~ bug:

Transaction tries to insert a conflicting row into a database and encounters a "duplicate key" error from Postgres. This error, however, never causes a transaction rollback, as it is silenced in Ruby.
Transaction becomes invalid but keeps running. When the application tries to access the DB for whatever reason (even with an innocent SELECT to list products), it triggers "current transaction is aborted" error. But that exception may be silenced too...
You get the idea. While things keep breaking inside the application, no one will know about it until the error bubbles up to the first piece of code that does not use a rescue block. As it might be quite far away from the code that failed originally, debugging becomes a nightmare.

PostgreSQL's built-in alternative to transaction-level locks

Hunting for rescues in the application code and rewriting the whole import logic was not an option, so I needed a quick win and Postgres gave me one! It has a built-in locking tool that can be used as an alternative for locking records in transactions. Meet session-level advisory locks. This is how I used them:

First, I removed all transaction wrappers that I added in the first place. Doing API interactions or any other non-database side-effects from a code within a transaction is a bad idea anyway because even if database changes are successfully rolled back, side-effects will already be applied and that may leave your application in an undesired state. Take a look at the isolator gem that will help you make sure your side-effects are well isolated.

Then I associated every import task with a shared lock based on some unique key (created from a user ID and a hash of operation name, for instance):

SELECT pg_advisory_lock_shared(42, user.id);

Shared locks may be obtained simultaneously by as many sessions as you want.

"Cancel import" operation deletes all pending tasks from the database and tries to take an exclusive lock for the same key. This will have to wait till all shared locks are released:

SELECT pg_advisory_lock(42, user.id)

And that is it!

Now canceling an import will automatically wait for all running import tasks to finish.
Moreover, we can use a little hack to set a timeout for our exclusive lock, because we don't want to wait for too long, blocking a web server thread, if some task hangs for whatever reason:

transaction do
  execute("SET LOCAL lock_timeout = '30s'")
  execute("SELECT pg_advisory_lock(42, user.id)")
rescue ActiveRecord::LockWaitTimeout
  nil # we are tired of waiting and give up (transaction is already rolled back at this point)
end

Rescuing exception outside a transaction block is safe because ActiveRecord would already roll back.

What to do with concurrent inserts of the same record?

Unfortunately, I am not aware of a solution that can be used together with concurrent inserts. Here are all the approaches that I know about, but all of them will lock concurrent inserts until the first transaction commits:

Use INSERT … ON CONFLICT UPDATE (available since PostgreSQL 9.5) to wait until the first transaction commits and returns a record inserted by it.
Lock some common record in the database before running validations for a new record: this will wait until record inserted by the first transaction will become visible and validations will work as expected.
Take some common advisory lock before running validations for a new record.

If you’re not afraid of dealing with Postgres errors, you can also do this:

def import_all_the_things
  # Start transaction here and at some point do
  Dep.create(user_id, chars)
rescue ActiveRecord::RecordNotUnique
  retry
end

And make sure this code is not wrapped inside a transaction block.

Why will it lock?

UNIQUE and EXCLUDE constraints would prevent potential conflicts to be written at the same time. E.g., if you have unique integer column and one transaction inserts 5, that will prevent other transactions from inserting 5 concurrently, but will not interfere with a transaction inserting 6. Since the least strict transaction isolation level supported by PostgreSQL is READ COMMITED, the second transaction is not allowed to see results of the first transaction that are not yet committed. INSERT of conflicting value must wait until the first transaction either commits (then it will fail) or rollbacks (then it will succeed). Read more in this article from the author of EXCLUDE constraints.

Preventing future havoc

If you know that some code can not be used inside a transaction for reasons described above, make sure it can never be wrapped in one. Here's how you can detect open transactions and fail fast:

# Here's how you define it
module NoTransactionAllowed
  class InTransactionError < RuntimeError; end

  def call(*)
    return super unless in_transaction?
    raise InTransactionError,
          "#{self.class.name} doesn't work reliably within a DB transaction"
  end

  def in_transaction?
    connection = ApplicationRecord.connection
    # service transactions (tests and database_cleaner) are not joinable
    connection.transaction_open? && connection.current_transaction.joinable?
  end
end

# And here's how you use it
class Deps::Import < BaseService
  prepend NoTransactionAllowed

  def call
    do_import
  rescue ActiveRecord::RecordNotUnique
    retry
  end
end

Now, whenever anyone tries to wrap a risky service in a transaction, they will get an exception (unless they’ve silenced it, of course).

Wrapping up

One lesson to be taken from this tale: be careful with exceptions. Only catch those you really know how to deal with, and nothing more. Never silence exceptions unless you are absolutely sure. The sooner error will be noticed—the easier it will be to debug.
And don't overuse database transactions!

Our gems isolator and after_commit_everywhere will make your transactions completely fool-proof.

palkan / isolator

Detect non-atomic interactions within DB transactions

Isolator

Detect non-atomic interactions within DB transactions.

Examples:

# HTTP calls within transaction
User.transaction do
  user = User.new(user_params)
  user.save
  # HTTP API call
  PaymentsService.charge!(user)
end
#=> raises Isolator::HTTPError
# background job
User.transaction do
  user.update!(confirmed_at: Time.now)
  UserMailer.successful_confirmation(user).deliver_later
end

#=> raises Isolator::BackgroundJobError

Of course, Isolator can detect implicit transactions too. Consider this pretty common bad practice–enqueueing background job from after_create callback:

class Comment < ApplicationRecord
  # the good way is to use after_create_commit
  # (or not use callbacks at all)
  after_create :notify_author

  private

  def notify_author
    CommentMailer.comment_created(self).deliver_later
  end
end

Comment.create(text: "Mars is watching you!")
#=> raises Isolator::BackgroundJobError

Isolator is supposed to be used in tests and on staging.

Installation

Add this line to your application's Gemfile:

# We suppose that Isolator is used in development and test
# environments.
group :development, :test

…

View on GitHub

Envek / after_commit_everywhere

Use ActiveRecord transactional callbacks outside of models, literally everywhere in your application.

`after_commit` everywhere

Allows to use ActiveRecord transactional callbacks outside of ActiveRecord models, literally everywhere in your application.

Inspired by these articles:

Installation

Add this line to your application's Gemfile:

gem 'after_commit_everywhere'

And then execute:

$ bundle

Or install it yourself as:

$ gem install after_commit_everywhere

Usage

Recommended usage is to include it to your base service class or anything:

class ServiceObjectBtw
  include AfterCommitEverywhere
  def call
    ActiveRecord::Base.transaction do
      after_commit { puts "We're all done!" }
    end
  end
end

Or just extend it whenever you need it:

extend AfterCommitEverywhere
ActiveRecord::Base.transaction do
  after_commit { puts "We're all done!" }
end

That's it!

But the main benefit is that it works with nested transaction blocks (may be even spread across many files in your codebase):

include AfterCommitEverywhere
ActiveRecord::Base.transaction do
  puts "We're in transaction now"
  ActiveRecord

…

View on GitHub

DEV Community: Andrey Novikov

Fullstaq Ruby: First impressions, and how to migrate your Docker/Kubernetes Ruby apps today

What is Fullstaq Ruby?

Why do we need it?

Now that's the difference!

Alternatives?

Cables vs. malloc_trim, or yet another Ruby memory usage benchmark

Vladimir Dementyev ・ Mar 19 '19 ・ 6 min read

How to install?

Recap

Reporting non-nullable violations in graphql-ruby properly

Lefthook, Crystalball, and git magic for smooth development experience

Setting up Lefthook

Arkweid / lefthook

Fast and powerful Git hooks manager for any type of projects.

Lefthook

Usage

Setting up Crystalball

toptal / crystalball

Regression Test Selection library for your RSpec test suite

Crystalball

Installation

Usage

Versioning

Development

Contributing

Keeping Crystalball up-to-date

Automate other routine tasks

Install missing gems on a git checkout

Automatically rollback and apply database migrations

Composing it together

Any gotchas?

Conclusion

Envek / lefthook-crystalball-example

Learn how to make git hooks to do most routine tasks for you: install gems, migrate the database, run tests, and linters.

Example application for Lefthook + Crystalball guide

Installation

Usage

The silence of the Ruby exceptions: a Rails/PostgreSQL database transaction thriller

Transaction locks to the rescue!

Not so fast...

A perfect storm

PostgreSQL's built-in alternative to transaction-level locks

What to do with concurrent inserts of the same record?

Preventing future havoc

Wrapping up

palkan / isolator

Detect non-atomic interactions within DB transactions

Isolator

Installation

Envek / after_commit_everywhere

Use ActiveRecord transactional callbacks outside of models, literally everywhere in your application.

after_commit everywhere

Installation

Usage

Further reading

Install missing gems on a `git checkout`

`after_commit` everywhere