SolidQueue now runs with multiple threads

#rails #ruby

Recently, I’ve personally been using SolidQueue more and more as a background worker. Around that time, SolidQueue v1.3.0 was released, and it included a feature that I found particularly interesting, so I’d like to introduce it here.

Background

SolidQueue defines multiple roles:

Supervisor
Dispatcher
Scheduler
Worker

In the early days of SolidQueue, there was an async mode where each role could run as a thread within a single process.

However, in order to simplify the implementation, async mode was removed starting with v0.4.0, and SolidQueue began assigning one or more processes to each role.

SolidQueue Used More Memory Than Sidekiq

Because each role runs in its own process, even the minimum SolidQueue setup ends up spawning 3–4 processes (the Scheduler can be omitted if recurring jobs are not needed). Sidekiq, on the other hand, runs in a single process, so a server that works fine with Sidekiq can easily run into memory constraints when using SolidQueue.

I personally ran workers on Heroku using a 512 MB dyno, but when migrating from Sidekiq to SolidQueue, memory was no longer sufficient and I had no choice but to upgrade to a 1 GB dyno.

Rails is very convenient for individual developers because it can be easily deployed and operated on inexpensive VPSs. However, when using SolidQueue, a relatively larger amount of memory (around 2 GB or more) was required.

The Return of Async Mode

Due to this memory usage issue, some people started to want async mode to come back. Simply saying “we want to reduce memory usage” wasn’t enough to move things forward, but the discussion gained momentum when it became clear that people on JRuby and Windows—who couldn’t use SolidQueue because fork is unavailable—also wanted to be able to use it.

Eventually, the PR that brings back async mode was merged. This change is included in the recently released v1.3.0.

You can enable it by starting SolidQueue with bin/jobs --async or by setting SOLID_QUEUE_SUPERVISOR_MODE=async, which makes SolidQueue use threads instead of processes.

The README says the following, so using processes (fork) is generally more stable. Still, async mode seems quite convenient for limited environments such as personal projects.