DEV Community

Kaziu
Kaziu

Posted on

ActiveJob Too? Sure, Let's Explain Jobs, Queues, and Adapters with a Restaurant Again

Yeah, I know, I know. But let me compare it to a restaurant anyway.


The cast

First, a mapping table. I'll stick with this analogy for the whole post.

Rails term In restaurant terms
Web request An order from a customer who walked in
Controller Cashier
Job Order ticket
perform_later Pinning the order ticket to the board
Queue The board where order tickets are pinned
Worker Cook
Adapter The type of board

Why do things "later" at all?

Say we have a feature that sends a welcome email when a user signs up.

class UsersController < ApplicationController
  def create
    user = User.create!(user_params)
    WelcomeMailer.with(user: user).welcome.deliver_now # ← you get stuck waiting a few seconds here
    render json: user
  end
end
Enter fullscreen mode Exit fullscreen mode

This is like the cashier taking an order, walking into the kitchen to cook it themselves, and not coming back to the register until it's done. Everyone in line behind has to wait the whole time.

For slow work like sending emails, exporting CSVs, or calling external APIs, we want to say "we'll do that later" and get back to the customer right away. That's where jobs come in.


Job = order ticket

A job is an order ticket that says "please take care of this."

class WelcomeMailJob < ApplicationJob
  def perform(user_id)
    user = User.find(user_id)
    WelcomeMailer.with(user: user).welcome.deliver_now
  end
end
Enter fullscreen mode Exit fullscreen mode

The cashier (controller) just writes the ticket and pins it to the board.

def create
  user = User.create!(user_params)
  WelcomeMailJob.perform_later(user.id) # ← just pins the ticket. Done in an instant
  render json: user
end
Enter fullscreen mode Exit fullscreen mode

perform_later literally means "perform this later." The cashier can go right back to the next customer.

Write the "table number" on the ticket, not the "dish"

Notice we pass user.id, not user itself, as the argument.

Nobody tapes the actual dish to an order ticket. You write "Table 3, hamburger steak," and the cook checks the latest info when it's time to cook. Jobs work the same way: the basic rule is pass just the ID, and re-fetch from the DB when the job runs. The data might have changed between being queued and actually running.


Queue = order ticket board

A queue is a board that keeps order tickets lined up in the order they arrived.

 Cashier                Order ticket board (queue)         Cook (worker)
┌──────┐   pin    ┌──────────────────────────┐  take   ┌──────┐
│      │ ───────▶ │ [T3] [T2] [T1]           │ ──────▶ │ Cook │
└──────┘          └──────────────────────────┘         └──────┘
Enter fullscreen mode Exit fullscreen mode
  • The cashier only pins, the cook only takes and cooks. Neither waits on the other
  • The cook is always on standby in the kitchen and grabs a ticket as soon as it's pinned
  • If orders flood in, the tickets just line up on the board. The kitchen doesn't get overwhelmed

"Why not just hire a cook for every order?"

You might think: skip the board and hire a new cook for each order to start cooking immediately. In Rails terms, that's "spin up a new process for every job."

That doesn't work, for these reasons:

  • Hiring is expensive: a Rails process takes several seconds to boot and hundreds of MB of memory. The training cost every time is way too high
  • Too many of them: if 1,000 orders come in per second, 1,000 people pile into the kitchen at once. Memory and DB connections (the size of the kitchen and the number of burners) hit their limits before the CPU does
  • Orders vanish: if a cook collapses mid-order, that order disappears without anyone knowing
  • No do-overs: if something fails, nobody remakes it

With a board and resident cooks, you can keep the headcount fixed, never lose an order, and reliably get everything done. A queue isn't "a mechanism for making people wait"; it's a mechanism for exactly that.


Adapter = type of board

There are all kinds of order ticket boards: a whiteboard, paper slips, an order system connected to HQ...

ActiveJob is built so that you write order tickets the same way no matter which board you use. The "plug converter" that absorbs the differences between boards is the adapter.

# You write order tickets the same way for every board
WelcomeMailJob.perform_later(user.id)

# Which board to use is switched with one line of config
config.active_job.queue_adapter = :async
Enter fullscreen mode Exit fullscreen mode

Let's look at the common boards.

:async ─ the staff just keep it in their heads

┌──────────── The restaurant (one Rails process) ────────────┐
│  Cashier ──▶ notes in the staff's head ──▶ in-house cook    │
└─────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

This is the Rails default. The queue and the cooks are all inside the Rails process.

  • It works without installing anything extra, so it's handy for learning and development
  • But the orders only exist in the staff's heads. Close the restaurant (restart the process, deploy) and every order that hasn't been cooked yet is completely forgotten

:inline ─ the cashier cooks it on the spot

No board; the cashier cooks it themselves right then and there. Even if you write perform_later, it doesn't run asynchronously.

Handy during development when you want to quickly check "does this job actually work?"

:test ─ the training board

It doesn't actually cook anything; it only records "was the order ticket pinned correctly?" You use it when testing with RSpec.

expect {
  post '/users', params: { ... }
}.to have_enqueued_job(WelcomeMailJob)
Enter fullscreen mode Exit fullscreen mode

:solid_queue / :sidekiq / :sqs etc. ─ a board outside the restaurant

For production. Order tickets are recorded in an order system outside the restaurant (a DB, Redis, Amazon SQS, etc.).

┌── Restaurant (Rails process) ──┐      ┌── Order system outside ──┐
│  Cashier                        │ ───▶ │  Order ticket board       │
└─────────────────────────────────┘      └───────────────────────────┘
   ↑ No matter how many times it          ↑ The orders stay here
     closes and reopens
Enter fullscreen mode Exit fullscreen mode
  • Even if the restaurant closes (Rails restarts or deploys), the orders stay
  • You can put cooks in a different kitchen (a different server) and add as many as you need
  • Failed orders can be re-pinned with "try again in 5 minutes"

That said, how much survives if the board itself breaks depends on the board.

  • Redis also keeps things in memory. If you haven't set up persistence (snapshots or an append-only log), a Redis restart wipes it out
  • DB and SQS store data on disk or across multiple locations, so they're less likely to lose things

The real difference from :async isn't "memory vs. disk"; it's whether the board closes along with the restaurant (Rails).

Adapter Where the board lives Lost on Rails restart? Lost when the board itself restarts? Main use
:async In the staff's heads (Rails memory) Lost ─ Learning / development
:inline None (cooked on the spot) ─ ─ Development / debugging
:test Training records ─ ─ Testing
:sidekiq Redis (memory on another server) Not lost Depends on persistence settings Production
:solid_queue DB (disk) Not lost Almost never lost Production
:sqs family Amazon SQS Not lost Stored across multiple locations, hard to lose Production

Things to watch out for in production

Once you use a board outside the restaurant, problems show up that you never saw with :async or :inline.

1. The same order ticket can arrive twice

With distributed queues like SQS, the same job can occasionally run twice. Getting two hamburger steaks would be a problem, right?

Build your jobs so that "running them any number of times gives the same result" (idempotency). For example, "if the sent flag is already set, do nothing."

2. Don't pin the ticket before the bill is settled

ActiveRecord::Base.transaction do
  user = User.create!(user_params)
  WelcomeMailJob.perform_later(user.id) # ← dangerous
  # ...what if an exception happens after this and it rolls back?
end
Enter fullscreen mode Exit fullscreen mode

If you call perform_later inside a transaction, these things can happen:

  • The cook sees the ticket first and heads to the kitchen, but the order isn't confirmed yet (it's before the commit, so User.find can't find it)
  • The order was canceled (rolled back), but the cook already started cooking

Pin the ticket after the bill is settled (after the transaction commits). Using after_commit is the standard approach.

Also, Rails 7.2 and later have a mechanism (enqueue_after_transaction_commit) that delays jobs enqueued during a transaction until after the commit.

Top comments (0)