Building a Rails Notification System That Survives Scale

Every Rails app grows a notification system eventually.
It usually starts like this:

class Comment < ApplicationRecord
  after_create :notify_watchers

  private

  def notify_watchers
    post.watchers.each do |user|
      NotificationMailer.new_comment(user, self).deliver_later
      user.notifications.create!(message: "New comment on #{post.title}")
    end
  end
end

This works beautifully — until a post has 40,000 watchers and someone comments. Now you have 40,000 inserts and 40,000 mailer jobs inside a request cycle, a timeout, a half-written notification table, and an on-call ping.

The fix isn't a gem. It's four architectural decisions. Let's walk through them.

1. Separate the event from the delivery

The single most important split: what happened is not who gets told and is not how they get told.

Event (comment.created)
  └─> Fan-out (who should know?)
        └─> Notification records (in-app inbox)
              └─> Deliveries (email, push, Slack, SMS)

Three separate concerns, three separate failure domains. If Firebase is down, your in-app inbox still works. If fan-out is slow, the comment still saved.

Model it explicitly:

# events: what happened, once
create_table :events do |t|
  t.string  :kind, null: false            # "comment.created"
  t.jsonb   :payload, null: false, default: {}
  t.references :actor, polymorphic: true
  t.timestamps
end

# notifications: one row per recipient
create_table :notifications do |t|
  t.references :recipient, null: false
  t.references :event, null: false
  t.string   :kind, null: false
  t.jsonb    :payload, null: false, default: {}
  t.string   :idempotency_key, null: false
  t.datetime :read_at
  t.timestamps
end
add_index :notifications, :idempotency_key, unique: true

That unique index is not decoration — it's what makes the whole pipeline safely retryable. More on that in a second.

2. Never fan out in the request

The web request creates the event and enqueues exactly one job. That's it.

class Comment < ApplicationRecord
  after_commit :publish_event, on: :create

  private

  def publish_event
    event = Event.create!(kind: "comment.created", actor: author,
                          payload: { comment_id: id, post_id: post_id })
    Notifications::FanOutJob.perform_later(event.id)
  end
end

Use after_commit, not after_create. Enqueueing inside a transaction is the classic race: the job starts, queries for a record the DB hasn't committed yet, and dies with RecordNotFound.

The fan-out job then resolves the audience and chunks it:

class Notifications::FanOutJob < ApplicationJob
  queue_as :notifications
  BATCH = 1_000

  def perform(event_id)
    event = Event.find(event_id)

    Notifications::Audience.for(event).find_in_batches(batch_size: BATCH) do |users|
      Notifications::WriteBatchJob.perform_later(event_id, users.map(&:id))
    end
  end
end

One long job becomes N short jobs. Short jobs retry cheaply, spread across workers, and don't hold a database connection for four minutes.

3. Bulk insert, don't loop

create! in a loop is 40,000 round trips. insert_all is 40 round trips at batch size 1,000.

class Notifications::WriteBatchJob < ApplicationJob
  def perform(event_id, recipient_ids)
    event = Event.find(event_id)
    now = Time.current

    rows = recipient_ids.map do |id|
      {
        recipient_id: id,
        event_id: event.id,
        kind: event.kind,
        payload: event.payload,
        idempotency_key: "#{event.id}:#{id}",
        created_at: now,
        updated_at: now
      }
    end

    Notification.insert_all(rows, unique_by: :index_notifications_on_idempotency_key)

    Notifications::DeliverBatchJob.perform_later(event_id, recipient_ids)
  end
end

unique_by turns this into an upsert-style no-op on conflict. Now if Sidekiq retries this job after a network blip, you don't double-notify anyone. Idempotency is what lets you retry aggressively, and retrying aggressively is what makes the system reliable.

You skip callbacks and validations with insert_all — that's the trade. Notifications are derived data with a fixed shape, so it's a trade worth making. Just keep the shape enforced in one place (the row builder above).

4. Make channels pluggable and preference-aware

Delivery is where the fan-out multiplies again: one notification, three channels. Keep each channel a small object with the same interface.

module Notifications
  module Channels
    class Email
      def self.deliver(notification)
        NotificationMailer.with(notification: notification).notify.deliver_now
      end
    end

    class Push
      def self.deliver(notification)
        PushClient.send(token: notification.recipient.device_token,
                        title: notification.title, body: notification.body)
      end
    end
  end
end

REGISTRY = { email: Channels::Email, push: Channels::Push, slack: Channels::Slack }

Then resolve preferences once, in bulk, instead of per-user:

class Notifications::DeliverBatchJob < ApplicationJob
  def perform(event_id, recipient_ids)
    notifications = Notification.where(event_id: event_id, recipient_id: recipient_ids)
                                .includes(:recipient)

    args = notifications.flat_map do |n|
      Notifications::Preferences.channels_for(n).map { |ch| [n.id, ch.to_s] }
    end

    Notifications::DeliverJob.perform_all_later(args.map { |a| Notifications::DeliverJob.new(*a) })
  end
end

perform_all_later (Rails 7.1+) pushes the whole batch to Redis in one call instead of one round trip per job. On Sidekiq you can also reach for Sidekiq::Client.push_bulk.

Each leaf job handles exactly one delivery, so failures are isolated:

class Notifications::DeliverJob < ApplicationJob
  retry_on Net::OpenTimeout, Errno::ECONNREFUSED,
           wait: :polynomially_longer, attempts: 5
  discard_on ActiveJob::DeserializationError

  def perform(notification_id, channel)
    notification = Notification.find(notification_id)
    Notifications::REGISTRY.fetch(channel.to_sym).deliver(notification)
  end
end

Give email, push, and in-app their own queues with their own concurrency. A backed-up SMS provider should never starve your push notifications.

The unread badge problem

Here's the query that quietly kills you at scale — it runs on every single page load:

current_user.notifications.where(read_at: nil).count

Two fixes, use both:

A partial index, so Postgres only indexes rows you actually query:

add_index :notifications, [:recipient_id, :created_at],
          where: "read_at IS NULL",
          name: "index_unread_notifications"

A read watermark, so "mark all as read" is one write instead of 10,000:

# users.notifications_read_at
def mark_all_read!
  update_column(:notifications_read_at, Time.current)
end

def unread_count
  notifications.where(read_at: nil)
               .where("created_at > ?", notifications_read_at || Time.at(0))
               .limit(100).count
end

The limit(100) is deliberate. Nobody needs to know they have 8,412 unread notifications — "99+" is the same information at a fraction of the cost.

What to watch in production

Four numbers tell you whether this is healthy:

• Fan-out lag — event created → last notification row written
• Delivery lag — notification created → provider accepted
• Queue depth per channel — the first thing to spike when a provider degrades
• Delivery failure rate by channel — a broken push cert looks exactly like a quiet week unless you're graphing this

Emit them as counters and histograms and alert on the p95, not the average. Notification systems fail at the tail.

The short version

• Write one event, enqueue one job in the request. Never loop.
• Fan out in chunks, insert in bulk.
• Put a unique idempotency key on every notification so retries are free.
• Split delivery into per-channel jobs on per-channel queues.
• Solve the unread badge with a partial index and a watermark.

None of this requires a new gem or a new service. It's the same tables you'd have written anyway — just arranged so that the number of recipients stops being your app's problem.

---

How are you handling fan-out in your app — dedicated tables, an outbox, or a third-party service? I'd be curious what's held up at scale.