DEV Community: A.Talaat

Building a Rails Notification System That Survives Scale

A.Talaat — Mon, 31 Aug 2026 22:14:43 +0000

Every Rails app grows a notification system eventually.
It usually starts like this:

class Comment < ApplicationRecord
  after_create :notify_watchers

  private

  def notify_watchers
    post.watchers.each do |user|
      NotificationMailer.new_comment(user, self).deliver_later
      user.notifications.create!(message: "New comment on #{post.title}")
    end
  end
end

This works beautifully — until a post has 40,000 watchers and someone comments. Now you have 40,000 inserts and 40,000 mailer jobs inside a request cycle, a timeout, a half-written notification table, and an on-call ping.

The fix isn't a gem. It's four architectural decisions. Let's walk through them.

1. Separate the event from the delivery

The single most important split: what happened is not who gets told and is not how they get told.

Event (comment.created)
  └─> Fan-out (who should know?)
        └─> Notification records (in-app inbox)
              └─> Deliveries (email, push, Slack, SMS)

Three separate concerns, three separate failure domains. If Firebase is down, your in-app inbox still works. If fan-out is slow, the comment still saved.

Model it explicitly:

# events: what happened, once
create_table :events do |t|
  t.string  :kind, null: false            # "comment.created"
  t.jsonb   :payload, null: false, default: {}
  t.references :actor, polymorphic: true
  t.timestamps
end

# notifications: one row per recipient
create_table :notifications do |t|
  t.references :recipient, null: false
  t.references :event, null: false
  t.string   :kind, null: false
  t.jsonb    :payload, null: false, default: {}
  t.string   :idempotency_key, null: false
  t.datetime :read_at
  t.timestamps
end
add_index :notifications, :idempotency_key, unique: true

That unique index is not decoration — it's what makes the whole pipeline safely retryable. More on that in a second.

2. Never fan out in the request

The web request creates the event and enqueues exactly one job. That's it.

class Comment < ApplicationRecord
  after_commit :publish_event, on: :create

  private

  def publish_event
    event = Event.create!(kind: "comment.created", actor: author,
                          payload: { comment_id: id, post_id: post_id })
    Notifications::FanOutJob.perform_later(event.id)
  end
end

Use after_commit, not after_create. Enqueueing inside a transaction is the classic race: the job starts, queries for a record the DB hasn't committed yet, and dies with RecordNotFound.

The fan-out job then resolves the audience and chunks it:

class Notifications::FanOutJob < ApplicationJob
  queue_as :notifications
  BATCH = 1_000

  def perform(event_id)
    event = Event.find(event_id)

    Notifications::Audience.for(event).find_in_batches(batch_size: BATCH) do |users|
      Notifications::WriteBatchJob.perform_later(event_id, users.map(&:id))
    end
  end
end

One long job becomes N short jobs. Short jobs retry cheaply, spread across workers, and don't hold a database connection for four minutes.

3. Bulk insert, don't loop

create! in a loop is 40,000 round trips. insert_all is 40 round trips at batch size 1,000.

class Notifications::WriteBatchJob < ApplicationJob
  def perform(event_id, recipient_ids)
    event = Event.find(event_id)
    now = Time.current

    rows = recipient_ids.map do |id|
      {
        recipient_id: id,
        event_id: event.id,
        kind: event.kind,
        payload: event.payload,
        idempotency_key: "#{event.id}:#{id}",
        created_at: now,
        updated_at: now
      }
    end

    Notification.insert_all(rows, unique_by: :index_notifications_on_idempotency_key)

    Notifications::DeliverBatchJob.perform_later(event_id, recipient_ids)
  end
end

unique_by turns this into an upsert-style no-op on conflict. Now if Sidekiq retries this job after a network blip, you don't double-notify anyone. Idempotency is what lets you retry aggressively, and retrying aggressively is what makes the system reliable.

You skip callbacks and validations with insert_all — that's the trade. Notifications are derived data with a fixed shape, so it's a trade worth making. Just keep the shape enforced in one place (the row builder above).

4. Make channels pluggable and preference-aware

Delivery is where the fan-out multiplies again: one notification, three channels. Keep each channel a small object with the same interface.

module Notifications
  module Channels
    class Email
      def self.deliver(notification)
        NotificationMailer.with(notification: notification).notify.deliver_now
      end
    end

    class Push
      def self.deliver(notification)
        PushClient.send(token: notification.recipient.device_token,
                        title: notification.title, body: notification.body)
      end
    end
  end
end

REGISTRY = { email: Channels::Email, push: Channels::Push, slack: Channels::Slack }

Then resolve preferences once, in bulk, instead of per-user:

class Notifications::DeliverBatchJob < ApplicationJob
  def perform(event_id, recipient_ids)
    notifications = Notification.where(event_id: event_id, recipient_id: recipient_ids)
                                .includes(:recipient)

    args = notifications.flat_map do |n|
      Notifications::Preferences.channels_for(n).map { |ch| [n.id, ch.to_s] }
    end

    Notifications::DeliverJob.perform_all_later(args.map { |a| Notifications::DeliverJob.new(*a) })
  end
end

perform_all_later (Rails 7.1+) pushes the whole batch to Redis in one call instead of one round trip per job. On Sidekiq you can also reach for Sidekiq::Client.push_bulk.

Each leaf job handles exactly one delivery, so failures are isolated:

class Notifications::DeliverJob < ApplicationJob
  retry_on Net::OpenTimeout, Errno::ECONNREFUSED,
           wait: :polynomially_longer, attempts: 5
  discard_on ActiveJob::DeserializationError

  def perform(notification_id, channel)
    notification = Notification.find(notification_id)
    Notifications::REGISTRY.fetch(channel.to_sym).deliver(notification)
  end
end

Give email, push, and in-app their own queues with their own concurrency. A backed-up SMS provider should never starve your push notifications.

The unread badge problem

Here's the query that quietly kills you at scale — it runs on every single page load:

current_user.notifications.where(read_at: nil).count

Two fixes, use both:

A partial index, so Postgres only indexes rows you actually query:

add_index :notifications, [:recipient_id, :created_at],
          where: "read_at IS NULL",
          name: "index_unread_notifications"

A read watermark, so "mark all as read" is one write instead of 10,000:

# users.notifications_read_at
def mark_all_read!
  update_column(:notifications_read_at, Time.current)
end

def unread_count
  notifications.where(read_at: nil)
               .where("created_at > ?", notifications_read_at || Time.at(0))
               .limit(100).count
end

The limit(100) is deliberate. Nobody needs to know they have 8,412 unread notifications — "99+" is the same information at a fraction of the cost.

What to watch in production

Four numbers tell you whether this is healthy:

Fan-out lag — event created → last notification row written
Delivery lag — notification created → provider accepted
Queue depth per channel — the first thing to spike when a provider degrades
Delivery failure rate by channel — a broken push cert looks exactly like a quiet week unless you're graphing this

Emit them as counters and histograms and alert on the p95, not the average. Notification systems fail at the tail.

The short version

Write one event, enqueue one job in the request. Never loop.
Fan out in chunks, insert in bulk.
Put a unique idempotency key on every notification so retries are free.
Split delivery into per-channel jobs on per-channel queues.
Solve the unread badge with a partial index and a watermark.

None of this requires a new gem or a new service. It's the same tables you'd have written anyway — just arranged so that the number of recipients stops being your app's problem.

How are you handling fan-out in your app — dedicated tables, an outbox, or a third-party service? I'd be curious what's held up at scale.

The Hidden Cost of Missing or Wrong Database Indexes in Rails

A.Talaat — Wed, 17 Jun 2026 19:17:43 +0000

Do you think your SQL Query Is Fine?!
Lets today talk about the hidden cost of missing or wrong Database indexes in Rails.

A few years ago, I was investigating a Rails endpoint that suddenly became slow.

Nothing unusual had happened.
No major deployment.
No infrastructure changes.
No traffic spike.
The endpoint that normally responded in under 100ms was now taking several seconds.
The Rails logs looked clean.
The SQL query looked reasonable.
The application servers were healthy.

The problem ......... A missing database index. One small database optimization turned a multi-second query into a millisecond operation.

If you've ever experienced unexplained performance degradation in a Rails application, there's a good chance missing indexes were involved.

let's go deeper in indexing.

1- What Is a Database Index?
Think of a database index as the index section at the back of a book.
Without an index, finding a specific topic means reading page after page until you locate the information.
With an index, you can jump directly to the correct page.
Databases work in a similar way.
When a query filters records using indexed columns, the database can locate matching rows quickly.
Without an index, the database may be forced to scan the entire table.
And that's where trouble begins.

2- The Problem Isn't Today
One of the reasons missing indexes are so dangerous is that they often go unnoticed during development.

Consider this scenario:

Development database: users table contains 500 rows
Production database: users table contains 50 million rows

A query that feels instant during development may become painfully slow in production.
The larger your dataset becomes, the more expensive table scans become.
This is why indexing is ultimately a scalability concern.

3- Why Missing Indexes Cause Performance Problems
When a query searches for records, the database has two options:

Best Case (Use an index).
Locate matching rows quickly
Return results
Done

Worst Case (Perform a full table scan).
Read row 1
Read row 2
Read row 3
...
Read row 50,000,000

The second option consumes:

More CPU
More memory
More disk I/O
More query execution time

As a result:

Requests become slower
Application workers remain busy longer
CPU usage increases
Database load grows

Eventually users begin noticing.

4- Common Places Where Missing Indexes Appear
Most Rails applications contain several predictable indexing candidates.

Foreign Keys:

belongs_to :user

If your queries frequently search by:

WHERE user_id = ?

An index is usually required.

Status Columns:

Order.where(status: "completed")

Frequently filtered columns are often strong indexing candidates.

Search Fields

User.find_by(email: email)

Email addresses, usernames, slugs, and external identifiers are common index targets.

Sorting Operations

Post.order(created_at: :desc)

Sorting large datasets can become expensive without proper indexing.

5- Why Rails Developers Accidentally Create This Problem
Most missing index issues come from one of these situations.

Feature-Driven Development

A feature is delivered.
Tests pass.
Users are happy.
Nobody thinks about query execution plans.
The feature works today.
The performance problem appears six months later.

Small Development Databases
Local databases rarely contain production-sized datasets.
Queries that appear instant locally may behave very differently with millions of records.

Schema Evolution

Applications grow.
New reports are added.
New filters are introduced.
New business requirements emerge.
The indexing strategy that worked last year may no longer be sufficient today.

6- How to Avoid Missing Indexes
The easiest strategy is simple:
Whenever you introduce a new query pattern, ask yourself:

Will this column be frequently searched, filtered, joined, or sorted?
If the answer is yes, consider whether an index should exist.

Examples:

User.find_by(email: email)
Order.where(customer_id: customer.id)
Post.order(created_at: :desc)

These queries often benefit from indexing.

7- Helpful Tools Every Rails Engineer Should Know

EXPLAIN

The first tool every engineer should learn.

User.where(email: email).explain

Rails can show the database execution plan and reveal whether indexes are being used.

PgHero

One of the most valuable tools for PostgreSQL-powered Rails applications.

PgHero can identify:

Missing indexes
Unused indexes
Slow queries
Expensive database operations

Many Rails teams discover performance issues with PgHero long before users report them.

Bullet

Most engineers know Bullet for detecting N+1 Queries.
However, it can also provide useful performance insights during development.

New Relic

Provides visibility into:

Slow database transactions
Query performance
Endpoint bottlenecks

Excellent for identifying problematic database behavior in production.

Datadog APM

Useful for tracing application requests and understanding where time is spent between Rails and PostgreSQL.

8- When Not to Add an Index
A common mistake is believing every column needs an index.
Indexes are powerful, but they are not free.

Every index:

Consumes storage
Slows INSERT operations
Slows UPDATE operations
Increases maintenance overhead

The goal is not to index everything.
The goal is to index the right things.
Good indexing is a balancing act.

The Silent Performance Killer in Rails: Understanding and Fixing N+1 Queries

A.Talaat — Tue, 16 Jun 2026 08:44:45 +0000

If you've ever investigated a slow Rails endpoint, chances are you've encountered one of the most common performance problems in the Rails ecosystem:

N+1 Queries.

The scary part?

Your application can appear perfectly healthy during development, pass all tests, and work flawlessly in production for months.

Then one day your dataset grows, traffic increases, and suddenly an endpoint that used to respond in 100ms now takes several seconds.

No infrastructure changes.

No database outages.

No major code deployments.

Just a silent performance killer hiding inside your ActiveRecord relationships.

In this article we'll explore:

What N+1 Queries are
Why they happen so frequently in Rails applications
How they impact performance
Techniques to prevent them
Tools that can help detect them before they reach production

What Is an N+1 Query?

An N+1 Query occurs when your application executes:

One query to fetch a collection of records
One additional query for each record in that collection

Let's imagine a simple blog application.

Each Post belongs to an Author.

class Post < ApplicationRecord
  belongs_to :author
end

Now imagine the following code:

posts = Post.limit(100)

posts.each do |post|
  puts post.author.name
end

At first glance this looks innocent.

However, Rails may execute:

SELECT * FROM posts LIMIT 100;

Then:

SELECT * FROM authors WHERE id = 1;
SELECT * FROM authors WHERE id = 2;
SELECT * FROM authors WHERE id = 3;
...

Instead of executing two queries, the application executes:

1 + 100 = 101 queries

This pattern is known as an N+1 Query.

Why Does This Happen?

The answer lies in one of Rails' most powerful features:

Lazy Loading.

ActiveRecord delays loading associated records until they are actually needed.

For example:

post.author

does not load the author when the post is fetched.

It loads the author only when that line is executed.

This behavior is incredibly convenient because it reduces unnecessary database access.

However, when used inside loops, it can accidentally create hundreds or even thousands of additional queries.

The framework is doing exactly what we asked it to do.

The problem is that we didn't realize how many times we were asking.

Why N+1 Queries Are Dangerous

Many engineers focus on query execution time.

The real problem is query volume.

Imagine:

10 users → 11 queries
100 users → 101 queries
1,000 users → 1,001 queries
10,000 users → 10,001 queries

As your data grows, the number of database round trips grows with it.

This creates:

Increased database load
Higher application CPU usage
More network overhead
Longer request times
Reduced scalability

What worked perfectly in staging may become a production nightmare months later.

How to Avoid N+1 Queries

The most common solution is eager loading.

Rails allows you to preload associations before they are accessed.

Instead of:

posts = Post.limit(100)

Use:

posts = Post.includes(:author).limit(100)

Rails now loads:

SELECT * FROM posts LIMIT 100;
SELECT * FROM authors WHERE id IN (...);

Only two queries.

No matter how many posts are returned.

This is one of the easiest performance wins available in Rails.

Common Places Where N+1 Queries Hide

Many engineers look only at controllers.

In reality, N+1 issues often appear in:

API Serializers

PostSerializer.new(posts)

Jbuilder Templates

json.author_name post.author.name

GraphQL Resolvers

field :author

View Templates

<%= post.author.name %>

Background Jobs

users.each do |user|
  user.account.plan
end

Any place that loops through records and accesses associations can potentially create N+1 Queries.

Helpful Tools for Detecting N+1 Queries

One of the best investments a Rails team can make is detecting N+1 Queries early.

Bullet Gem

The most popular N+1 detection tool in the Rails ecosystem.

It automatically alerts developers when associations should be eager loaded.

Example notification:

USE eager loading detected
Add to your query: .includes(:author)

Rack Mini Profiler

Provides detailed request analysis.

Helps identify:

Slow queries
Query counts
Rendering bottlenecks
Performance hotspots

Skylight

Excellent for production monitoring.

It helps visualize database activity and identify endpoints suffering from excessive query counts.

New Relic

Provides detailed transaction tracing and database monitoring.

Very useful for identifying performance degradation caused by query-heavy endpoints.

Datadog APM

Widely used in large-scale Rails environments.

Offers visibility into:

SQL execution
Request traces
Database latency
Application bottlenecks

Final Thoughts

N+1 Queries are dangerous because they don't usually break functionality.

Your tests pass.

Your code looks clean.

Users may not notice the problem immediately.

But as data grows, the hidden cost grows with it.

Understanding N+1 Queries is one of the first steps toward becoming a performance-conscious Rails engineer.

And the good news?

Many N+1 issues can be fixed with a single line of code.

In the next article, we'll dive deeper into advanced eager loading strategies, compare includes, preload, and eager_load, and discuss how to detect N+1 Queries in production environments before your users notice them.

Why your Ruby on Rails application has high CPU usage?

A.Talaat — Sun, 14 Jun 2026 10:45:13 +0000

Have you ever received an alert saying:
CPU usage is above 90%

Or maybe one of your production servers suddenly became slow, requests started timing out, and users began reporting performance issues.

As Rails engineers, one of the most common production incidents we face is unexpectedly high CPU utilization.

The tricky part?
High CPU usage is rarely the actual problem.

It's usually a symptom of something else happening inside your application, infrastructure, database, background jobs, or even your code architecture.

I will explore the most common reasons behind high CPU consumption in Ruby on Rails applications.
won't go too deep into each topic because every single one deserves.

Let's get started with the reasons:

1. N+1 Database Queries

Typical Symptoms

Slow API responses
Increased database load
High application CPU usage

2. Missing Database Indexes

Typical Symptoms

Slow endpoints
High database CPU
Increasing request latency

3. Inefficient ActiveRecord Queries

Typical Symptoms

Excessive memory consumption
Long request execution times
CPU spikes during large queries

4. Expensive Background Jobs

Typical Symptoms

CPU spikes during scheduled jobs
Delayed Sidekiq queues
Increased worker execution times

5. Memory Pressure and Garbage Collection

Typical Symptoms

CPU spikes without increased traffic
Slow request processing
High GC activity

6. Excessive JSON Serialization

Typical Symptoms

API endpoints slower than expected
High CPU during response generation
Large response payloads

7. Traffic Spikes and Uneven Load Distribution

Typical Symptoms

Only specific nodes experience high CPU
Random server crashes
Uneven infrastructure metrics

8. External Service Bottlenecks

Typical Symptoms

Growing request queues
Increased response times
Thread pool saturation

9. Inefficient Caching Strategies

Typical Symptoms

Repeated database queries
CPU spikes during peak traffic
Reduced cache hit ratio

10. Ruby Code That Doesn't Scale

Typical Symptoms

CPU increases with data growth
Slow batch operations
Performance degradation over time

Final Thoughts
When CPU usage becomes high, avoid jumping directly to conclusions.
The real challenge isn't reducing CPU usage.
The real challenge is identifying what's consuming the CPU.
In future articles, we'll dive deeper into each of these topics, including:

Detecting N+1 queries in production
Database indexing strategies for Rails applications
Sidekiq performance tuning
Ruby Garbage Collection optimization
API serialization best practices
Infrastructure troubleshooting techniques

Remember:
High CPU is usually a symptom. Your job as an engineer is to find the disease.