How I Slayed the N+1 Query Beast: Lessons from Lord of the Rings

The Quest Begins (The “Why”)

I still remember the day my side‑project went from “cool prototype” to “slow‑as‑molasses nightmare”. I had just shipped a feature that listed a user’s blog posts together with the latest comment on each post. The UI looked slick, but every time I opened the page the server seemed to take a coffee break. I opened the logs and saw a terrifying pattern:

SELECT * FROM posts WHERE user_id = 42;
SELECT * FROM comments WHERE post_id = 1 ORDER BY created_at DESC LIMIT 1;
SELECT * FROM comments WHERE post_id = 2 ORDER BY created_at DESC LIMIT 1;
SELECT * FROM comments WHERE post_id = 3 ORDER BY created_at DESC LIMIT 1;
...

One query to fetch the posts, then one extra query for each post to grab its latest comment. If a user had 50 posts, that was 51 round‑trips to the database. I felt like I was stuck in a loop, watching the same scene over and over—definitely not the heroic adventure I’d signed up for.

That’s when I realized I had stumbled into the infamous N+1 query problem. The dragon was real, and it was devouring my app’s performance. Time to grab my sword and learn how to defeat it.

The Revelation (The Insight)

The treasure I uncovered wasn’t a mythical artifact; it was a simple mindset shift: fetch what you need in batches, not one‑by‑one. Most ORMs (ActiveRecord, Sequelize, Prisma, Django ORM, etc.) give you two powerful tools for this:

1. Eager loading (sometimes called includes, select_related, prefetch_related, or join).
2. Explicit joins / aggregate queries when you only need a few columns from the related table.

The idea is to let the database do the heavy lifting in a single round‑trip (or a few, but far fewer than N+1). Once you start thinking in terms of “give me all the data I’ll need up front”, the N+1 monster shrinks to a harmless critter.

I’ll show you the before‑and‑after code using a typical Rails‑style ActiveRecord example, but the same principles apply wherever you’re writing queries.

Wielding the Power (Code & Examples)

The Trap: Naive Loop (The “Before”)

# app/controllers/posts_controller.rb
def index
  @posts = Post.where(user_id: current_user.id)   # 1 query
end

# app/views/posts/index.html.erb
<% @posts.each do |post| %>
  <div class="post">
    <h3><%= post.title %></h3>
    <p><%= post.body %></p>
    <% if post.comments.any? %>
      Latest comment: <%= post.comments.order(created_at: :desc).first.body %>
    <% end %>
  </div>
<% end %>

What happens here?

• Line 2 loads all posts (1 query).
• Inside the view, each iteration calls post.commons.any? and then post.comments.order(...).first.
• Those calls trigger a separate query for every post → the dreaded N+1.

If you have 30 posts, you’re issuing 31 queries. Not exactly the “fast‑travel” you want.

The Victory: Eager Loading (The “After”)

# app/controllers/posts_controller.rb
def index
  # Grab posts AND preload their comments in one go
  @posts = Post.where(user_id: current_user.id)
               .includes(:comments)   # <-- the magic line
end

Now the ORM issues two queries:

SELECT * FROM posts WHERE user_id = 42;
SELECT * FROM comments WHERE post_id IN (/* list of post ids */);

ActiveRecord then associates the comments with their parent posts in memory. The view stays exactly the same, but now no extra queries are fired inside the loop.

When You Only Need a Column

Sometimes you don’t need the whole comment object—just the latest comment’s body. You can push that work into the DB with a LEFT JOIN LATERAL (PostgreSQL) or a subquery, keeping it to a single query:

# app/models/post.rb
class Post < ApplicationRecord
  has_many :comments

  # Scope to attach the latest comment body as an attribute
  scope :with_latest_comment, -> {
    left_joins(:comments)
      .select('posts.*, latest_comments.body AS latest_comment_body')
      .joins("LEFT JOIN LATERAL (
                SELECT body
                FROM comments
                WHERE comments.post_id = posts.id
                ORDER BY created_at DESC
                LIMIT 1
              ) latest_comments ON true")
  }
end

# app/controllers/posts_controller.rb
def index
  @posts = Post.where(user_id: current_user.id)
               .with_latest_comment
end

<!-- app/views/posts/index.html.erb -->
<% @posts.each do |post| %>
  <div class="post">
    <h3><%= post.title %></h3>
    <p><%= post.body %></p>
    <% if post.latest_comment_body.present? %>
      Latest comment: <%= post.latest_comment_body %>
    <% end %>
  </div>
<% end %>

Now we’re down to one query that returns the posts plus the latest comment body for each. The DB does the filtering, sorting, and limiting—exactly what it’s good at.

Common Traps to Avoid

• Calling size or count on an association inside a loop – each call can trigger another query. Use counter_culture caches or pre‑load counts with left_joins and group.
• Assuming includes always solves everything – if you later add a where condition on the joined table, you may need references or eager_load instead. Keep an eye on the generated SQL (run ActiveRecord::Base.logger = Logger.new(STDOUT) in development to see it).
• Over‑eager loading – loading massive associations you never use can waste memory. Only eager‑load what you actually need in the view or serializer.

Why This New Power Matters

After I switched to eager loading (and later to the lateral join trick for the latest comment), my page load time dropped from ~1.2 seconds to under 150 ms on a modest dataset. The server could handle far more concurrent users, and my hosting bill stopped looking like a phone number.

More importantly, I stopped dreading the “show me the data” page. I could now add richer features—like showing the top three tags per post, or a count of reactions—without fearing a sudden performance cliff. The N+1 beast was no longer a looming threat; it was a solved puzzle, and I felt like Neo when he first sees the Matrix code: everything just clicked.

If you’re building any web app that talks to a relational database, mastering batch fetching is like getting a +1 sword in your inventory. It’s not flashy, but it makes every subsequent quest (feature, refactor, scaling effort) far easier.

Your Turn

Grab a piece of your own code where you’re looping over a collection and fetching related data inside that loop. Run the query logger, spot the N+1 pattern, and try one of the techniques above—whether it’s includes, prefetch_related, select_related, or a custom join/explicit subquery.

Challenge: Reduce the number of database queries on that page by at least half, and share your before/after query counts in the comments. I can’t wait to hear how you slayed your own N+1 dragon!

Happy querying, and may your indexes be ever in your favor. 🚀