How a Silent N+1 Query Almost Melted Our Production Database (and How We Smashed It)

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Every developer fears the silent killer: a bug that doesn’t throw a 500 Internal Server Error, doesn't crash the server instantly, but quietly turns your API into a slow, memory-choking nightmare.

This is the story of how a seemingly harmless 3-line code change triggered an N+1 query monster, and how we used Sentry performance monitoring to track it down and smash it before a major launch.

---

😱 The Incident: High Latency & Spiking CPU

It started on a Tuesday afternoon. Users began reporting that our dashboard's "Recent Activity" feed was taking 6–8 seconds to load.

Looking at our metrics:

• API response times shot up from 120ms to 6,500ms.
• Database CPU utilization spiked to 92%.
• Zero exceptions were being thrown. Everything technically "worked"—it was just painfully slow.

---

🔍 The Investigation: Following the Traces

We pulled up Sentry Performance Tracing to inspect the API endpoint /api/v1/user/activity.

When we looked at the waterfall span breakdown, the problem jumped out immediately:


[GET] /api/v1/user/activity
└── SELECT * FROM activities LIMIT 50 ( 15ms )
└── SELECT * FROM users WHERE id = 12 ( 12ms )
└── SELECT * FROM users WHERE id = 18 ( 14ms )
└── SELECT * FROM users WHERE id = 42 ( 11ms )
... (47 identical individual queries later) ...


Instead of fetching all user profiles in a single query, the application was executing one query for the activity feed, plus 50 additional database queries—one for every single user row!

---

🐛 The Root Cause

A quick git blame revealed a pull request merged earlier that day. A well-intentioned cleanup in our serializer removed eager loading:

❌ Before (Broken Code):

`javascript
// Missing eager loading of the user relation!
const activities = await Activity.findAll({
limit: 50,
order: [['createdAt', 'DESC']]
});

const response = await Promise.all(
activities.map(async (activity) => ({
id: activity.id,
action: activity.action,
user: await activity.getUser() // 💥 50 separate DB calls in a loop!
}))
);
`

---

🛠️ The Smash: Fixing the Leak

To fix this, we updated the query to use eager loading (JOIN), fetching all related user data in a single SQL query:

✅ After (Fixed Code):

`javascript
const activities = await Activity.findAll({
limit: 50,
order: [['createdAt', 'DESC']],
include: [{ model: User, attributes: ['id', 'username', 'avatar'] }] // 🚀 Single JOIN query
});

const response = activities.map((activity) => ({
id: activity.id,
action: activity.action,
user: activity.User
}));
`

---

📊 The Results

• Database Queries: Reduced from 51 queries down to 1 single query.
• Response Time: Dropped from 6,500ms ➡️ 85ms (a 98.6% speedup!).
• DB CPU Load: Plummeted back down to normal levels (< 15%).

---

💡 Lessons Learned

1. Beware of async operations inside loops/maps: Whenever you see await inside a .map(), double-check if you're causing an N+1 problem.
2. Monitor transaction spans: Errors aren't the only metric that matters. Sentry’s performance spans and transaction traces saved us hours of blind guessing.
3. Set up query thresholds: Catching query bloat in staging before it hits production is the ultimate flex!

Comments

[Mustafa ERBAY]

Nice write-up. N+1 queries are one of those problems that often don’t appear as application errors, yet they can become production incidents surprisingly quickly.

One nuance I’d add is that the N+1 pattern isn’t limited to await inside a loop. The real issue is repeated lazy loading of related entities, whether it happens through ORM accessors, property access, or explicit async calls. await is often just where the symptom becomes visible.

I also like the emphasis on tracing. In my experience, query count alone isn’t enough—you also want to correlate it with database execution time, connection pool utilization, lock waits, and the generated execution plans. Sometimes replacing N+1 with a large JOIN is the right solution, but in other cases it can produce row multiplication or excessive result sets. The best fix depends on the access pattern, not just the query count.

Performance bugs rarely come from a single slow query. More often they come from an efficient query executed hundreds of times. That’s why end-to-end tracing is so valuable—it exposes patterns that individual query timings can’t.

[Mia Keller]

Thanks for taking the time to write such a thoughtful reply, Mustafa!

You bring up a fantastic point about lazy loading—it definitely catches people off guard even when there isn't an explicit await in a loop, especially with ORMs that transparently handle relations under the hood.

Also, spot on regarding the trade-offs of large JOINs! Cartesian products and memory spikes from huge result sets can definitely become their own nightmare. In our case, the bounded limit of 50 items made a single include straightforward, but in a deeper nested or unbounded query, batching or separate parameterized queries (WHERE IN (...)) would definitely be worth evaluating.

Curiously, how do you usually set up alerts or thresholds for connection pool utilization or lock waits in your stack before they escalate into incidents? Would love to hear how your team approaches that!