The new customer who took down everyone: an ORM include that ate our CRM

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

A real production war story about how
one endpoint, one include too deep, and one very enthusiastic new customer combined into a
global outage — and what actually fixed it (after the first fix didn't).

The 15 minutes where everything was down

It started the way these things always do: a new customer onboarded, we imported their data, and
minutes later everyone's app started falling over. Not just the new customer — the whole
multi-tenant CRM. Dashboards spun. Deal pages hung. Then requests started timing out across the
board.

The trigger was data volume. The new account came in with a lot of history — far more
opportunities, contacts, activities, and line items than our typical tenant. Nothing about that
should take down other customers… except it did.

Following the smoke

The outage lined up almost perfectly with traffic to one endpoint: the one that returns the full
details of an opportunity/deal. On paper it's a read. In practice it had grown, over a couple
of years, into a monster.

It was a single Sequelize query with a tree of includes:

Opportunity.findByPk(id, {
  include: [
    { model: Account, include: [Contact, Address] },
    { model: LineItem, include: [Product, { model: PriceTier, include: [Currency] }] },
    { model: Activity, include: [User, Note] },
    { model: Stage, include: [Pipeline] },
    // …and more
  ],
})

Every include became a JOIN (or a follow-up query), and several of them pulled in their own
nested includes. For a normal-sized deal it was fine. For the new customer's fattest
opportunities — hundreds of line items, each with products and price tiers, plus a long activity
history — a single "show me this deal" call was fanning out into an enormous result set and a
pile of un-indexed lookups. The query blew past our HTTP timeout, the request died, the client
retried, and the retries piled onto an already-struggling database. Connections backed up, and
because every tenant shared the same pool, everyone felt it.

Fix #1: the one that felt good and wasn't

Under pressure, we did the obvious thing: we raised the timeout. The endpoint stopped
"failing," the alarms went quiet, and we told ourselves we'd bought time to do it properly.

That was a band-aid on a severed artery. All we'd done was let the slow query run longer while
holding a connection hostage the entire time. A week or two later, with a bit more data and a bit
more concurrency, the exact same failure came back — except now each hung request lingered even
longer before dying, so the pile-up was worse. Raising a timeout doesn't make a query fast; it
just changes how long you wait to find out it's slow.

Fix #2: the one that actually held

The real fix had two halves.

1. Stop returning everything at once. We split the single mega-endpoint into several focused
calls. The deal page didn't actually need every nested activity, note, and price tier on first
paint — it needed the core opportunity, and then the related collections lazily, as the user
scrolled to them. So we replaced the one-query-to-rule-them-all with:

• a lean GET /opportunity/:id that returns just the opportunity and its immediate summary, and
• separate, paginated endpoints for the heavy children (/opportunity/:id/line-items, /activities, etc.), each returning only the columns the UI renders.

That alone collapsed the worst-case payload from "the entire deal graph" to something bounded and
predictable, and it meant a single fat deal could no longer monopolize a connection for seconds.

2. Index the paths we actually query. The child lookups were hitting foreign keys and filter
columns that weren't indexed, so they degraded from index seeks to sequential scans as the tables
grew — which is exactly why the problem only showed up with a large tenant. We added the missing
indexes (the opportunity_id foreign keys on the child tables, plus the columns we filtered and
sorted on) so those per-collection queries stayed fast regardless of how much history a customer
had.

Split the work, return only what's needed, and index the access paths — and the outage stopped
recurring. We haven't seen it since. (I'm knocking on wood as I type this.)

What I took away from it

• A timeout increase is not a fix. It's a way to postpone the same failure into a worse one. If raising a limit makes the symptom disappear, the real bug is still there — you've just muted the alarm.
• include depth is a loaded gun in any ORM. Every convenient nested include is a JOIN or an N+1 waiting to matter. It's invisible at small scale and catastrophic at large scale.
• In multi-tenant systems, one customer's data volume is everyone's problem when they share a connection pool. "It's slow for one account" quietly means "it's fragile for all of them."
• Slowness usually lives at the access path. The endpoint looked like an application problem; the fix was mostly shape of the data returned + indexes. Measure before you rewrite.

The unglamorous version of the lesson: return less, and make sure the database can find what you
ask for. Both halves mattered — decomposition kept any one request bounded, and indexing kept
each of those smaller requests fast as the data grew.