This is a submission for DEV's Summer Bug Smash: Clear the Lineup, powered by Sentry.
Submitting for: Clear the Lineup · Best Use of Sentry
My Sentry trace had no database spans. The database was 77% of the request.
7,580 ms to 1,671 ms. 24.8 MB of serialized result down to 6.0 MB. Same fifteen numbers on screen.
Fifteen numbers. Four stat tiles, three donut slices, six bars, four recent invoice rows. That is the entire visible output of the page.
It took 7.58 seconds.
On my laptop the same page took 111 ms, because my laptop had four invoices in it. Same code, same query, same render, same everything. The only variable that moved was how much there was.
So I installed @sentry/nextjs, opened the trace, and found one grey box with dashboard written on it and nothing underneath.
The instrument was silent. I read that as good news for about ten minutes.
Reproduce every number in this post: github.com/SoumyaEXE/dashboard-perf-repro - seeder, benchmark harness, and the before and after query pair on a throwaway MongoDB. The application repo stays private because it holds real client invoices.
TL;DR
A Next.js 16 dashboard took 7.58 s to render fifteen aggregate numbers. The first Sentry trace showed no database spans at all, because mongoIntegration() and mongooseIntegration() are not among the default integrations in @sentry/node. Turning them on revealed that MongoDB was 77% of the request, and that the slow spans were getMore (cursor continuation), not find. That distinction matters: a slow find means an index problem, a long getMore means a payload problem. The root cause was an unprojected Invoice.find() pulling a base64 logo out of all 2,004 documents, plus a JavaScript reduction over 4,806 tasks to produce eight rows. Three changes, a .select() projection, a $facet aggregation, and a comparator that returns 0, took it to 1.67 s and dropped the serialized result from 24.8 MB to 6.0 MB.
If your trace shows no database spans, verify the integration before you conclude the database is fast.
The measurement rules, before any number below
Two different numbers appear in this post and they are not the same measurement. The 14.89 s in the trace screenshots is a development request: Turbopack compiling, dev overhead, 100% Sentry sampling. The 7,580 ms in every before and after table is the benchmark: the same server functions, called directly, no instrumentation attached. The trace diagnosed the bug. The benchmark scored it.
Every byte figure is a serialized result size, Buffer.byteLength(JSON.stringify(result)), produced by my own harness. It is not compressed network traffic. Real wire numbers need MongoDB profiler data or driver-level metrics, and I did not collect those, so I am not going to imply I did.
Every timing is the median of five runs against the same seeded corpus, before and after.
| Metric | Before | After | Change |
|---|---|---|---|
| Total server-function time | 7,580 ms | 1,671 ms | 4.5x faster |
getInvoicesAdmin() |
6,669 ms | 1,597 ms | 4.2x faster |
getDashboardOverview() |
911 ms | 74 ms | 12.3x faster |
| Serialized MongoDB result | 24,849 KB | 6,048 KB | 76% lower |
| Overview serialized result | 2,490 KB | 2.0 KB | 99.9% lower |
MongoDB getMore spans |
Present | Absent | Result fell below the cursor batch threshold |
| Self-contradictory comparator | true |
false |
Correctness defect, not a speed one |
Project overview
DevLabs is the agency platform my team runs on: Next.js 16, React 19, MongoDB, Mongoose, roughly 34,000 lines. Marketing site, internal dashboard, and a client-facing portal holding invoices, contracts and customer records.
This is not a demo I built to have something to profile. Clients open this page. The data on it is real customer names and real invoice totals, which is exactly why the instrumentation section below has a privacy subsection instead of a shrug, and why the application repo is private. Every diff in this post is the actual change that shipped, and the public repro repo above runs the same query shapes against a schema you can seed yourself.
Why a Sentry trace can show no database spans
The first trace had one opaque server span. No find. No getMore. Nothing that looked like a database at all.
The tempting read is the database is fine, go look at React. So before I went looking at React, I asked the package what it was actually loading:
$ node -e "const n=require('@sentry/node'); \
const d=n.getDefaultIntegrations({}); \
console.log('total:', d.length); \
console.log('mongo:', d.map(i=>i.name).filter(x=>/[Mm]ongo/.test(x)))"
total: 17
mongo: []
Seventeen default integrations on this version. Zero of them knew what MongoDB was.
A trace with no database spans has not told you the database is fast. It has told you nobody is watching it. Those are wildly different sentences and only one of them is in your dashboard.
// src/instrumentation.ts
const integrations = isNode
? await import("@sentry/node").then((node) => [
node.mongoIntegration(),
node.mongooseIntegration(),
])
: [];
Sentry.init({
dsn: DSN,
tracesSampleRate,
integrations,
sendDefaultPii: false,
});
Two lines. The grey box turned into this:
| Span | Duration |
|---|---|
mongodb {"getMore": ...}
|
4.42 s |
mongoose.projects.find |
2.67 s |
mongodb {"getMore": ...}
|
1.35 s |
mongoose.customers.find |
134 ms |
mongodb {"find": {"firebaseUid": "?"}}
|
86 ms |
Two getMore spans totalling 5.77 seconds in a single dashboard load.
What a slow MongoDB getMore span actually tells you
getMore is cursor continuation. When a result does not fit in one batch, the driver goes back for the rest. A getMore span on its own proves nothing except that a result set was large enough to need more than one trip.
That is still the sentence that reframed the whole investigation. A slow find would have sent me to indexes and query predicates, which is where I would have gone by instinct and where I would have found nothing, because the predicate here was {} and there is no index that makes {} faster. Long getMore next to a 21.8 MB serialized result says something different: the query is not struggling to locate the documents, it is struggling to carry them.
The dev trace made the shape obvious once the spans existed:
http.server GET /dashboard ................. 14.89 s
db mongoose.invoices.find ................ 13.32 s
db mongoose.users.findOne ................ 845 ms
resolve page components ................... 3.93 ms
build component tree ...................... 11.73 ms
Sentry span breakdown: db 77% · default 12% · function.nextjs 12%
Those children do not add up to the parent and they are not supposed to. Spans overlap, and smaller framework spans are omitted. 14.89 s is the transaction, the rest is attribution. Also worth naming: the saslStart, saslContinue and createIndexes spans are connection handshake, they fire once per process, and I threw them out of the recurring diagnosis rather than counting them as dashboard cost.
Three point nine three milliseconds to resolve the page components. Eleven point seven to build the tree. React was never the problem, and if the integrations had stayed off I would have spent the evening proving that the slow way.
77% of the transaction, in a request I had already been told contained no database work.
Making my laptop lie convincingly
The bug could not be reproduced locally because four invoices is not a data volume, it is a rounding error. So the first thing I wrote was not a fix. It was a seeder.
It points at whatever MONGODB_URI is configured and follows two rules I would want any script pointed at a real database to follow:
- It only ever inserts. It never updates or deletes a document it did not create.
- Every generated document is tagged, so cleanup can only ever touch generated data.
const SEED_TAG = "[perf-seed]";
// Stands in for the base64 data URI on a real uploaded logo.
const FAKE_LOGO = `data:image/png;base64,${"iVBORw0KGgoAAAANSUhEUg".repeat(360)}`;
That second constant is the whole post. If your seeder generates tidy little documents, your seeded database is not your production database wearing a costume, it is a different database that happens to have the same row count. Seed the ugly fields.
npm run seed:perf -- --yes # 2,000 invoices, 40 projects
npm run bench:dashboard # read-only
Corpus: 2,004 invoices, 42 projects, 4,806 tasks visible to the overview query, 5 runs, median reported.
getInvoicesAdmin() 6668.8 ms 22358.6 KB serialized result
getDashboardOverview() 911.4 ms 2490.0 KB serialized result, 4806 tasks scanned
8 activity rows kept
total server time 7580.2 ms
Four thousand eight hundred and six tasks scanned. Eight rows kept.
What the page was actually doing
flowchart TD
A["GET /dashboard"] --> B["getInvoicesAdmin()"]
A --> C["getDashboardOverview()"]
B --> D["Invoice.find()<br/>no filter · no limit · no projection"]
D --> E["2,004 full documents<br/>base64 logo in every invoice<br/>21.8 MB serialized"]
E --> F["invoiceFieldsFrom() × 2,004"]
C --> H["Project.find().select('name board activity')"]
H --> I["every board, every task, every log row<br/>2.4 MB · 4,806 tasks"]
I --> J["reduce in JavaScript<br/>8 activity rows + counters"]
F --> K["return 2,004 invoice rows"]
J --> K
K --> L["render 15 numbers"]
style E fill:#c62828,color:#fff
style I fill:#c62828,color:#fff
style L fill:#2e7d32,color:#fff
Three defects, and none of them are exotic.
Defect 1: an unprojected Mongoose find() shipping a picture with every row
const invoices = await Invoice.find()
.populate(INVOICE_POPULATE)
.sort({ createdAt: -1 })
.lean();
No projection means every field of every document. The invoice schema carries fields.companyDetails.logo, and uploaded logos live there as base64 data URIs. So every one of the 2,004 invoices came over the wire carrying a complete copy of an image.
The list renders a serial number, a customer name and a total. It has never once rendered the logo. It was being handed one anyway, two thousand times, on every load.
+const INVOICE_LIST_EXCLUDE =
+ "-fields.companyDetails.logo -fields.metadata -access";
const invoices = await Invoice.find()
+ .select(INVOICE_LIST_EXCLUDE)
.populate(INVOICE_POPULATE)
.sort({ createdAt: -1 })
.lean();
serialized result: 22,358 KB -> 6,046 KB
execution time: 6,669 ms -> 1,597 ms
One line. The detail view still selects the full document, so nothing on screen changed anywhere. The list contract is identical. The only thing removed is fields the caller was never reading.
Defect 2: the comment was true, the query was not
The code said this:
// Aggregated CRM data for the dashboard Overview tab. One query pass,
// everything trimmed to what the cards render.
The code then did this:
Project.find().select("name board activity").lean()
Every board. Every task on every board. The entire append-only activity log, for every project. Then JavaScript reduced all of it down to eight activity rows, four counters and four project bars, and threw the rest away.
The comment was not lying about the intent. It was describing a reduction that was happening in the wrong process. So I moved it to the one that already has the data:
Project.aggregate([
{
$facet: {
recentActivity: [
{ $unwind: "$activity" },
{ $sort: { "activity.time": -1 } },
{ $limit: 8 },
{
$project: {
_id: 0,
projectId: "$_id",
projectName: "$name",
text: "$activity.text",
time: "$activity.time",
},
},
],
taskTotals: [
{ $unwind: "$board" },
{ $unwind: "$board.tasks" },
{ $group: { _id: "$board.tasks.status", count: { $sum: 1 } } },
],
perProject: [
{ $unwind: "$board" },
{ $unwind: "$board.tasks" },
{
$group: {
_id: "$_id",
name: { $first: "$name" },
total: { $sum: 1 },
done: {
$sum: {
$cond: [{ $eq: ["$board.tasks.status", "Done"] }, 1, 0],
},
},
},
},
{ $sort: { total: -1 } },
{ $limit: 4 },
],
},
},
]);
$facet runs three independent reductions in one request, so the three cards do not become three round trips.
execution time: 911 ms -> 74 ms
serialized result size: 2,490 KB -> 2.0 KB
Two point zero kilobytes. That is what fifteen numbers actually weigh.
Defect 3: a JavaScript comparator that disagrees with itself
Two files in this path contained this:
.sort((a, b) => (b.createdAt > a.createdAt ? 1 : -1))
It never returns 0. Feed it two invoices with the same timestamp and it says a comes first. Hand it the same two in the other order and it says b comes first. That is not a sort order, that is an opinion that depends on the input.
Once a comparator violates its contract, the output is at the mercy of array length, input order and whatever the engine felt like doing that day. And equal timestamps are not a thought experiment here: bulk imports, seed runs and fast manual entry all produce them.
-.sort((a, b) => (b.createdAt > a.createdAt ? 1 : -1))
+.sort((a, b) => (
+ a.createdAt === b.createdAt
+ ? 0
+ : a.createdAt < b.createdAt
+ ? 1
+ : -1
+))
self-contradictory comparator: true -> false
This runs in a client component, so inconsistent ordering could in principle produce different server and browser output. I did not reproduce a hydration mismatch in the 64-element benchmark, so that stays a consequence I can argue for and not a symptom I measured. This fix is worth zero milliseconds. It is in here because it was sitting in the same code path and it was wrong.
The trace, after
The getMore spans are gone. Not shorter. Gone, because the projected result fits in batches the driver does not need to chase.
The bars getting smaller is the boring evidence. The shape changing is the real one:
Before: full documents -> oversized result -> long getMore spans
After: projected fields -> smaller result -> no getMore at all
Best Use of Sentry
Distributed Tracing did the diagnosis, and it did it by showing me nothing. An empty waterfall under a 7.6 second request is a finding. It is the finding that reframed the entire investigation, because mongoIntegration() and mongooseIntegration() are not in the 17 defaults, so the database was invisible rather than fast.
Once the spans existed, getMore was the tell. A slow find sends you to indexes. A long getMore beside a 21.8 MB result says the query is not struggling to locate documents, it is struggling to carry them. Different bug, different fix, and I would have spent the evening in the wrong file without that span.
Span attributes tag the flow so the trace answers itself on the next incident:
const span = Sentry.getActiveSpan();
span?.setAttribute("invoice.count", invoices.length);
span?.setAttribute("invoice.projection", "list");
span?.setAttribute("invoice.serialized_kb", serializedKb);
Session Replay, configured for a page that renders real client names and real invoice totals:
Sentry.replayIntegration({
maskAllText: true,
blockAllMedia: true,
});
sendDefaultPii: false covers the event payload. Masking and media blocking cover the recording. Different data paths, and setting one does not set the other. Replay on a billing screen without those two flags is a data export with a nice UI.
Sampling is deliberate rather than inherited. 100% in development, because while I am diagnosing I want every request. Sampled in production, because I do not need every request to know where time goes.
What Seer said about it
[SCREENSHOT: Seer root cause analysis panel for the GET /dashboard transaction]
[Quote Seer's actual root cause text here, verbatim, then one or two
sentences on where it agreed with your manual diagnosis and where it
did not. The interesting version of this section is the disagreement,
not the agreement.]
The signal I was missing, and the one I added afterwards
Traces found this bug. Traces would not have caught it.
Traces are sampled, and the one request a client complains about is reliably the one that got sampled out. The bug never threw, so Issues stayed empty the entire time. The page returned 200 for months while it was ruining someone's afternoon.
Errors, traces, logs, metrics. I had one of the four pointed at this page, and it was the sampled one. So after the fix I turned on the two that would have paged me instead of the client.
// LOG: the decision point, on every request, not sampled.
Sentry.logger.info("dashboard invoice list", {
rows: invoices.length,
projection: "list",
serialized_kb: serializedKb,
});
Logs are searchable by client and by route without hoping the right request survived sampling. If one account starts returning three thousand rows where everyone else returns forty, the log says so before anybody files a ticket.
The metric is the aggregate version of the same question. dashboard.result_bytes, a distribution keyed by route, is the line that moves for weeks before a page ever feels slow. Every unprojected find() is a bet that N stays small. A distribution on result size is that bet, plotted. Latency alerts fire after the page is already bad. Payload size trends up first, and it trends up quietly.
What I did not fix, and what I refuse to claim
Post-fix benchmark:
getInvoicesAdmin() 1,597 ms
getDashboardOverview() 74 ms
total server time 1,671 ms
1.671 s is not the finish line. It is the point where guessing stops paying. The 5.9 seconds I removed came out of measurement. The remaining 1.597 s has not been decomposed into query, population, transform, serialization and connection timings yet, and until it is I am not assigning it to a cause. Guessing is how a base64 logo survived in a list view in the first place.
What I did verify is that the page still says the same things:
| Output | Before | After |
|---|---|---|
| Activity rows | 8 | 8 |
| Project bars | 4 | 4 |
| Task-status totals | Identical | Identical |
| Invoice totals | Identical | Identical |
| Intended invoice ordering | Preserved | Preserved |
"Identical" means the displayed values and the fields the list reads. The raw invoice documents are deliberately not byte-identical, because removing unused fields was the entire point.
The optimization I turned down
The obvious next move is computing invoice totals inside MongoDB and never loading the documents at all. Here is why I did not:
export function getTotalValue(data: ZodCreateInvoiceSchema): number {
let total = getSubTotalValue(data) - getDiscountValue(data);
for (const detail of data.invoiceDetails.billingDetails) {
if (detail.type === "percentage") {
total += (total * detail.value) / 100;
} else {
total += detail.value;
}
}
return Math.round(total * 100) / 100;
}
The billing details are sequential. Each percentage applies to the running total, so order of operations changes the answer. Porting that into an aggregation pipeline means two implementations of financial arithmetic that have to stay behaviourally identical forever, and the day they drift is the day a client gets an invoice with the wrong number on it.
Faster page, second source of truth for money. No thanks. I made the documents smaller instead.
Reproduce it
Everything below runs against the public repro repo:
dashboard-perf-repro
Minimal reproduction for My dashboard took 7.6 seconds to render fifteen numbers.
An unprojected Invoice.find() carrying a base64 logo in every document
and a JavaScript reduction over thousands of tasks to produce eight rows
Seed it, benchmark it, apply the two fixes, benchmark again.
The application this came from is a private agency platform holding real client invoices. This repo reproduces the query shapes and the failure mode, not the application.
Run it
cp .env.example .env # point MONGODB_URI at a THROWAWAY database
npm install
npm run seed -- --yes # 2,000 invoices, 40 projects
npm run bench # read-only, median of 5 runs
npm run clean -- --yes # removes only documents tagged [perf-seed]
What you should see
| Metric | Before | After |
|---|---|---|
getInvoicesAdmin() |
~6,700 ms | ~1,600 ms |
getDashboardOverview() |
~910 ms | ~74 ms |
| Serialized invoice result | ~22 MB | ~6 MB |
| Overview serialized result | ~2.5 MB | ~2 KB |
Absolute milliseconds…
git clone https://github.com/SoumyaEXE/dashboard-perf-repro
cd dashboard-perf-repro
cp .env.example .env # point MONGODB_URI at a throwaway database
npm install
npm run seed -- --yes # 2,000 invoices, 40 projects
npm run bench # read-only, median of 5
The harness is read-only and calls the same functions the dashboard calls:
const RUNS = 5;
function median(values: number[]) {
const sorted = [...values].sort((a, b) => a - b);
const middle = Math.floor(sorted.length / 2);
return sorted.length % 2
? sorted[middle]
: (sorted[middle - 1] + sorted[middle]) / 2;
}
async function time<T>(fn: () => Promise<T>): Promise<[T, number]> {
const started = performance.now();
const result = await fn();
return [result, performance.now() - started];
}
async function serializedResultBytes(model: Model<any>, select?: string) {
const query = model.find({});
if (select) query.select(select);
const result = await query.lean();
return Buffer.byteLength(JSON.stringify(result), "utf8");
}
const times: number[] = [];
for (let run = 0; run < RUNS; run += 1) {
times.push((await time(getInvoicesAdmin))[1]);
}
console.log(`time: ${median(times).toFixed(1)} ms`);
console.log(
`serialized result: ${(await serializedResultBytes(Invoice) / 1024).toFixed(1)} KB`,
);
console.log("rendered: 4 tiles, 3 donut counts, 6 bars, 4 rows");
One caveat on the tables above: function-level and summary byte totals differ slightly (22,358 vs 24,849 KB) because they are serialized at different scopes. I left both numbers in rather than quietly picking the flattering one. Your absolute milliseconds will differ from mine depending on where your database lives. The ratio is the part that should reproduce.
Three greps, and you probably have this bug
1. Unprojected list queries.
rg "\.find\(\)" --type ts -A 3 | rg -v "select|limit|projection"
Read every hit and ask the only question that matters: does the caller read every field of every document it just asked for?
2. Large embedded fields. Base64 images, rich text bodies, append-only logs, uncapped arrays. One fat field multiplied by a few thousand documents beats any index you were about to add.
3. Input cardinality versus output cardinality. If a function reads N documents and always returns a constant number of rows, the reduction belongs in the database. Scanning 4,806 tasks to display four numbers is not a subtle signal.
And the one that made this whole investigation possible: if the trace shows no database spans, verify the integration before you conclude the database is fast.
FAQ
Why does my Sentry trace show no database spans?
Because the database integration is probably not loaded. @sentry/node ships 17 default integrations and neither mongoIntegration() nor mongooseIntegration() is among them. Check what you actually have with getDefaultIntegrations({}).map(i => i.name) before you conclude the database is fast.
What does a long MongoDB getMore span mean?
getMore is cursor continuation. The driver returns for another batch when a result set does not fit in one. Long getMore time points at result size, not at a missing index. If your predicate is {} there is no index that helps, so look at what fields you are pulling back instead.
Will adding .select() to a Mongoose query break the callers?
Only if a caller reads a field you excluded. Exclude fields at the list layer and keep the full document on the detail route. In this case the list rendered a serial number, a customer name and a total, and had never once rendered the logo it was being handed.
When should a reduction move from JavaScript into a MongoDB aggregation?
When input cardinality and output cardinality diverge. Reading 4,806 tasks to display four numbers means the reduction is happening in the wrong process. $facet lets several independent reductions run in one round trip.
Is a JavaScript comparator that never returns 0 actually a bug?
Yes. Array.prototype.sort requires a consistent comparator. One that never returns 0 gives contradictory answers for equal elements, and the resulting order depends on array length and engine internals. Equal timestamps are common from bulk imports and seed runs.
What actually changed
Three diffs. A .select(), a $facet, and a comparator that returns 0.
None of it is clever. That is the point. The dashboard was not badly written, it was written against four invoices, and it kept being correct right up until the client got busy enough to make it slow. The code did not rot. The business grew, and every unprojected find() in a codebase is a quiet bet that N stays small.
The expensive part was never the fix. It was believing an empty trace.
7.58 s to 1.67 s, same fifteen numbers on screen.
Top comments (12)
What I liked most is that every optimization started with a measurement, not a guess.
The sentence that stuck with me was that the bug only existed at production scale. That’s a great reminder that performance testing with realistic data volumes is just as important as testing functionality. A fast query against an empty database doesn’t tell you much.
haha , man i did much brainstorming lol
😂 It shows. The nice part is that you didn’t stop at “it’s slow”—you kept measuring until you understood why it was slow. That’s the difference between optimization and guesswork. I also liked that you published the benchmark harness instead of just the final numbers. Makes the results much more trustworthy.
too slow 🔥
yeah man!
Such a satisfying optimization breakdown! Creating a tagged seed generator to catch volume issues in staging was a great move—do you now run those benchmark scripts as part of your CI pipeline or automated testing checks?
Ik ci/cd tests not in full depth but yeah learning those Will write a blog on how we can automate cron job using GitHub!
An empty dev database hides production bugs' is a truth every developer learns the hard way. Writing the idempotent benchmark harness first before even attempting a fix was such a clean, systematic approach.
actually sentry saved me that day
This is a strong debugging post because you included measurements, the real causes, and before-and-after results. I especially appreciated the admission that the invalid comparator did not produce the visible failure you expected. That kind of honesty makes the rest more credible. One question, though: 1.67 seconds still seems fairly long for this dashboard. What accounts for most of the remaining time?
Yeah thats true man
hope @sergical will like this!