The Rewrite We Didn't Do

The upgrade was MongoDB 5.0.17 to 5.0.18: a patch release, the kind you approve without a second look. After it, a recalculation in one customer's environment was taking about two days to finish.

When something breaks that badly, the room fills with big ideas. Maybe the computation engine has reached its limits. Maybe the database needs rethinking. Maybe this is finally the moment for the big rewrite the team had been discussing. Two days feels like a fundamental problem, so it seems to deserve a fundamental fix.

We didn't rewrite anything. The fix turned out to be much smaller, and finding it taught us more than the fix did.

A platform that was already slowing down

The client runs a carbon accounting platform. Enterprise customers bring in their business data, the platform matches that activity to emissions information, and a computation engine calculates their footprint. Customers change an assumption, recalculate and explore the result, so speed isn't a nice-to-have. It's how the product gets used.

By the time I joined, embedded with their engineering team, that loop was getting slow. Change a parameter and you waited minutes. Competitors were offering faster experiences, and the database bill kept climbing, because one way the team had kept up was by moving to bigger instances.

You've probably seen that pattern. Things slow down, someone bumps the instance size, it gets a bit better, and a few months later you're back where you started, paying more for it. Bigger hardware hides a problem without telling you what it is. So when the two-day recalculation arrived, nobody could say where the time was going.

Finding where the time went

There was no end-to-end view of what happened after a customer pressed recalculate. So we built one by hand. We traced requests from the API downwards, matched log lines across services by their correlation IDs, followed the query path into the database, and sat with the engineers who owned each stage of the pipeline to piece the run together.

It was slow, careful work, and it kept narrowing the question. Not "why is the system slow?" but "which step is slow?", then "which queries in that step?", then "what is the database actually doing with them?". At that last level the questions are concrete: is each query using an index or reading the whole collection, and how many documents does it examine for every one it returns?

// Illustrative, not the client's query or schema
db.activities
  .find({ orgId: orgId, period: { $gte: from, $lt: to } })
  .explain("executionStats")

// What to read in the result:
//   winningPlan: IXSCAN (uses an index) or COLLSCAN (reads everything)
//   totalDocsExamined compared with nReturned

The answer was indexes. Some needed rebuilding, and once they were back in place, the same recalculation came down from about two days to about 15 minutes. Other environments saw similar improvements.

I'll be honest about the gap in this story: we never fully pinned down why the upgrade caused it, and I'm not going to guess. What we do know is what fixed it, and how close we came to fixing the wrong thing. If we had trusted the size of the symptom, we could have spent months rebuilding an engine that didn't need rebuilding.

The real fix was the next investigation

Rebuilding the indexes took far less time than finding out that indexes were the problem. That was the part worth fixing.

So we gave the platform the visibility it had been missing. We built an observability layer on OpenTelemetry, chosen so the services weren't tied to any one vendor's monitoring code, with a collector running alongside the services and the signals flowing into the company's existing AWS stack: logs and metrics in CloudWatch, traces in X-Ray, and Grafana dashboards on top.

The tooling wasn't the important decision, though. Adoption was. If instrumenting a service is a project, teams put it off, and the next incident starts from zero again. So we wrapped the setup in a small shared TypeScript library, common-observability, that gave a service traces and metrics with about one line of code:

// Illustrative
import { initObservability } from "@company/common-observability";

initObservability({ serviceName: "calculation-engine" });

After that, the conversation about performance changed. "The system is slow" became "this step dominates the request, and here's the query behind it." The next investigation would start with evidence, not detective work.

A second question: does the data model still fit?

While we were looking closely at how the application used MongoDB, we found something separate from the regression, and it's the part most likely to apply to your system too.

Many of the access patterns were relational. Collections were joined with $lookup again and again across large workloads, and data was duplicated where it didn't need to be. We moved towards read-optimised collections, shaped around the questions the application actually asks, and precomputed results where the workload justified it.

To be clear, $lookup didn't cause the regression, and this isn't a story against MongoDB, which was a perfectly reasonable choice. The problem was the fit between the data model and the way the product had come to be used. That fit drifts as a product grows, and nobody notices until something forces a closer look.

Before you approve a rewrite

Sometimes the rewrite, or the new database, really is the answer. But it should be the conclusion of an investigation, not the response to a symptom. Before you commit to one, I'd want these answered:

1. Can you name the step that's slow, and show the trace? If not, you're sizing a fix for a problem you haven't found.
2. What changed just before it got worse? Deploys, upgrades, data growth, a new customer's data. Check the boring changes first, patch releases included.
3. Have you read the query plans for the hot paths? Index use, and documents examined against documents returned.
4. Is bigger hardware hiding the problem? If the instance size only ever goes up, you're paying for the problem instead of fixing it.
5. Is the case for the big change written down? A proposal engineers can push back on, and a time-boxed proof of concept before the commitment. On this engagement, engineers did push back, and the decisions were better for it.

The better first move is almost always the same: make the work visible, find the specific thing that's wrong, and size the fix to the problem. And then make the next investigation cheaper than this one.

Has a "harmless" upgrade ever bitten you? I'd like to hear the story in the comments.

I also told this story on video, if you'd rather watch:

Originally published on the Tech Swamy blog. This was one thread of a longer engagement, and the full case study covers the rest. If your team is spending more time fighting its system than building on it, that's the work we do at Tech Swamy.