At Whoz, we build a SaaS platform that helps professional services companies manage their talent staffing. At the heart of our product lies a concept called a worklog — a record of time spent by a user on a given activity. Every consultant, every day, on every project, generates worklogs. It sounds simple. And for years, it was.
Then the numbers caught up with us.
The Problem: A Collection That Never Stops Growing
Our worklog MongoDB collection had reached 530 million documents, representing just over 32 GB of data. And the growth rate was accelerating — not just because we were onboarding more clients, but because users were increasingly splitting their activity into finer-grained entries, generating more worklogs per person per day than ever before.
A worklog document looks roughly like this:
{
"date": "2024-03-15",
"talentId": "abc123",
"workspaceId": "ws456",
"duration": 0.5,
"activityType": "TASK",
"taskId": "task789"
}
Simple enough. But at 530 million of them, even the most routine operations become painful:
- Backup: nearly 1 hour
- Restore: up to 4 hours
- Schema migrations: we hadn't dared run one at full scale yet — and that alone was a warning sign
Every year, the collection grows faster than the year before. The backup and restore windows were becoming operationally risky. We needed to act.
Exploring Our Options
We identified three potential approaches before settling on a solution.
Option 1 — MongoDB Sharding
Sharding is MongoDB's native horizontal scaling mechanism. It distributes a collection across multiple shards, each backed by its own replica set.
On paper, it looked like a match. In practice, we ran into a fundamental mismatch with our actual needs.
Our core issue wasn't query throughput — worklogs from three years ago are rarely queried, and when they are, performance expectations are low. Our issue was operational overhead: backup time, restore time, and the cost of running large batch operations over the full dataset.
Sharding would have solved a different problem. Each shard requires its own replica set — typically three nodes for high availability. Deploying that infrastructure just to store cold historical data felt like enormous overhead for a problem that didn't need distributed compute power. We would have been paying the full cost of a distributed system to solve what was essentially a data locality problem.
We ruled it out.
Option 2 — WiredTiger Tiered Storage
MongoDB's storage engine, WiredTiger, has low-level capabilities that allow it to work with different storage backends. We came across references suggesting it might be possible to offload data files to an S3-compatible bucket — which would have been a perfect fit: keep recent worklogs on fast local storage, push historical data to cheap object storage, all transparently.
It was an exciting idea. We dug into the MongoDB configuration documentation, and reached out to MongoDB support.
The answer was clear: this configuration is not supported by MongoDB. The WiredTiger engine does have internal tiering capabilities in some contexts, but MongoDB does not expose or support them for general use. Dead end.
Option 3 — Application-Level Partitioning
With sharding too heavy and WiredTiger tiering unsupported, we turned to a simpler, more surgical approach: split the collection at the application level.
The concept is straightforward: move historical worklogs into a separate collection, keep recent ones where they are, and expose both transparently to the application. But even within this approach, we had design choices to make.
One collection per year?
The most granular option would have been to create one collection per year — worklog_2021, worklog_2022, worklog_2023, and so on. This would give us very fine control over which data to archive, back up, or eventually drop.
The problem is complexity. Every query touching multiple years would need to union several collections. Adding a new year means updating routing logic. Schema migrations become a multi-collection operation. And MongoDB views support $unionWith, but chaining it across five or six collections starts to feel fragile.
The operational overhead of managing N collections grows with time — and that was precisely the kind of complexity we were trying to avoid.
One collection per time period?
A middle ground would be a sliding window approach: one collection for "recent" data, one for "older" data, with the boundary moving over time. Clean in theory, but it introduces a periodic re-partitioning process that needs to be automated and monitored — moving documents between collections on a schedule, updating the view, handling the transition window carefully.
That's not a trivial piece of infrastructure to get right.
Two collections, one fixed cutoff
We settled on the simplest option that solved our problem: exactly two collections, with a fixed cutoff date.
-
worklog— recent worklogs, actively queried -
worklog_past_part— historical worklogs, rarely accessed
This has an immediate practical benefit: a single $unionWith in a MongoDB view is enough to expose both collections as one. The application doesn't need to know about the split. No routing logic, no dynamic collection names, no complex aggregation pipelines to maintain.
The tradeoff is that the cutoff doesn't move automatically. As years pass, worklog will grow again and we'll need to run the partitioning process again. It's a known limitation — periodic re-partitioning is on our roadmap. But for a first iteration, accepting this constraint in exchange for simplicity was the right call.
Choosing the Cutoff Date
One decision that might seem minor but had real consequences: where do you draw the line?
We looked at the distribution of our worklog volume over time. The pattern was clear — the vast majority of active queries target the current year and the previous one. Data older than that is accessed infrequently, mostly for reporting or auditing purposes.
We settled on January 1st of the previous year as the cutoff. Anything before that date goes into worklog_past_part. This gave us a meaningful reduction in the hot collection size while keeping the most operationally relevant data immediately accessible.
One important consequence of this choice: the cutoff is not a moving window. We don't automatically migrate worklogs every year. This is something we'll need to revisit — a periodic re-partitioning process is on our roadmap. For now, the one-time split already delivers the operational relief we needed.
The Architecture in Three Lines
worklog → recent worklogs (hot)
worklog_past_part → historical worklogs (cold)
view_worklog_all → $unionWith view over both collections
The application reads from view_worklog_all when it needs the full dataset, and writes directly to worklog for new entries. The view is read-only, which comes with its own set of constraints — but we'll cover that in detail in the next articles.
Why Simplicity Won
Looking back, the decision to go with application-level partitioning over sharding or tiered storage came down to one thing: matching the solution to the actual problem.
We didn't need more compute. We didn't need transparent storage tiering. We needed to reduce the operational footprint of a collection that had grown too large — while keeping the application largely unchanged.
Two collections and a view gave us exactly that.
In the next article, we'll look at how the MongoDB $unionWith view let us avoid rewriting hundreds of complex queries — and what the limits of that approach are.
This is part 1 of a 5-part series on MongoDB application-level partitioning at Whoz.
Top comments (0)