Artur Garcia Costa

Posted on Jan 22 • Originally published at foojay.io

The Cost of Not Knowing MongoDB - Part 2: appV5R0 to appV5R4

#database #mongodb #performance #infrastructure

Application Version 5 Revision 0 and Revision 1: A simple way to use the Bucket Pattern
Application Version 5 Revision 2: Using the Bucket Pattern with the Computed Pattern
Application Version 5 Revision 3: Removing an aggregation pipeline anti-pattern
Application Version 5 Revision 4: Doubling down on the Computed Pattern

Article Introduction

Welcome to the second part of the series, "The Cost of Not Knowing MongoDB". Building upon the foundational optimizations explored in Part 1, this article delves into advanced MongoDB design patterns that can dramatically transform application performance.

In Part 1, we achieved significant improvements through field concatenation, data type optimization, and strategic field naming. However, as identified in the Issues and Improvements of appV4, these approaches represent only the beginning of what's possible with MongoDB schema design. This part introduces a paradigm shift from micro-optimizations to architectural patterns that fundamentally change how data is stored and retrieved.

The journey through appV5R0 to appV5R4 demonstrates the progressive implementation of two powerful MongoDB design patterns: the Bucket Pattern and the Computed Pattern.

Through comprehensive performance analysis and detailed implementation examples, this part reveals both the tremendous potential and important limitations of these advanced patterns, setting the stage for the revolutionary approaches explored in Part 3.

Application Version 5 Revision 0 and Revision 1 (appV5R0 and appV5R1): A simple way to use the `Bucket Pattern`

Introduction

When generating the oneYear totals report, the Get Reports function will need to retrieve an average of 60 documents and, in the worst-case scenario, 365 documents. To access each document, one index entry must be visited, and one disk read operation must be performed.

One way to reduce the number of index entries and documents retrieved to generate the report is to use the Bucket Pattern. According to the MongoDB documentation, "The bucket pattern separates long series of data into distinct objects. Separating large data series into smaller groups can improve query access patterns and simplify application logic."

Looking at our application from the perspective of the Bucket Pattern, so far, we have bucketed our data daily by a user, each document containing the status totals for one user in one day. For the two application versions presented in this section, appV5R0 and appV5R1, we’ll bucket the data by month (appV5R0) and by quarter (appV5R1).

As these are our first implementations using the Bucket Pattern, let’s make it as simple as possible.

For appV5R0, each document groups the events by month and user. Every document will have a field of type array called items to which each event document will be pushed. The event document pushed to the array will have its status field names shorthanded to its first letter, the same way we did in appV3 and appV4, and the date to which the event refers.

The _id field will have a logic similar to the one used in appV4, with the values of key and date concatenated and stored as hexadecimal/binary information. The difference is the date value—instead of being composed by year, month, and day (YYYYMMDD)—will only have year and month (YYYYMM), as we are bucketing the data by month.

For appV5R1, we have almost the same implementation as appV5R0, with the difference being that we’ll bucket the events by quarter, and the date value used to generate the _id field will be composed of year and quarter (YYYYQQ) instead of year and month (YYYYMM).

To build the _id field based on the key and date values for the appV5R0, the following TypeScript function was created:

const buildId = (key: string, date: Date): Buffer => {
  const [YYYY, MM] = date.toISOString().split();

  return Buffer.from(`${key}${YYYY}${MM}`, "hex");
};

To build the _id field based on the key and date values for the appV5R1, the following TypeScript functions were created:

const getQQ = (date: Date): string => {
  const month = Number(getMM(date));

  if (month >= 1 && month <= 3) return "01";
  else if (month >= 4 && month <= 6) return "02";
  else if (month >= 7 && month <= 9) return "03";
  else return "04";
};

const buildId = (key: string, date: Date): Buffer => {
  const [YYYY] = date.toISOString().split("-");
  const QQ = getQQ(date);

  return Buffer.from(`${key}${YYYY}${QQ}`, "hex");
};

This implementation reflects the knowledge of an intermediate MongoDB developer, for using the Bucket Pattern in its simplest form possible

Schema

The application implementation presented above would have the following TypeScript document schema denominated SchemaV5R0:

type SchemaV5R0 = {
  _id: Buffer; // Concatenated user key + time period (YYYYMM or YYYYQQ)
  items: Array<{
    date: Date;
    a?: number; // approved count
    n?: number; // noFunds count
    p?: number; // pending count
    r?: number; // rejected count
  }>;
};

Bulk Upsert

Based on the specification presented, we have the following updateOne operation for each event generated by this application version:

const operation = {
  updateOne: {
    filter: { _id: buildId(event.key, event.date) },
    update: {
      $push: {
        items: {
          date: event.date,
          a: event.approved,
          n: event.noFunds,
          p: event.pending,
          r: event.rejected,
        },
      },
    },
    upsert: true,
  },
};

filter:

Target the document where the _id field matches the concatenated value of key, year, and month/quarter.
The buildId function converts the key+year+month/quarter into a binary format.

update:

Uses $push to append the new event to the items array

upsert:

Ensures a new document is created if no matching document exists.

Get Reports

To fulfill the Get Reports operation, five aggregation pipelines are required, one for each date interval. Each pipeline follows the same structure, differing only in the filtering criteria in the $match stage:

const pipeline = [
  {
    $match: {
      _id: {
        $gte: buildId(request.key, reportStartDate),
        $lte: buildId(request.key, reportEndDate),
      },
    },
  },
  {
    $unwind: {
      path: "$items",
    },
  },
  {
    $match: {
      "items.date": {
        $gte: reportStartDate,
        $lt: reportEndDate,
      },
    },
  },
  {
    $group: {
      _id: null,
      approved: { $sum: "$items.a" },
      noFunds: { $sum: "$items.n" },
      pending: { $sum: "$items.p" },
      rejected: { $sum: "$items.r" },
    },
  },
  { $project: { _id: 0 } },
];

{ $match: {...} }

The _id field is a binary representation of the concatenated key and date values.
The $gte operator specifies the start of the date range, while $lt specifies the end.
The result of buildId contains information by month/quarter, not day, as we need to build the report, so further filtering by day will be necessary

{ $unwind: {...} }

Deconstructs the items array, creating separate documents for each event within the matched buckets.

{ $match: {...} }

Applies precise date filtering at the individual event level, ensuring only events within the exact report date range are included.
It can be seen that we have already filtered by date, but as presented in the explanation of the first stage, we filtered by month/quarter, and to generate the report, we need to filter by day.

{ $group: {...} }

Group the filtered documents into a single result.
The _id field is set to null to group all matching documents from the previous stage together.
Computes the sum of the a, n, p, and r fields using the $sum operator.

$project

Removes the _id field from the final result.

Indexes

These implementations leverage the existing _id index exclusively, eliminating the need for additional compound indexes. The Bucket Pattern's consolidation of multiple events into a single document reduces index size and improves cache efficiency.

Initial Scenario Statistics

Collection Statistics

To evaluate the performance of appV5R0 and appV5R1, we inserted 500 million event documents into the collections using the schema and Bulk Upsert function described earlier. For comparison, the tables below also include statistics from previous comparable application versions:

Collection	Documents	Data Size	Avg. Document Size	Storage Size	Indexes	Index Size
appV4	359,615,279	19.66GB	59B	6.69GB	1	9.50GB
appV5R0	95,350,431	19.19GB	217B	5.06GB	1	2.95GB
appV5R1	33,429,649	15.75GB	506B	4.04GB	1	1.09GB

Event Statistics

To evaluate the storage efficiency per event, the Event Statistics are calculated by dividing the total Data Size and Index Size by the 500 million events.

Collection	Data Size/Event	Index Size/Event	Total Size/Event
appV4	42.2B	20.4B	62.6B
appV5R0	41.2B	6.3B	47.5B
appV5R1	33.8B	2.3B	36.1B

Analyzing the tables above, we can see that going from appV4 to appV5R0, we practically didn’t have improvements when looking at Data Size, but when considering the Index Size, the improvement was quite considerable. The index size for appV5R0 is 69% of the size of appV4.

When considering going from appV4 to appV5R1, the gains are even more impressive. In this case, we reduced the Data Size by 20% and the Index Size by 89%.

Looking at the event stats, we had considerable improvements in the Total Size/events, but what really catches the eye is the improvement in the Index Size/events, which is three times smaller for appV5R0 and nine times shorter for appV5R1.

This huge reduction in the index size is due to the use of the Bucket Pattern, where one document will store data for many events, reducing the total number of documents and, as a consequence, reducing the number of index entries.

With these impressive improvements regarding index size, it’s quite probable that we’ll also see impressive improvements in the application performance. One point of attention in the values presented above is that the index size of the two new versions is smaller than the memory size of the machine running the database, allowing the whole index to be kept in the cache, which is very good from a performance point of view.

Load Test Results

Executing the load test for appV5R0 and appV5R1 and plotting it alongside the results for appV4 and Desired rates, we have the following results for Get Reports and Bulk Upsert.

Get Reports Rates

For the first time, the application is able to reach the target rate for both appV5R0 and appV5R1. appV5R1 nearly reaches all desired rates during the initial test quarter. Both versions demonstrate a clear performance advantage when compared to appV4, and appV5R1 shows significantly better results than appV5R0.

Get Reports Latency

Both new versions have notably lower latencies than appV4, and without degrading in the final half of the test. The appV5R1 reaches a peak latency of 211ms while appV5R0 reaches a peak latency of 530ms.

Bulk Upsert Rates

Both new versions almost reach all the desired rates throughout the test duration, degrading only in the final 20 minutes. It's possible to see that appV5R1 has a better performance than appV5R0.

Bulk Upsert Latency

Even though appV4 is able to reach lower latencies than appV5R1and appV5R0 for some parts of the first half of the test, this lower value is due to the requests being queued instead of the implementation being better. For the final half of the test, the two new versions are clearly a better solution with better values. Both new versions have the same value for peak latency, but the average latency for appV5R1 is lower than appV5R0.

Performance Analysis

The results clearly establish that quarterly bucketing (appV5R1) provides superior performance compared to monthly bucketing (appV5R0), validating the principle that larger bucket sizes can improve performance when appropriately balanced against query complexity.

Issues and Improvements

Because we made this first implementation of the Bucket Pattern as simple as possible, some clear possible optimizations weren’t considered. The main one is how we handle the items array field. In the current implementation, we just push the event documents to it, even when we already have events for a specific day.

A clear optimization here is one that we have been using from appV1 to appV4, where we create just one document per key and date/day, and when we have many events for the same key and date/day, we just increment the status of the document based on the status of the event.

Applying this optimization, we’ll reduce the size of the documents because the array of items will have fewer elements. We’ll also reduce the computational cost of generating the reports because we are pre-computing the status totals by day. This build pattern of pre-computing is quite common in that it has its own name, Computed Pattern.

Application Version 5 Revision 2 (appV5R2): Using the Bucket Pattern with the Computed Pattern

Introduction

As discussed in the Issues and Improvements of appV5R0 and appV5R1, we can use the Computed Pattern to pre-compute the total status by day in the items array field when inserting a new event. This reduces the computation cost of generating the reports and also reduces the document size by having fewer elements in the items array field.

Most of this application version will be similar to the appV5R1, where we bucketed the events by quarter. The only difference will be in the Bulk Upsert operation, where we will update an element in the items array field if an element with the same date of the new event already exists, or insert a new element in items if an element with the same date of the new event doesn’t exist.

The implementation showcases senior-level MongoDB development practices, utilizing advanced aggregation pipeline features within update operations.

Schema

The application implementation presented above would have the following TypeScript document schema denominated SchemaV5R0:

type SchemaV5R0 = {
  _id: Buffer; // User key + quarter (YYYYQQ)
  items: Array<{
    date: Date;
    a?: number; // approved total for the day
    n?: number; // noFunds total for the day
    p?: number; // pending total for the day
    r?: number; // rejected total for the day
  }>;
};

Bulk Upsert

Based on the specification presented, we have the following updateOne operation for each event generated by this application version:

const operation = {
  updateOne: {
    filter: { _id: buildId(event.key, event.date) },
    update: [
      { $set: { result: sumIfItemExists } },
      { $set: { items: returnItemsOrCreateNew } },
      { $unset: ["result"] },
    ],
    upsert: true,
  },
};

This updateOne operation has a similar logic to the one in appV5R1, with the only difference being the update logic.

update:

The complete code for this update logic is quite big, hard to get your head around quickly, and would also make the process of browsing through the article a little cumbersome. Because of that, here we have a pseudocode of it.
Our goal in this update operation is to increment the status of an element in the items array if an element with the same date of the new event already exists, or create a new element if there isn’t one with the same date. It’s not possible to achieve this functionality with the MongoDB Update Operators. The way around it is to use Update with Aggregation Pipeline, which allows a more expressive update statement.
To facilitate the understanding of the logic used in each stage of the aggregation pipeline, a simplified JavaScript version of the functionalities will be provided:

$set: { result: sumIfItemExists }:

Set the field result to the logic of the variable sumIfItemExists. As the name suggests, this logic will iterate through the items array looking for elements with the same date as the event. If there is one, this element will have the status present in the event summed/added to it. As we need a way to keep track of whether an element with the same date of the event was found and the event status was registered, there is an environment boolean variable called found that will keep track of it.
```
 const result = items.reduce(
   (accumulator, element) => {
     if (element.date === event.date) {
       element.a += event.a || 0;
       element.n += event.n || 0;
       element.p += event.p || 0;
       element.r += event.r || 0;

       accumulator.found = true;
     }

     accumulator.items.push(element);

     return accumulator;
   },
   { found: false, items: [] }
 );
```
The result variable/field will be generated using a reduce method on the items array field from the document we want to update. The initial value for the reduce method is an object with the fields found and items. The field accumulator.found has an initial value of false and is responsible for signaling if an element in the reduced execution had the same date as the event we want to register. If there is one element with the same date as the event, element.date === event.date, the status values of the element will be incremented by the status of the event and the accumulator.found field will be set to true, indicating that the event was registered. The accumulator.items array field will have the element of each iteration pushed to it, becoming the new items array field.
$set: { items: returnItemsOrCreateNew }:
Set the field items to the resulting logic of the variable returnItemsOrCreateNew. With a little effort of imagination, the name suggests that the logic present in the variable will return the items field of the previous stage if an element with the same date of the event was found, found == true, or return a new array generated by the concatenation of the items array field of the previous stage with a new array field containing the event element when an element with the same date of the event was not found during the reduced iterations, found == false.
```
 let items = [];

 if (result.found == true) {
   items = result.items;
 } else {
   items = result.items.concat([event]);
 }
```
$unset: ["result"]:
Removes the temporary result field created during the aggregation process.

This sophisticated update operation achieves the equivalent of an "upsert within an array" - functionality that requires careful orchestration of MongoDB's aggregation capabilities.

Get Reports

const pipeline = [
  {
    $match: {
      _id: {
        $gte: buildId(request.key, reportStartDate),
        $lte: buildId(request.key, reportEndDate),
      },
    },
  },
  { $unwind: { path: "$items" } },
  {
    $match: {
      "items.date": {
        $gte: reportStartDate,
        $lt: reportEndDate,
      },
    },
  },
  {
    $group: {
      _id: null,
      approved: { $sum: "$items.a" },
      noFunds: { $sum: "$items.n" },
      pending: { $sum: "$items.p" },
      rejected: { $sum: "$items.r" },
    },
  },
  { $project: { _id: 0 } },
];

This aggregation operation has the same logic as the one in appV5R1.

Indexes

No additional indexes are required, maintaining the single _id index approach established in the appV4 implementation.

Initial Scenario Statistics

Collection Statistics

To evaluate the performance of appV5R2, we inserted 500 million event documents into the collection using the schema and Bulk Upsert function described earlier. For comparison, the tables below also include statistics from previous comparable application versions:

Collection	Documents	Data Size	Avg. Document Size	Storage Size	Indexes	Index Size
appV5R1	33,429,468	15.75GB	506B	4.04GB	1	1.09GB
appV5R2	33,429,649	11.96GB	385B	3.26GB	1	1.16GB

Event Statistics

To evaluate the storage efficiency per event, the Event Statistics are calculated by dividing the total Data Size and Index Size by the 500 million events.

Collection	Data Size/Event	Index Size/Event	Total Size/Event
appV5R1	33.8B	2.3B	36.1B
appV5R2	25.7B	2.5B	28.2B

Analyzing the tables above, we have the expected result presented in the introduction, from appV5R1 to appV5R2. The only noticeable difference is the 24% reduction in the Data Size.

This reduction in the Data Size and Document Size will help in the performance of our application by reducing the time spent reading the document from the disk and the processing cost of decompressing the document from its compressed state.

Load Test Results

Executing the load test for appV5R2 and plotting it alongside the results for appV5R1 and Desired rates, we have the following results for Get Reports and Bulk Upsert.

Get Reports Rates

Both versions have very similar rates, with appV5R2 being slightly better than appV5R1 for the final half of the test.

Get Reports Latency

Both versions have very similar latencies, with appV5R2 reaching lower peak values when compared to appV5R1 for the final half of the test.

Bulk Upsert Rates

Both versions have very similar rates, with appV5R2 being slightly better than appV5R1 for the final 20 minutes of the test, but still not reaching the desired rate.

Bulk Upsert Latency

Both versions have very similar latencies, with appV5R1 reaching lower peak values when compared to appV5R2 for the final 20 minutes of the test.

Performance Analysis

The results show modest improvements in Get Reports performance but slight degradation in Bulk Upsert performance. This outcome reflects the fundamental trade-off inherent in the Computed Pattern: increased write complexity in exchange for simplified read operations.

With writes occurring 4.5 times more frequently than reads, the increased computational cost of the complex aggregation pipeline during writes roughly balances the reduced computational cost during reads. The MongoDB documentation confirms this expectation: "If reads are significantly more common than writes, the computed pattern reduces the frequency of data computation."

In our load testing scenario, writes significantly outnumber reads, making the Computed Pattern's benefits less pronounced. However, this implementation provides a valuable reference architecture for applications with different read/write patterns.

Issues and Improvements

Let’s try to extract more performance from our application by searching for improvements in our current operations. Looking at the aggregation pipeline of Get Reports, we find a very common anti-pattern when fields of type array are involved. This anti-pattern is the $unwind followed by a $match, which happens in the second and third stages of our aggregation pipeline.

This combination of stages can hurt the performance of the aggregation pipeline because we are increasing the number of documents in the pipeline with the $unwind stage to later filter the documents with the $match. In other words, to get to a final state with fewer documents, we’re going through an intermediate state where we increase the number of documents.

In the next application revision, we’ll see how we can achieve the same final result using only one stage and without having an intermediate stage with more documents.

Application Version 5 Revision 3 (appV5R3): Removing an aggregation pipeline anti-pattern

Introduction

As presented in the Issues and Improvements of appV5R2, we have an anti-pattern in the aggregation pipeline of Get Reports that can harm the query performance. This anti-pattern is characterized by a $unwind stage followed by a $match. This combination of stages will first increase the number of documents, $unwind, to later filter them, $match. In a simplified way, to get to a final state, we’re going through a costly intermediary state.

One possible solution around this anti-pattern is to use the $addFields stage with the $filter operator on the items array field. With this combination, we would replace the items array field using the $addFields stage with a new array field generated by the $filter operator in the items array, where we would filter all elements where the date is inside the report's date range.

But, considering our aggregation pipeline with the optimization presented above, there is an even better solution. With the $filter operator, we will loop through all elements in the items field and only compare their dates with the report dates to filter the elements. As the final goal of our aggregation pipeline is to get the status totals of all elements within the report's date range, instead of just looping through the elements in items to filter them, we could already start to calculate the status totals. We can obtain this functionality by using the $reduce operator instead of the $filter.

The implementation represents senior-level MongoDB development practices, showcasing how sophisticated operators can eliminate performance bottlenecks while maintaining code clarity and functionality.

Schema

The application implementation presented above would have the following TypeScript document schema denominated SchemaV5R0:

type SchemaV5R0 = {
  _id: Buffer;
  items: Array<{
    date: Date;
    a?: number;
    n?: number;
    p?: number;
    r?: number;
  }>;
};

Indexes

No additional indexes are required, maintaining the single _id index approach established in the appV4 implementation.

Bulk Upsert

Based on the specification presented, we have the following bulk updateOne operation for each event generated by the application:

const operation = {
  updateOne: {
    filter: { _id: buildId(event.key, event.date) },
    update: [
      { $set: { result: sumIfItemExists } },
      { $set: { items: returnItemsOrCreateNew } },
      { $unset: ["result"] },
    ],
    upsert: true,
  },
};

This updateOne operation has the same logic as the one in appV5R2

Get Reports

const pipeline = [
  { $match: docsFromKeyBetweenDate },
  { $addFields: itemsReduceAccumulator },
  { $group: groupSumStatus },
  { $project: { _id: 0 } },
];

This aggregation operation has a similar logic to the one in appV5R1, with the differences being the change of the second stage from $unwind to $addFields and the change of a variable name in $group stage. The complete code for this aggregation pipeline is quite complicated. Because of that, we will have just a pseudocode for it here.

{ $addFields: itemsReduceAccumulator }:

Adds a new field to the document called totals that will have the status totals.
Uses $reduce to iterate through the items array, applying date filtering and status accumulation in a single operation.

The following JavaScript code is logic equivalent to the real aggregation pipeline code.

 // Equivalent JavaScript logic:
 const totals = items.reduce(
   (accumulator, element) => {
     if (element.date >= reportStartDate && element.date < reportEndDate) {
       accumulator.a += element.a || 0;
       accumulator.n += element.n || 0;
       accumulator.p += element.p || 0;
       accumulator.r += element.r || 0;
     }
     return accumulator;
   },
   { a: 0, n: 0, p: 0, r: 0 }
 );

{ $group: groupSumStatus }:

Group the totals of each document in the pipeline into final status totals using $sum operations.

 const groupSumStatus = {
   _id: null,
   approved: { $sum: "$totals.a" },
   noFunds: { $sum: "$totals.n" },
   pending: { $sum: "$totals.p" },
   rejected: { $sum: "$totals.r" },
 };

Initial Scenario Statistics

Collection Statistics

To evaluate the performance of appV5R3, we inserted 500 million event documents into the collection using the schema and Bulk Upsert function described earlier. For comparison, the tables below also include statistics from previous comparable application versions:

Collection	Documents	Data Size	Avg. Document Size	Storage Size	Indexes	Index Size
appV5R2	33,429,649	11.96GB	385B	3.26GB	1	1.16GB
appV5R3	33,429,492	11.96GB	385B	3.24GB	1	1.11GB

Event Statistics

To evaluate the storage efficiency per event, the Event Statistics are calculated by dividing the total Data Size and Index Size by the 500 million events.

Collection	Data Size/Event	Index Size/Event	Total Size/Event
appV5R2	25.7B	2.5B	28.2B
appV5R3	25.7B	2.4B	28.1B

As the document schema and Bulk Upsert operations for appV5R3 are the same as appV5R2, there is nothing to reason about in this section between the two revisions.

Load Test Results

Executing the load test for appV5R3 and plotting it alongside the results for appV5R2 and Desired rates, we have the following results for Get Reports and Bulk Upsert.

Get Reports Rates

We can say that both versions have a similar performance, with each one reaching better rates throughout the test duration.

Get Reports Latency

It's almost indistinguishable which version has better latency values.

Bulk Upsert Rates

We can say that both versions have a similar performance, with appV5R3 being slightly better at the 20 final minutes of the test.

Bulk Upsert Latency

Even though both version have very similar latency values, we can see that appV5R2 has a slightly lower latency values for the first three quarters of the test, while the appV5R3 has a considerable better latency values for the final final quarter of the test.

Performance Analysis

While the optimized aggregation pipeline is demonstrably more efficient in terms of CPU and memory usage, the performance improvements are minimal. This outcome reveals that the current bottleneck is not a computational overhead, but a disk I/O limitation.

MongoDB Atlas metrics show the IOWAIT metric reaching nearly 15% of CPU usage, indicating that the CPU frequently waits for disk operations to complete. This disk bottleneck will become more apparent in subsequent versions and represents a fundamental infrastructure limitation that cannot be resolved through schema optimization alone.

The relatively modest performance gains demonstrate that optimizing beyond the current bottleneck yields diminishing returns, highlighting the importance of identifying and addressing the primary constraint in any system optimization effort.

Issues and Improvements

We’ve just seen that our implementation's limitation is the disk. To solve that, we have two options: Upgrade the disk where MongoDB stores data or change our implementation to reduce disk usage.

As the goal of this series is to show how much performance we can achieve with the same hardware by modeling how our application stores and reads data from MongoDB, we won’t upgrade the disk. A change in the application modeling for MongoDB will be left for the next article, appV6Rx.

For appV5R4, we will double down on the Computed Pattern and pre-compute the status totals by quarter, not just day. Even though we know it probably won’t provide better performance for things discussed in the "Load test result" of appV5R2, let’s flex our MongoDB and aggregation pipeline knowledge, and also provide a reference code example for the cases where the Computed Pattern is a good fit.

Application Version 5 Revision 4 (appV5R4): Doubling down on the Computed Pattern

Introduction

As presented in the issues and improvements of appV5R3, for this revision, we’ll double down on the Computed Pattern even though we have good evidence that it won’t provide a better performance—but, you know, for science.

We’ll also use the Computed Pattern to pre-compute the status totals for each document. As each document stores the events per quarter and user, our application will have on each document the status totals per quarter and user. These pre-computed totals will be stored in a field called totals.

One point of attention in this implementation is that we are adding a new field to the document, which will also increase the average document size. As seen in the previous revision, appV5R3, our current bottleneck is disk, another indication that this implementation won’t have better performance.

The implementation complexity increases significantly, requiring careful coordination between daily item management and quarterly total maintenance, showcasing the sophisticated techniques employed by senior MongoDB developers.

Schema

The application implementation presented above would have the following TypeScript document schema denominated SchemaV5R1:

type SchemaV5R1 = {
  _id: Buffer;
  totals: {
    a?: number; // Quarter total approved
    n?: number; // Quarter total noFunds
    p?: number; // Quarter total pending
    r?: number; // Quarter total rejected
  };
  items: Array<{
    date: Date;
    a?: number; // Daily total approved
    n?: number; // Daily total noFunds
    p?: number; // Daily total pending
    r?: number; // Daily total rejected
  }>;
};

Indexes

No additional indexes are required, maintaining the single _id index approach established in the appV4 implementation.

Bulk Upsert

Based on the specification presented, we have the following updateOne operation for each event generated by this application version:

const operation = {
  updateOne: {
    filter: { _id: buildId(event.key, event.date) },
    update: [
      { $set: newReportFields }, // Update quarterly totals
      { $set: { result: sumIfItemExists } }, // Process daily items
      { $set: { items: returnItemsOrCreateNew } }, // Update items array
      { $unset: ["result"] }, // Cleanup temporary field
    ],
    upsert: true,
  },
};

This updateOne operation has a similar logic to the one in appV5R3, with the only difference being an extra stage in the update aggregation pipeline logic to pre-compute the document status totals.

update:

To facilitate the understanding of the logic used in the aggregation pipeline, a simplified JavaScript version of the functionalities will be provided:

{ $set: newReportFields }:

Set the field totals to the resulting operation of incrementing each one of the possible status fields by the status provided in the event document.

 if (totals.a != null) {
   totals.a += event.a || 0;
 } else {
   totals.a = event.a || 0;
 }

 if (totals.n != null) {
   totals.n += event.n || 0;
 } else {
   totals.n = event.n || 0;
 }

 if (totals.p != null) {
   totals.p += event.p || 0;
 } else {
   totals.p = event.p || 0;
 }

 if (totals.r != null) {
   totals.r += event.r || 0;
 } else {
   totals.r = event.r || 0;
 }

Get Reports

const pipeline = [
  { $match: docsFromKeyBetweenDate },
  { $addFields: itemsReduceAccumulator },
  { $group: groupSumStatus },
  { $project: { _id: 0 } },
];

This aggregation operation has a similar logic to the one in appV5R3, with the only differences being the implementation in the $addFields stage. The complete code for this aggregation pipeline is quite complicated. Because of that, we will have just a pseudocode for it here.

{ $addFields: itemsReduceAccumulator }:

A similar implementation to the one in appV5R3
The main difference is if the quarter’s date range is within the limits of the report’s date range, we can use the pre-computed totals instead of calculating the value through a $reduce operation.

The following JavaScript code is logic equivalent to the real aggregation pipeline code.

 // Equivalent JavaScript logic:
 let totals;

 if (documentQuarterWithinReportDateRange) {
   // Use pre-computed quarterly totals
   totals = document.totals;
 } else {
   // Fall back to item-level aggregation
   totals = document.items.reduce(
     (accumulator, element) => {
       if (
         element.date >= reportStartDate &&
         element.date < reportEndDate
       ) {
         accumulator.a += element.a || 0;
         accumulator.n += element.n || 0;
         accumulator.p += element.p || 0;
         accumulator.r += element.r || 0;
       }

       return accumulator;
     },
     { a: 0, n: 0, p: 0, r: 0 }
   );
 }

Initial Scenario Statistics

Collection Statistics

To evaluate the performance of appV5R4, we inserted 500 million event documents into the collection using the schema and Bulk Upsert function described earlier. For comparison, the tables below also include statistics from previous comparable application versions:

Collection	Documents	Data Size	Avg. Document Size	Storage Size	Indexes	Index Size
appV5R3	33,429,492	11.96GB	385B	3.24GB	1	1.11GB
appV5R4	33,429,470	12.88GB	414B	3.72GB	1	1.24GB

Event Statistics

To evaluate the storage efficiency per event, the Event Statistics are calculated by dividing the total Data Size and Index Size by the 500 million events.

Collection	Data Size/Event	Index Size/Event	Total Size/Event
appV5R3	25.7B	2.4B	28.1B
appV5R4	27.7B	2.7B	30.4B

As discussed in this revision introduction, the additional totals field on each document in the collection increased the document size and the overall storage size. The Data Size of appV5R4 is 7,7% bigger than appV5R3 and the Total Size/events is 8,2%. Because disk is our limiting factor, the performance of appV5R4 will probably be worse than appV5R3.

Load Test Results

Executing the load test for appV5R4 and plotting it alongside the results for appV5R3 and Desired rates, we have the following results for Get Reports and Bulk Upsert.

Get Reports Rates

It's clear that appV5R4 has worse rate values when compared to appV5R3, only slightly beating the previous version for the first quarter of the test.

Get Reports Latency

For the first two quarters, the test appV5R4 has a lower latency, but for the final two quarters, appV5R3 gets the lead.

Bulk Upsert Rates

Both versions have very similar rate values and also fall short of the desired rate in the final 20 minutes.

Bulk Upsert Latency

The new version, appV5R4, is only able to match the latency values of appV5R3 for the first quarter of the test, falling short for the rest of the three quarters.

Issues and Improvements

As spoiled in the previous Issues and improvements, to improve our application’s performance, we need to change our MongoDB implementation in a way that reduces disk usage. To achieve this, we need to reduce the document size.

You may think it is not possible to reduce our document size and overall collection/index size even more because we are already using just one index, concatenating two fields into one, using shorthand field names, and using the Bucket Pattern. But there is one thing called the Dynamic Schema that can help us.

In the Dynamic Schema, the values of a field become field names. Thus, field names also store data and, as a consequence, reduce the document size. As this pattern will require big changes in our current application schema, we’ll start a new version, appV6Rx, which we’ll play around with in the third part of this series.

Conclusion

That is the end of the second part of the series. We covered Bucket Pattern and Computed Pattern, and the many ways we can use these patterns to model how our application stores its data in MongoDB, and the big performance gains it can provide when used properly.

Here is a quick review of the improvements made between the application versions:

appV4 to appV5R0/appV5R1: This is the simplest possible implementation of the Bucket Pattern, grouping the events by month for appV5R0 and by quarter for appV5R1.
appV5R1 to appV5R2: Instead of just pushing the event document to the items array, we started to pre-compute the status totals by day, using the Computed Pattern.
appV5R2 to appV5R3: This improved the aggregation pipeline for Get Reports, preventing a costly intermediary stage. It didn’t provide performance improvements because our MongoDB instance is currently disk-limited.
appV5R3 to appV5R4: We doubled down on Computed Pattern to pre-calculate the totals field even though we knew the performance wouldn’t be better—but, just for science.

We had noticeable improvements in the version presented in this second part of the series when compared to the versions from the first part of the series. appV0 to appV4. appV5R3 showed the best performance of them all, but it still can’t reach all the desired rates. For the third and final version of this series, our application versions will be developed around the Dynamic Schema Pattern, which will reduce the overall document size and help with the current disk limitation.

For any further questions, you can go to the MongoDB Community Forum, or if you want to build your application using MongoDB, the MongoDB Developer Center has lots of examples and tutorials in many different programming languages.

Table Of Contents

Article Introduction

Application Version 5 Revision 0 and Revision 1 (appV5R0 and appV5R1): A simple way to use the Bucket Pattern

Introduction

Schema

Bulk Upsert

Get Reports

Indexes

Initial Scenario Statistics

Collection Statistics

Event Statistics

Load Test Results

Get Reports Rates

Get Reports Latency

Bulk Upsert Rates

Bulk Upsert Latency

Performance Analysis

Issues and Improvements

Application Version 5 Revision 2 (appV5R2): Using the Bucket Pattern with the Computed Pattern

Introduction

Schema

Bulk Upsert

Get Reports

Indexes

Initial Scenario Statistics

Collection Statistics

Event Statistics

Load Test Results

Get Reports Rates

Get Reports Latency

Bulk Upsert Rates

Bulk Upsert Latency

Performance Analysis

Issues and Improvements

Application Version 5 Revision 3 (appV5R3): Removing an aggregation pipeline anti-pattern

Introduction

Schema

Indexes

Bulk Upsert

Get Reports

Initial Scenario Statistics

Collection Statistics

Event Statistics

Load Test Results

Get Reports Rates

Get Reports Latency

Bulk Upsert Rates

Bulk Upsert Latency

Performance Analysis

Issues and Improvements

Application Version 5 Revision 4 (appV5R4): Doubling down on the Computed Pattern

Introduction

Schema

Indexes

Bulk Upsert

Get Reports

Initial Scenario Statistics

Collection Statistics

Event Statistics

Load Test Results

Get Reports Rates

Get Reports Latency

Bulk Upsert Rates

Bulk Upsert Latency

Issues and Improvements

Conclusion

Application Version 5 Revision 0 and Revision 1 (appV5R0 and appV5R1): A simple way to use the `Bucket Pattern`