DEV Community: Graeme Robinson

Installing MongoDB 7.0 on the Raspberry Pi 5 Using XFS

Graeme Robinson — Sat, 20 Jan 2024 00:57:42 +0000

The Raspberry Pi is awesome, and I've been using it for various fun projects for years. In preparing for this article, I counted six of various vintages doing things around my house, from enabling my garage doors to be opened through Apple Carplay in my car as I approach home, to contributing air traffic control data from aircraft landing and departing nearby Denver International Airport to the FlightAware project.

As a MongoDB fan (and — full disclosure — an employee), I've grown accustomed to the power, flexibility, and ease of use of working with document data model databases in my projects. While the excellent MongoDB Atlas cloud offering is always an option (including its big-enough-to-be-useful free tier), for some projects running on devices like Raspberry Pi, it's nice to have a local database. Until recently, though, the most up-to-date version of MongoDB that could be installed on a Raspberry Pi was version 4.4.18. Later versions — including all 5.x, 6.x, and 7.x releases — require the ARMv8.2-A microarchitecture when running on ARM64 processors, but the processors used by Raspberry Pi on its models up to and including Raspberry Pi 4 only supported ARMv8.0. This means more recently introduced MongoDB features haven't been available on Pi deployments.

With the recent introduction of the rather splendid Raspberry Pi 5, however, ARMv8.2-A is now supported, and all current MongoDB releases can once again be installed on this amazing device. Pi 5 is blazing fast compared with prior versions, too, and it's plenty capable of allowing you to run all kinds of fun projects and experimentations.

In this article, I'll show you how to install MongoDB Community Edition version 7.0.x on a Pi 5 running Pi OS "bookworm" (the latest versions of MongoDB and Pi OS respectively as of January 2024). Much of this is based on an earlier article written by my colleague Mark Smith, but Mark was constrained by the Pi hardware available at the time he wrote his article to install MongoDB 4.4, so we'll bring that advice up to date.

What you will need

A Raspberry Pi 5b: These come in 4GB and 8GB variants. Depending on how ambitious you're planning to be with your projects, I'd recommend an 8GB board, but I've had MongoDB up and running on older Raspberry Pis with as little as 2GB of RAM.
A microSD memory card: This will be the Pi's primary storage. You could probably get a working installation on a 4GB card, but that's not going to leave much room for data or logs. For this article, I'll be using a 128GB U3 card from Samsung. A good quality card from a reputable brand with high read and write speeds is worth the investment.
A USB flash drive: We'll be using this to temporarily boot your Raspberry Pi in order to modify the partitions on your main microSD card before we use it for the first time. You'll need a drive with around 3GB of storage for this and we will be reformatting it so make sure you back up any existing data that's on it first.
A computer to run the Raspberry Pi Imager: Any machine running Windows or Mac OS will work as long as it can read your microSD card and your USB flash drive. I'll be using my main Mac laptop for this.

Now we know what we need, let's review the steps we're going to follow:

1. Flash Raspberry Pi OS onto your microSD card and USB flash drive.
2. Modify the partitions on your microSD card.
3. Create XFS filesystems to run MongoDB on.
4. Install and configure MongoDB.

Let's get started.

Flash Pi OS onto your microSD card and USB flash drive

Raspberry Pi makes a handy tool available for flashing an operating system image onto storage devices. You can download it from the Raspberry Pi website. We'll use this tool to flash an operating system image onto both our microSD card and our USB flash drive

The tool gives you a bunch of options as to which operating system to flash, and any of the 64-bit Pi OS or Ubuntu (desktop or server — not core) options will work. Just be sure not to select a 32-bit version. For this article, I'm going to use Pi OS 64-bit Lite and the instructions assume you do too, but there shouldn't be too many places where you'll need to modify any of the instructions if — for example — you want to run Ubuntu instead. Pi OS Lite does not include a desktop environment, but the image is quite a bit smaller as a result, and I'll be doing everything in a terminal over SSH anyway.

Let's start by flashing your microSD card.

Select your microSD card as the target storage and click Next.

If you've selected one of the Pi OS options to flash, you'll now be asked if you want to apply any OS customizations. I highly recommend you do this. It'll allow you to preconfigure your Pi's hostname, select a username and password (and avoid the pitfall of using a default username and password), apply wireless network settings, set locale details, and enable SSH — including using public key authentication if you wish. If you don't do this now, you'll need to do all those things manually when you first boot your Pi.

Once you've finished configuring the OS customizations you want to apply, start the flashing process. This will take a few minutes to complete. Once it's complete, remove the card from your computer.

IMPORTANT: Do not insert the card into your Pi and boot from it yet.

Repeat these steps to flash Raspberry Pi OS onto your USB flash drive.

Modify the partitions on your microSD card

Linux supports several different filesystems, each with its own strengths and weaknesses. It's possible, common even, to have Linux systems with a mix of partitions using different filesystems with each partition being used for a different purpose or application type.

Without going into detail, the XFS filesystem is particularly well suited to database-type workloads and is MongoDB's recommended filesystem. Pi OS and Ubuntu are both based on Debian Linux and the default filesystem on Debian is ext4. In reality, MongoDB will run just fine on ext4 for any workload you are likely to run on a Raspberry Pi, but you will get a warning message when logging in to the MongoDB shell if it is not running on XFS. If, like me, warning messages bug the heck out of you, the following steps will show you how to set up XFS on your Pi OS image, starting with modifying the partitions on your microSD card. While setting up XFS partitions is not absolutely necessary, it's a useful exercise to go through in case you ever find yourself setting up servers for larger workloads in the future.

When the Raspberry Pi Imager flashed your card, it created two partitions — a ~500MB FAT32 boot partition and a second ~2GB ext4 data partition:

If you do nothing else, the first time you insert your microSD card into your Pi and boot from it, the ext4 data partition will be expanded to use up all remaining space on the card. This would be a problem for us as it would leave no space for the XFS partitions we want to add.

To prevent this, and before we boot our Pi from the microSD card, we're going to first boot it using the USB flash drive and then carry out the following steps:

1. Manually resize the ext4 data partition on our microSD card to the size we want it to be.
2. Using the remaining space on the microSD card, add two additional partitions: one for MongoDB's data files and one for its logs.

By taking these steps, when the Pi first boots from the microSD card, the ext4 file system will be automatically expanded to fill its now larger partition, but the partition itself will not be expanded to take up any additional space on the card.

Note: Over the years, Raspberry Pi has updated and modified the way that the first-boot partition resizing works, and there are a number of articles on the web showing methods to prevent it that are no longer applicable.

Insert your USB flash drive into your Pi, but do not insert your microSD card yet. Power on your Pi.

When it first boots, your Pi will power-cycle a couple of times, but after 20 to 30 seconds, it should be available for you to SSH into (assuming you enabled SSH during the flash process). If you didn't enable SSH or for some reason you can't determine the IP address it was allocated, you may need to initially connect it to a keyboard and monitor to log on directly, enable SSH, and get its IP address. Once you are able to SSH onto the Pi, insert your microSD card into the microSD card reader and use the lsblk command to inspect its disk and partitions:



graeme@pi5Temp:~ $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    1   7.5G  0 disk 
├─sda1        8:1    1   512M  0 part /boot/firmware
└─sda2        8:2    1     7G  0 part /
mmcblk0     179:0    0 119.4G  0 disk 
├─mmcblk0p1 179:1    0   512M  0 part 
└─mmcblk0p2 179:2    0     2G  0 part

The output on my machine shows two devices. The partitions on the sda device are already mounted at /boot/firmware and /, so that's the device the Pi booted from i.e. the USB flash drive. That means the other device — mmcblk0 — is our microSD card and the device we want to partition. The reported size of the devices confirms this.

Running as a superuser, start the parted partitioning utility and select the device to partition:



graeme@pi5Temp:~ $ sudo parted
GNU Parted 3.4
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) select /dev/mmcblk0
Using /dev/mmcblk0

(parted) print                                                            
Model: SD YD4QD (sd/mmc)
Disk /dev/mmcblk0: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  541MB   537MB   primary  fat32        lba
 2      541MB   2739MB  2198MB  primary  ext4

My microSD card is 128GB, so I'm going to expand my ext4 partition to be 30GB, then add a 20GB log partition for MongoDB and use the remaining space for a MongoDB data partition. Let's start by resizing the ext4 partition. Note that it was listed as partition 2:



(parted) resizepart                                                       
Partition number? 2                                                       
End?  [2739MB]? 30.5GB                                                    
(parted) print                                                            
Model: SD YD4QD (sd/mmc)
Disk /dev/mmcblk0: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  541MB   537MB   primary  fat32        lba
 2      541MB   30.5GB  30.0GB  primary  ext4

Now, we'll add the 20GB MongoDB log partition:



(parted) mkpart                                                           
Partition type?  primary/extended? primary                                
File system type?  [ext2]? xfs                                            
Start? 30.5GB                                                             
End? 50.5GB                                                               
(parted) print                                                            
Model: SD YD4QD (sd/mmc)
Disk /dev/mmcblk0: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  541MB   537MB   primary  fat32        lba
 2      541MB   30.5GB  30.0GB  primary  ext4
 3      30.5GB  50.5GB  20.0GB  primary  xfs          lba

Finally, add the MongoDB data partition using the remaining space (~77.5GB):



(parted) mkpart                                                           
Partition type?  primary/extended? primary                                
File system type?  [ext2]? xfs                                            
Start? 50.5GB                                                             
End? 128GB                                                                
(parted) print                                                            
Model: SD YD4QD (sd/mmc)
Disk /dev/mmcblk0: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  541MB   537MB   primary  fat32        lba
 2      541MB   30.5GB  30.0GB  primary  ext4
 3      30.5GB  50.5GB  20.0GB  primary  xfs          lba
 4      50.5GB  128GB   77.7GB  primary  xfs          lba

Close parted and do a final check to make sure everything looks good using lsblk:



(parted) quit                                                             
Information: You may need to update /etc/fstab.

graeme@pi5Temp:~ $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    1   7.5G  0 disk 
├─sda1        8:1    1   512M  0 part /boot/firmware
└─sda2        8:2    1     7G  0 part /
mmcblk0     179:0    0 119.4G  0 disk 
├─mmcblk0p1 179:1    0   512M  0 part 
├─mmcblk0p2 179:2    0  27.9G  0 part 
├─mmcblk0p3 179:3    0  18.6G  0 part 
└─mmcblk0p4 179:4    0  72.3G  0 part

If everything looks good, you can power down your Pi, remove the USB flash drive (we won't be needing it any more) and with your microSD card still in the Pi's microSD card reader, power up the Pi again.

Create XFS filesystems to run MongoDB on

Your Pi will now boot using the microSD card. As before when we booted using the USB flash drive, your Pi will power-cycle a couple of times, but after 20 to 30 seconds it should be available for you to SSH into. Note you may have to delete some entries in the ssh known_hosts file on the machine from which you are connecting to the Pi to avoid SSH connection errors due to the remote host having changed identity. On "UNIX-ish" operating systems like Mac and Linux, this is usually in a hidden directory called .ssh in your home directory. On Windows, it will depend on terminal application you are using. PuTTY, for example, stores known hosts in the registry at \HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys.

Once you are able to SSH onto the Pi, do so and inspect its disk and partitions:



graeme@pi5PIOS:~ $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.4G  0 disk 
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
├─mmcblk0p2 179:2    0  27.9G  0 part /
├─mmcblk0p3 179:3    0  18.6G  0 part 
└─mmcblk0p4 179:4    0  72.3G  0 part

graeme@pi5PIOS:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.8G     0  3.8G   0% /dev
tmpfs           805M  5.1M  800M   1% /run
/dev/mmcblk0p2   28G  1.7G   25G   7% /
tmpfs           4.0G     0  4.0G   0% /dev/shm
tmpfs           5.0M   48K  5.0M   1% /run/lock
/dev/mmcblk0p1  510M   63M  448M  13% /boot/firmware
tmpfs           805M     0  805M   0% /run/user/1000

In particular, we're looking to confirm our ext4 filesystem has been expanded to fill its full 30GB partition, and we have the two empty partitions ready to have XFS filesystems added.

As everything looks good, we can add the XFS filesystems. Start by making sure Pi OS is fully up-to-date by running apt-get update and apt-get upgrade.



graeme@pi5PIOS:~ $ sudo apt-get -y update
graeme@pi5PIOS:~ $ sudo apt-get upgrade

Now, we can install XFS itself:



graeme@pi5PIOS:~ $ sudo apt-get -y install xfsprogs

If you recall the output from lsblk, our two empty partitions are mmcblk0p3 and mmcblk0p4. To create an XFS file system on each, run the following commands:



graeme@pi5PIOS:~ $ sudo mkfs.xfs /dev/mmcblk0p3
graeme@pi5PIOS:~ $ sudo mkfs.xfs /dev/mmcblk0p4

Now we have our two partitions with XFS filesystems, we need to mount them in the right locations within our directory structure. MongoDB writes logs to /var/log/mongodb and data files to /var/lib/mongodb, so let's go ahead and create those:



graeme@pi5PIOS:~ $ sudo mkdir /var/log/mongodb
graeme@pi5PIOS:~ $ sudo mkdir /var/lib/mongodb

We can now mount the filesystems. mmcblk0p3 is our 20GB log partition and mmcblk0p4 is our larger data partition, so the commands to run are:



graeme@pi5PIOS:~ $ sudo mount /dev/mmcblk0p3 /var/log/mongodb
graeme@pi5PIOS:~ $ sudo mount /dev/mmcblk0p4 /var/lib/mongodb

Confirm the mounts were successful:



graeme@pi5PIOS:~ $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mmcblk0     179:0    0 119.4G  0 disk 
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
├─mmcblk0p2 179:2    0  27.9G  0 part /
├─mmcblk0p3 179:3    0  18.6G  0 part /var/log/mongodb
└─mmcblk0p4 179:4    0  72.3G  0 part /var/lib/mongodb

The final step in this stage is to add entries to the fstab file so that the new XFS filesystems are mounted each time your Pi is booted. To do this, we'll use the blkid command to get the PARTUUID of the two XFS partitions:



graeme@pi5PIOS:~ $ sudo blkid /dev/mmcblk0p3
/dev/mmcblk0p3: UUID="7fee4d44-6820-4e90-861f-defd195e5205" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="de9255b2-03"
graeme@pi5PIOS:~ $ echo 'PARTUUID=de9255b2-03 /var/log/mongodb xfs defaults 1 1 ' | sudo tee -a /etc/fstab

graeme@pi5PIOS:~ $ sudo blkid /dev/mmcblk0p4
/dev/mmcblk0p4: UUID="d8a6e88b-fba9-4719-96d4-327829c5cd8f" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="de9255b2-04"
graeme@pi5PIOS:~ $ echo 'PARTUUID=de9255b2-04 /var/lib/mongodb xfs defaults 1 1 ' | sudo tee -a /etc/fstab

Reboot your Pi and do a final check to confirm the XFS file systems have been mounted as expected.

Install and configure MongoDB

With our disk partitioned and XFS file systems in place, we're now ready to install MongoDB.

Do not simply run sudo apt-get install mongoDB. Like much software, the version of MongoDB in the standard repositories for most Linux distributions is very old and most likely no longer supported. The steps in this guide will show you how to install the latest version from MongoDB's own repositories.

At the time of writing, there are pre-built MongoDB packages for both Debian and Ubuntu which, like Pi OS, is Debian-based. However, the Debian packages only support AMD_64 architectures, whereas the Ubuntu packages also support the ARM64 architecture used by the Raspberry Pi. As of January 2024, the latest Pi OS release is based on Debian 12 ("bookworm"), as is Ubuntu 22.04 LTS ("Jammy"), so we'll use the MongoDB packages for that version of Ubuntu. It's worth pointing out that supported platforms and versions will evolve over time, so it's always worth confirming availability in the official MongoDB installation documentation.

To get started, we need to install the MongoDB public GPG key:



curl -fsSL https://pgp.mongodb.com/server-7.0.asc | \
   sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg \
   --dearmor

Next, create an apt source list file for the Ubuntu 22.04 packages:



echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list

Reload the local package database:



sudo apt-get update

And, finally, install MongoDB:



sudo apt-get install -y mongodb-org

Yay — MongoDB is now installed!!! Before we can run it, though, we need to change ownership of the /var/log/mongodb and /var/lib/mongodb directories to a user and group created by the installation process.



sudo chown mongodb:mongodb /var/log/mongodb/
sudo chown mongodb:mongodb /var/lib/mongodb/

Pi OS uses the systemd init system, so to start, stop, or check the run status of MongoDB, we use the systemctl command:



graeme@pi5PIOS:~ $ sudo systemctl start mongod
graeme@pi5PIOS:~ $ sudo systemctl status mongod
graeme@pi5PIOS:~ $ sudo systemctl stop mongod

If you want MongoDB to start automatically each time your Pi boots, use the systemctl command to "enable" it:



graeme@pi5PIOS:~ $ sudo systemctl enable mongod

Start MongoDB and use the systemctl status command to confirm it is running. If everything is good, you should now be able to connect using the monogsh MongoDB shell:



graeme@pi5PIOS:~ $ mongosh
Current Mongosh Log ID: 659f7f5de9e80a30c96e4be1
Connecting to:      mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.1.1
Using MongoDB:      7.0.5
Using Mongosh:      2.1.1

For mongosh info see: https://docs.mongodb.com/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.

------
   The server generated these startup warnings when booting
   2024-01-10T22:38:09.322-07:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
   2024-01-10T22:38:09.322-07:00: vm.max_map_count is too low
------

test>

So far so good, but we want to fix those two warnings that were generated. The first, about access control, is the important one, and we'll handle that in a moment, but first, let's get rid of the max_map_count warning. As with running on XFS, this is also probably unnecessary for any workload we're likely to deploy on a Raspberry Pi, but the MongoDB Production Operations Checklist documentation recommends setting it to 102400, so let's do it.



echo 102400 | sudo tee /proc/sys/vm/max_map_count
#Persist across reboots
echo vm.max_map_count=102400 | sudo tee -a /etc/sysctl.conf

Restart MongoDB using sysctl restart mongod, then connect with mongosh again and verify the max_map_count warning is no longer displayed:



graeme@pi5PIOS:~ $ sudo systemctl restart mongod
graeme@pi5PIOS:~ $ mongosh
Current Mongosh Log ID: 65a011a4cbbd56fd63aafa0e
Connecting to:      mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.1.1
Using MongoDB:      7.0.5
Using Mongosh:      2.1.1

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

------
   The server generated these startup warnings when booting
   2024-01-11T09:04:43.178-07:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
------

test>

Great — we can now deal with the access control warning and, as we said before, this one is important.

By default, MongoDB is installed to allow anyone access to the database without authentication, but only from the device on which it is running. This means that right now, we can only access it from the Raspberry Pi itself. That might be fine if all access is from applications running on the same device, as could well be the case with deployments on something like a Raspberry Pi. In most cases, though, we will want to open up network access to MongoDB, and we will certainly want to restrict access to only authenticated users.

Let's start by adding a user. Using the mongosh shell, run the following command:



use admin
db.createUser( { user: "admin",
    pwd: "YOURPASSWORDGOESHERE",
    roles: [ "userAdminAnyDatabase",
             "dbAdminAnyDatabase",
             "readWriteAnyDatabase"] } )

Replace YOURPASSWORDGOESHERE with an actual password (all the usual recommendations about password complexity and length apply). The roles granted to user admin by this command give them access to all data and allows them to administer other users, so keep that password secure.

Now we'll modify MongoDB's configuration file to tell it to enforce user authentication and to listen for connections on all of the Pi's network adaptors (rather than the default of listening only on it's internal loopback adaptor).

Using your preferred editor, modify /etc/mongod.conf. You need to change the bindIp setting from 127.0.0.1 to 0.0.0.0 (keep port set to 27017), and you need to uncomment the security: entry and add a new value under it, authorization: enabled:



net:
  port: 27017
  bindIp: 0.0.0.0

security:
  authorization: enabled

Note this is a YAML file, so the indentation is important.

Restart MongoDB once again and connect with mongosh — now, there should be no warnings:



graeme@pi5PIOS:~ $ sudo systemctl restart mongod
graeme@pi5PIOS:~ $ mongosh
Current Mongosh Log ID: 65a01ab52a2e9d6bb7c7a3f4
Connecting to:      mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.1.1
Using MongoDB:      7.0.5
Using Mongosh:      2.1.1

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

test>

It's worth pointing out at this point that what we've done is open up access to MongoDB from any device that can establish a network connection to your Raspberry Pi. I'm assuming you are running on a home or office network with a firewall between you and the open internet. Pi OS itself does not have any firewall protections enabled by default. If you want to add them, ufw is a pretty good option. Suffice it to say, it's almost never a good idea to expose a database server to the open internet, and certainly not one without authentication enabled. You will also want to enable TLS encryption if you're deploying on, or enabling access from, anything other than your home network. Check out the MongoDB security checklist for more security advice.

To test network connectivity and authentication, move to another computer with mongosh installed (I'm using my Mac laptop), and attempt to connect to MongoDB on your Pi. We'll do this twice: once using an anonymous login and once using the credentials for the admin user we created. On both attempts, we'll attempt to run a command to list all databases and verify the anonymous connection is blocked from doing so:



graeme@M-YYV2WV29DD ~ % mongosh --host 10.0.0.191
Current Mongosh Log ID: 65a01c72db8f065eddfc3e9b
Connecting to:      mongodb://10.0.0.191:27017/?directConnection=true&appName=mongosh+2.1.1
Using MongoDB:      7.0.5
Using Mongosh:      2.1.1

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

test> db.adminCommand({listDatabases: 1})
MongoServerError: Command listDatabases requires authentication



graeme@M-YYV2WV29DD ~ % mongosh --host 10.0.0.191 -u "admin"
Enter password: ********
Current Mongosh Log ID: 65a01e554a71d545c503bc2e
Connecting to:      mongodb://<credentials>@10.0.0.191:27017/?directConnection=true&appName=mongosh+2.1.1
Using MongoDB:      7.0.5
Using Mongosh:      2.1.1

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

test> db.adminCommand({listDatabases: 1})
{
  databases: [
    { name: 'admin', sizeOnDisk: Long('135168'), empty: false },
    { name: 'config', sizeOnDisk: Long('110592'), empty: false },
    { name: 'local', sizeOnDisk: Long('73728'), empty: false }
  ],
  totalSize: Long('319488'),
  totalSizeMb: Long('0'),
  ok: 1
}

Congratulations — you now have a running deployment of the latest version of MongoDB on your Raspberry Pi. Want to use time-series and clustered collections for your home IoT project? We got ya. Want to keep your data secure using state-of-the-art queryable encryption? We got ya too. So go, have fun, create something amazing, and tell us about it in the comments.

MongoDB Design Reviews: how applying schema design best practices resulted in a 60x performance improvement

Graeme Robinson — Tue, 29 Aug 2023 13:19:44 +0000

The transition from working with legacy relational database systems to NoSQL databases such as MongoDB requires developers to change the way they model and represent data if they are to realize the full benefits of making the switch.

Whilst MongoDB has sometimes been referred to — incorrectly — as “schemaless,” the reality is that schema design is every bit as important in MongoDB as it is in any database system, and the choices you make as a schema designer and data modeler will make or break the performance of your application in MongoDB as much, if not moreso, than they will in any traditional RDBMS.

As a developer advocate on the MongoDB Strategic Accounts team, I assist customer development teams who are transitioning existing workloads or creating new workloads in MongoDB, by providing tailored data modeling sessions, also known as “design reviews.” During these sessions, we review the customer’s specific workload and provide feedback, advice, and suggestions on how best to model the data in their workload for optimal performance.

At the end of the design review, customers will have a framework schema design tailored to their workload based on best practice schema design patterns developed by MongoDB over the years.

In this article, we discuss a specific design review and show how, through a combination of schema design and query optimizations, the session resulted in the customer seeing a 60x improvement in the performance of one of their aggregation pipelines and allowed their application to meet its SLA targets.

The customer portfolio activity application

A few months ago, we received a request from a customer in the financial services industry to carry out a design review of an application they were building in MongoDB Atlas. Among other things, the application was designed to provide regional account managers with aggregated performance data for each stock in a customer’s portfolio over a requested time period, in a given region.

When the customer contacted us, the aggregation pipeline they had designed to generate the data was taking between 20 and 40 seconds to complete where the application SLA called for a sub two-second response time. The database design as far as this aggregation was concerned was relatively simple, consisting of only two collections.

The documents in the first collection contained customer information, including the region to which the customer belonged, and an array of stock symbols with one entry for each stock in their portfolio.



{
  "_id": "US4246774937",
  "region": "US",
  "firstname": "Jack",
  "lastname": "Bateman",
  "portfolio": [
    "NMAI",
    "CALA",
    "MNA"
  ]
}

The documents in the second collection contained stock data covering a minute of trading activity, with one document being produced for each stock. The information in these documents included the stock symbol, the volume of shares of that stock traded, the opening price, high price, low price, closing price, and start and finish timestamps for the covered minute.



{
  "_id": {
    "$oid": "63e15e9ad0c75e43cd1831db"
  },
  "symbol": "CALA",
  "volume": 10464,
  "opening": 0.14,
  "high": 0.14,
  "low": 0.14,
  "closing": 0.14,
  "start": {
    "$date": "2023-02-06T19:54:00.000Z"
  },
  "end": {
    "$date": "2023-02-06T19:55:00.000Z"
  }
}

The aggregation pipeline was being executed against the customer collection, with its output designed to provide the following:

“For each customer in a selected region, provide the opening price, trading volume, high price, low price, and closing price for a specified time period for each stock in that customer’s portfolio.”

To provide this output, the aggregation pipeline had been defined with five stages:

An initial $match stage that selected only documents for customers in the desired region
An $unwind stage that would duplicate the selected customer documents once for each stock in the customer’s portfolio — i.e. if the customer had 10 stocks in their portfolio, this stage would create 10 documents — one for each stock — replacing the original document that had all 10 stocks listed in an array
A $lookup stage that for each document created by the prior $unwind stage, would perform a second aggregation pipeline against the stock data collection, taking the stock symbol and date range as input and returning the aggregated data for that stock
A second $unwind stage to flatten the array created by the prior $lookup stage (that, in this case, always only contained one entry) down to an embedded object
A final $group stage to recombine the documents for each customer back into a single document

The output document for a customer with two stocks in their portfolio would look like this:



{
  "_id": "US1438229432",
  "region": "WEST",
  "firstname": "Otto",
  "lastname": "Cast",
  "portfolio": [
    "ISR",
    "CTS",
  ],
  "stockActivity": [
    {
      "opening": 0.42,
      "high": 0.42,
      "low": 0.4003,
      "closing": 0.4196,
      "volume": {
        "$numberLong": "40611"
      },
      "symbol": "ISR"
    },
    {
      "opening": 42.7,
      "high": 42.98,
      "low": 41.62,
      "closing": 42.93,
      "volume": {
        "$numberLong": "45294"
      },
      "symbol": "CTS"
    }
  ]
}

An examination of the explain plan for the pipeline showed that the two $match stages — one in the main pipeline and one in the sub-pipeline within the $lookup stage — were both correctly using the indexes set up to support the pipeline. This eliminated missing or incorrectly defined indexes — one of the most common sources of performance issues we see in MongoDB — as the source of the issues in this case.

Assessing the workload

Whenever we design a data model or schema for MongoDB, best practice calls for starting by understanding and quantifying the target workload. At the start of the design review, we were able to ascertain the following:

The application contained data on approximately 10,000 customers, evenly distributed across six regions within the United States.
Each customer had, on average, 10 stocks listed in their portfolio, with the highest number of stocks for any one customer being 20.
Activity data for approximately 16,000 stocks was being tracked.
Approximately two million stock activity records were being generated daily. Not every stock produced an update every minute, and data was only being collected for the eight-hour US market day, Monday through Friday.
During market hours, approximately 4,200 new stock updates were received each minute.
Stock activity data for the last four complete quarters, plus the current quarter to-date, was being maintained. This translated to approximately 650 million stock activity documents taking up around 35 GB of storage.
Data more than four complete quarters old was being purged from the system, so data volumes were pretty stable.
The data produced by the aggregation pipeline was being used by managers to produce end-of-day, end-of-month, and end-of-quarter reports. Data was not being requested for periods of less than a day, and the data for the current day only became available after the US markets closed.
Reports were being generated by managers, on average, 150 times per day.
The reports were not being used to make real-time trading decisions.

The application was being run on a MongoDB Atlas M40 three-node cluster, and the metrics above indicated that this workload should not be excessively sized for the cluster, with the only question mark being whether, at 16GB, there was sufficient memory to maintain an adequately sized working set of data in memory to handle the pipeline requests without data swapping taking place.

With an understanding of the nature and scale of the workload established, we then turned our attention to the structure of the aggregation pipeline.

Assessing the aggregation pipeline

The aggregation pipeline, as originally designed by the application development team, looked as follows:



[
  {$match: {region: "WEST"}},
  {$unwind:{path: "$portfolio"}},
  {$lookup:{
    from: "stockData",
    let: {
      symbol: "$portfolio",
      start: ISODate("2022-11-07T00:00:00.000+00:00"),
      end: ISODate("2022-11-08T00:00:00.000+00:00")
    },
    pipeline: [
      {$match:{
        $expr:{ $and: [
          {$eq: ["$symbol", "$$symbol"]},
          {$gte: ["$start", "$$start"]},
          {$lt: ["$end", "$$end"]},
        ]}
      },
      {$group:{
        _id: "$symbol",
        opening: {$first: "$opening"},
        high: {$max: "$high"},
        low: {$min: "$low"},
        closing: {$last: "$closing"},
        volume: {$sum: "$volume"}
      }},
      {$set:{
        "symbol": "$_id",
        "_id": "$$REMOVE"
      }}
    ],
    as: "stockData"
  }},
  {$unwind: {path: "$stockData"}},
  {$group:{ 
      _id: "$_id",
      region:{$first: "$region"},
      firstname:{$first: "$firstname"},
      lastname:{$first: "$lastname"},
      portfolio:{$addToSet: "$portfolio"},
      stockActivity:{$push: "$stockData"}
  }}
]

On running a test query retrieving data for a one-day window of trading activity for all customers in the “WEST” region, we saw a response time of just under 29 seconds.

There were two items in this pipeline that immediately caught our attention.

Firstly, the pipeline used an $unwind stage to allow the subsequent $lookup stage to be run once for each stock in each customer’s portfolio. In fact, this $unwind, and its subsequent reconstruction of the data in the final $group stage, was unnecessary. If an array is passed to a $lookup stage as the localfield value, the $lookup will automatically be run for each entry in the array. Refactoring the pipeline to take this approach reduced it to two stages: the initial $match stage and the subsequent $lookup stage. The revised pipeline looked like this:



[ 
 {$match:{region: "WEST"}},
 {$lookup:{
    from: "stockData",
    localField: "portfolio",
    foreignField: "symbol",
    let: {
       start: ISODate(
          "2022-11-07T00:00:00.000+00:00"
       ),
       end: ISODate(
          "2022-11-08T00:00:00.000+00:00"
       ),
    },
    pipeline: [
      {$match:{
        $expr:{ $and: [
          {$eq: ["$symbol", "$$symbol"]},
          {$gte: ["$start", "$$start"]},
          {$lt: ["$end", "$$end"]},
        ]}
      },
      {$group:{
        _id: "$symbol",
        opening: {$first: "$opening"},
        high: {$max: "$high"},
        low: {$min: "$low"},
        closing: {$last: "$closing"},
        volume: {$sum: "$volume"}
      }},
      {$set:{
        "symbol": "$_id",
        "_id": "$$REMOVE"
      }}
    ],
    as: "stockActivity",
 }}
]

Eliminating the $group stage in particular proved beneficial, and re-running our test query gave a response time of 19 seconds. This was a significant improvement, but still well short of the target sub two-second response time.

The second issue we saw in the pipeline was the use of the $lookup stage. $lookup essentially carries out the equivalent of what would be a left outer join in a relational database. Joins in any database system — relational or NoSQL — are computationally expensive operations. One of the key benefits of the document model used by MongoDB is its ability to allow us to avoid joins through the use of embedding and hierarchical documents. However, in this case, the application development team had correctly identified that embedding the stock activity documents in each customer’s document would lead to excessively sized documents and huge arrays — both MongoDB anti-patterns. Data denormalization and some level of duplication to improve query performance in MongoDB is often encouraged. However, in this workload, with write operations outnumbering read operations by a considerable margin, the extent and subsequent update cost of duplicating the stock activity data into the customer documents was determined to be a poor trade-off.

Although embedding the stock activity documents in the customer documents was ruled out as an approach, examining exactly what was happening with the $lookup stage was revealing in terms of understanding why the pipeline was taking as long as it was to execute. For example, running the pipeline to generate data for one calendar quarter for all customers in the WEST region resulted in the following metrics:

The initial $match stage returned 1,725 customer documents.
With an average of 10 stocks in each customer’s portfolio, the subsequent $unwind stage expanded the number of documents in the pipeline to 18,214.
The $lookup stage was then executed once for each of those 18,214 records.
For each execution of the lookup stage, one calendar quarter of data for a given stock needed to be aggregated. This resulted in approximately 25,000 one-minute stock activity records needing to be aggregated during each of the 18,214 executions of the $lookup sub-pipeline.
As the same stock could appear in multiple customers’ portfolios, in many cases, the $lookup sub-pipeline was being executed for the same stock multiple times.

During actual execution, the MongoDB aggregation engine would be able to apply some optimizations - in particular caching results of prior runs of the $lookup stage allowing them to be reused by subsequent runs supplying the same parameters - so the overall performance was not quite as high as the metrics would at first suggest, but it was still a lot of work being executed, some of which was duplicative.

With this understanding, the next stage in our design review was to look to see how schema design patterns could be applied to optimize the pipeline performance.

Applying schema design patterns — the computed pattern

The first issue we looked to address was the number of stock activity documents being aggregated by each execution of the $lookup sub-pipeline.

Stock activity documents were being written to the database on a minute-by-minute basis, but during our workload assessment at the start of the design review, we determined that users were never querying at anything less than per-day granularity. With this in mind, we decided to investigate if the computed design pattern could be applied.

The computed design pattern emphasizes pre-calculating and saving commonly requested data so that the same calculation is not repeated each time the data is requested. In our case, the pipeline was repeatedly aggregating the same per-minute data into daily, monthly, quarterly, or yearly totals. So, we decided to see what impact pre-calculating those totals and storing them in a new collection, and having the $lookup pipeline access those pre-calculated values, would have.

To do this, we suggested adding the following processes to the application:

At the end of each US trading session, the per-minute documents for each stock would be aggregated to give a “daily” document with the trading volume and starting, closing, high, and low prices for each stock. These “daily” documents would be stored in a new collection and the per-minute documents deleted from the original collection, meaning it never contained more than one day’s worth of per-minute documents.

At the start of each month, the “daily” documents for each stock would be aggregated to give a “monthly” document for each stock. The “monthly” documents would be stored in the same new collection as the daily documents.
At the start of each quarter, the “monthly” documents for each stock would be aggregated to give a “quarterly” document for each stock. The “quarterly” documents would also be stored in the same new collection as the daily and monthly documents.
In order to differentiate between the types of documents in the new collection, they would include a “type” field with a value of either “D”, “M”, or “Q” for “daily”, “monthly”, or “quarterly” respectively. This, along with the stock symbol and the starting date for the period covered, would form a compound _id value for each document.
Data for the four prior complete quarters plus the current quarter to date would be maintained. At the start of each new quarter, data for the oldest quarter would be deleted, preventing the size of the collection and its associated indexes from growing indefinitely.

An example of the new document design would look like this:



{
  "_id": {
    "symbol": "MDB",
    "time": {"$date": "2022-11-06T00:00:00.000Z"},
    "type": "D"
  },
  "closing": 218.51,
  "high": 218.9599,
  "low": 216.0501,
  "opening": 218.7,
  "volume": 336998
}

With these changes in place, it would be possible to form a query for any range of days, months, or quarters in the dataset. The metrics for the new collection designs were encouraging too. Tracking data for the same 16,000 stocks as before:

The new collection would contain roughly 5.4 million documents at most (i.e., just before a quarter end). This compared with roughly 640 million per-minute documents in the original stock activity collection.
The original stock activity collection, which would still be used to gather the per-minute updates, would only ever hold a maximum of two million documents (for the current day’s updates), rather than the 640 million documents previously.
A “worst case” query requesting data for the complete date range on the day prior to a quarter end would require 30 “daily” documents, two “monthly” documents, and four “quarterly” documents — a total of 36 documents — to be aggregated per stock. Compare this with the approximately 154,000 documents that would need to be aggregated per stock to do the same calculation using per-minute documents.

Modifying the pipeline to use this new structure, it now looked as follows:



[
  {$match:{region: "WEST"}},
  {$lookup:{
    from: "stockDataEnhanced",
    localField: "portfolio",
    foreignField: "_id.symbol",
    pipeline: [
      {$match: {
        $and: [
          {"_id.type": "D"},
          {"_id.time": ISODate("2022-11-07T00:00:00.000+00:00")}
        ],
      },
      {$sort:{"_id.time": 1}},
      {$group:{
        _id: "$symbol",
        opening: {$first: "$opening"},
        high: {$max: "$high"},
        low: {$min: "$low"},
        closing: {$last: "$closing"},
        volume: {$sum: "$volume"}
      }},
      {$set:{
        symbol: "$_id",
        "_id": "$$REMOVE"
      }}
    ],
    as: "stockActivity",
  }
]

Executing the revised pipeline gave a response time of 1800 ms — below our two-second target SLA! However, the design review team felt there were further improvements that could be made.

Applying schema design patterns — the extended reference and single collection patterns

Having resolved the problem of the large numbers of documents needing to be aggregated with each execution of the $lookup stage sub-pipeline, and simultaneously reducing the overall size of the database by almost 98%, we turned our attention to the other significant issue in the original pipeline: that in any given execution, the same aggregation calculations could be carried out multiple times for any given stock.

To address this, we revisited our understanding of the relationships in the data and how we wanted to represent those relationships — a step that, along with quantifying our workload and assessing its access patterns, and applying best practice schema design patterns — forms the basis of our approach to data modeling in MongoDB.

In this case, we were starting our pipeline with a $match stage on the customer documents to find all the customers in a given region because we needed their portfolio information and that’s where it was stored — a seemingly logical design.
However, looking at the way the data was being accessed, if we could add and maintain a list of regions for which a given stock would need to be calculated and add that to each pre-computed stock activity document for that stock, we could then initiate our pipeline against the pre-computed stock activity collection and, importantly, only aggregate data for each required stock once. Determining which regions a stock needed to be associated with would involve calculating the set of regions to which all customers holding that stock in their portfolio belonged.

Embedding the region data in the pre-computed stock activity documents like this is a variant of the extended reference schema design pattern. This pattern emphasizes embedding a subset of fields from related documents in a parent document so that all the related data can be retrieved with a single query and avoiding the use of a $lookup based join. Rather than embed the entire child documents, the pattern encourages embedding only those fields needed to satisfy a query predicate or those included in the query return. This helps keep the overall size of the parent document within reasonable limits.

Using the extended reference pattern comes at the cost of needing to propagate changes to child data to multiple parent documents, so the pattern is particularly useful when the referenced data does not change often. In highly normalized RDBMS designs, it is not uncommon to see joins being repeatedly performed to lookup tables that contain values that haven’t changed, in some cases, in decades.

In our workload, the pattern imposed the cost of possible updates to the regions each stock was associated with whenever a customer’s portfolio changed. But as this happened relatively rarely, the cost was deemed acceptable given the potential query performance improvements.

Applying these changes, the pre-computed stock activity documents now looked like this:



{
  "_id": {
    "symbol": "MDB",
    "time": {"$date": "2022-11-06T00:00:00.000Z"},
    "type": "D"
  },
  "closing": 218.51,
  "high": 218.9599,
  "low": 216.0501,
  "opening": 218.7,
  "regions": [
    {"region": "WEST"},
    {"region": "NORTH_EAST"},
    {"region": "CENTRAL"}
  ],
  "volume": 336998
}

The next question was, if we started the pipeline from the stock activity data, how would we then tie this back to the customer data? The application team’s first thought was to have another $lookup stage. But on further review, we suggested they instead utilize the polymorphic nature of MongoDB collections and store the customer documents within the same collection as the pre-computed stock activity data using a single-collection schema design pattern.

Single-collection patterns emphasize storing documents of varying types, but that are related and accessed together, within the same collection. By using a common set of attributes across all of the document types in the collection, and indexing those attributes appropriately, a single database search can retrieve all the related documents with a single database operation, saving on network round-trips and marshaling/de-marshaling overheads.

In our case, we elected to add customer documents to the stock activity collection using the following document shape:



{
  "_id": {
    "customerID": "US4246774937",
    "symbol": "NMAI",
    "type": "C"
  },
  "firstname": "Jack",
  "lastname": "Bateman",
  "portfolio": [
    "NMAI",
    "PUBM",
    "MNA"
  ],
  "regions": [
    {
      "region": "US"
    }
  ]
}

The key things to note with these documents are the “symbol” and “type” fields within the compound “_id” object, and the customer’s region being moved to a regions array. This made the field names and data types consistent with the stock activity daily, monthly, and quarterly documents. Note also we added one customer document for each stock within the customer’s portfolio. This allowed effective indexing of the data in the collection at the expense of some duplication of data. However, as the customer data changed relatively infrequently, this was deemed acceptable.

With these changes in place, we could now define a pipeline that both avoided repeated stock aggregation calculations, and avoided the use of an expensive $lookup stage. The stages in the revised pipeline were:

A $match stage to find all documents where the region array included the target region and either the the “_id.type” field was “C” for customer, or the combination of “_id.type” and “_id.time” indicated this was a stock activity document for the time period we were calculating. (The match stage query could be updated to include whatever combination of quarterly, monthly, and daily activity documents needed to cover any time period requested.)
A $group stage to aggregate the stock activity data for each stock, and also build an array of customer documents for each stock. As part of the data aggregation, an array of opening and closing prices from each of the individual activity documents being aggregated was built, relying on the index on the collection to ensure these were added to each array in chronological order.
A $set stage to replace the opening price array and closing price array built by the prior $group stage with the first and last entry in each array respectively to give the overall opening and closing price for the requested time period for each stock.
Finally, an $unwind and $group stage combination to reorganize the data by customer rather than stock, and to shape it to our desired output design.

The revised pipeline, run against the pre-calculated stock activity collection, now looked like this:



[
  $match: {
    $and: [
      {"regions.region": "WEST"},
      {$or:[
        {"_id.type": "customer"},
        {
          "_id.type": "day",
          "_id.time": ISODate("2023-02-07T00:00:00.000+00:00")
        }
      ]}
    ]
  }}
  {$group:{
    _id: "$_id.symbol",
    volume: {$sum: "$volume"},
    opening: {$push: "$opening"},
    high: {$max: "$high"},
    low: {$min: "$low"},
    closing: {$push: "$closing"},
    customers: {
      $addToSet: {
        $cond: {
          if: {$eq: ["$_id.type", "customer"]},
          then: "$$ROOT",
          else: "$$REMOVE"
        }
      }
    }
  }},
  {$set:{
    closing: {$last: "$closing"},
    opening: {$first: "$opening"}
  }},
  {$unwind: {path: "$customers"}},
  {$group:{
    _id: "$customers._id.customerID",
    region: {
      $first: {
        $getField: {
          field: "region",
          input: {$arrayElemAt: ["$customers.regions",0]}
        }
      }
    },
    firstname: {$first: "$customers.firstname"},
    lastname: {$first: "$customers.lastname"},
    portfolio: {$first: "$customers.portfolio"},
    stockActivity: { $addToSet: {
      symbol: "$_id",
      volume: "$volume",
      opening: "$opening",
      high: "$high",
      low: "$low",
      closing: "$closing",
    }}
  }}
]

A final test execution of this version of the pipeline gave a response time of 377 ms — more than four times faster than the application target response time.

Conclusion

As is our normal practice for design reviews, we had provided the application development team with a list of questions regarding the nature and sizing of their workload ahead of time, and they came to the session well prepared.

With this preparation completed, the design review session itself lasted around an hour, during which we went through our standard data modeling process of:

Assessing the workload and its access patterns.
Reviewing the relationships in the data.
Applying best practice schema design patterns.

By the end of session, we had collectively managed to improve the performance of the aggregation pipeline by 60x compared with its original design, handily exceeding the application target SLA, whilst simultaneously significantly reducing the storage requirements of the application. Everyone agreed it had been a very productive session.

Think your team could benefit from a design review session with data modeling experts from MongoDB? Please reach out to your account representative to find out more about booking a session with our data modeling experts, either virtually or at select MongoDB .local events in your city!

If you would like to learn more about MongoDB data modeling and aggregation pipelines, we recommend the following resources:

Daniel Coupal and Ken Alger’s excellent series of blog posts on MongoDB schema patterns
Daniel Coupal and Lauren Schaefer’s equally excellent series of blog posts on MongoDB anti-patterns
Paul Done’s ebook, Practical MongoDB Aggregations
MongoDB University Course, M320 - MongoDB Data Modeling