Upgrade from MongoDB 8.0 to 9.0 Without Surprises: A Realistic Walkthrough with Docker

A new major version of your database is exciting right up until you have to run it in production. The release notes are long, the upgrade procedure has more than one step, and the question that actually matters is hard to answer from a document: what happens to my application when I do this?

MongoDB 9.0 is generally available, and it brings performance gains over 8.0, including up to 35% faster find-one queries and up to 30% faster update-one queries. Those figures are "up to" numbers that depend on hardware, workload, and data, so the best way to find out what 9.0 does for you is to try it on your own workload, and the first step to doing that is a safe upgrade path.

In this tutorial, we'll walk through a realistic upgrade of a three-member replica set from MongoDB 8.0 to 9.0 using Docker. We'll check the health of the deployment, take a backup and verify it restores, make a go/no-go decision, perform a rolling upgrade one member at a time, and then raise the feature compatibility version. Along the way, we'll look at what changes for an application and how to measure your own workload with $queryStats.

The Prerequisites

Prior to starting this tutorial, you'll need a few things in place:

• Docker with Compose v2, and at least 2 GB of memory available to it
• curl for downloading the sample dataset

The expectation is that you're comfortable running commands in a terminal. You don't need Node.js installed on your machine because the sample application runs inside a container.

It's worth being clear about why Docker is in the picture. We're using it for the sake of this tutorial, because it lets us create a three-member replica set on a single machine, reset it, and run through the whole process as many times as we want. Docker is not a requirement for upgrading to MongoDB 9.0. The same procedure applies to a self-managed replica set running on virtual machines or bare metal, where you'd replace the MongoDB packages or binaries on each member instead of swapping a container image. If you run on MongoDB Atlas, the platform performs the rolling upgrade for you, and we'll touch on what that looks like later.

We'll use the sample_mflix dataset that you may already know from MongoDB Atlas, but since we're not using Atlas for the main walkthrough, we'll restore it to our own deployment. Start by creating a project directory with a place for each part of the project:

mkdir mongodb-9-upgrade
cd mongodb-9-upgrade
mkdir data docker backup scripts
mkdir -p app/src

The above commands create the project and the directories we'll fill in as we go. We'll end up with the following shape:

.
├── app/
│   ├── package.json
│   └── src/
│       ├── db.js
│       ├── baseline.js
│       └── utcDemo.js
├── backup/
│   ├── backup.sh
│   └── verify.sh
├── data/
│   └── sampledata.archive
├── docker/
│   ├── compose.yml
│   └── restore.sh
└── scripts/
    ├── run-app.sh
    └── queryStats.js

The sample dataset is a single archive that contains several databases. Download it into the data directory:

curl -o data/sampledata.archive https://atlas-education.s3.amazonaws.com/sampledata.archive

The archive is roughly 400 MB, so it may take a minute. We'll only restore the sample_mflix database from it later.

What Changes in MongoDB 9.0

Before touching any servers, it helps to know what you're signing up for. The compatibility changes page is the authoritative list, and you should read it in full before upgrading a real deployment. The changes most likely to affect an application fall into three groups.

Some changes happen silently, meaning nothing errors and you get a different answer:

• Server-side JavaScript in $function, $accumulator, $where, and mapReduce now evaluates Date operations in UTC rather than in the time zone of the mongod host.
• The $queryStats key and keyHash for aggregate queries that set allowPartialResults are serialized differently, so anything that matches on keyHash for those queries may see new values. The queryShapeHash doesn't change.

Some changes can cause new errors:

• A single operation is limited to the greater of 1 GB or 20% of server memory. Queries that previously used unbounded memory can fail with ExceededMemoryLimit, or with QueryExceededMemoryLimitNoDiskUseAllowed if disk use isn't allowed.
• All ingested BSON is validated by default, so invalid BSON that older versions accepted now returns an error.
• An accumulator with an empty field name in $group now returns an error.
• A query can fail with QueryKilledError if an indexed field becomes multikey while the query runs.
• A change stream opened with a readPreference of primary or secondary returns error code 11602 from getMore after an election. Compliant drivers resume on their own, but manual getMore loops need to resume with resumeAfter.

And some changes are new tooling worth knowing about:

• $queryStats is on by default, samples 1% of operations, and now covers update, insert, and delete.
• $function, $accumulator, and $where are no longer deprecated, and they run on a sandboxed WebAssembly-based engine.

We'll demonstrate the time zone change later because it's the one most likely to go unnoticed.

Create a Three-Member Replica Set on MongoDB 8.0

A realistic upgrade starts from a realistic deployment. In production, you'd run a replica set with at least three members, and the upgrade procedure for a replica set is a rolling one that takes advantage of that. We'll recreate the same shape locally.

Create a docker/compose.yml file with the following:

name: mongo9blog

x-mongo: &mongo
  command: ["--replSet", "rs0", "--bind_ip_all", "--wiredTigerCacheSizeGB", "0.25"]
  environment:
    TZ: America/New_York

services:
  mongo1:
    <<: *mongo
    image: ${MONGO1_IMAGE:-mongodb/mongodb-community-server:8.0.34-ubuntu2204}
    volumes:
      - mongo1-data:/data/db
  mongo2:
    <<: *mongo
    image: ${MONGO2_IMAGE:-mongodb/mongodb-community-server:8.0.34-ubuntu2204}
    volumes:
      - mongo2-data:/data/db
  mongo3:
    <<: *mongo
    image: ${MONGO3_IMAGE:-mongodb/mongodb-community-server:8.0.34-ubuntu2204}
    volumes:
      - mongo3-data:/data/db

volumes:
  mongo1-data:
  mongo2-data:
  mongo3-data:

There are a few things happening in this file. The x-mongo block at the top holds the settings that all three members share, and each service pulls them in with <<: *mongo.

x-mongo: &mongo
  command: ["--replSet", "rs0", "--bind_ip_all", "--wiredTigerCacheSizeGB", "0.25"]
  environment:
    TZ: America/New_York

The --replSet rs0 flag makes each mongod a replica set member. The cache size is capped at a quarter of a gigabyte so that three members fit comfortably on a laptop alongside everything else you're running. The TZ value runs the servers in the New York time zone, which will matter when we look at the server-side JavaScript change. Production servers are very often set to UTC, and if yours are, you won't see that particular difference, but the demo is still a good way to understand it.

The more important part for the upgrade is how each member gets its image:

  mongo1:
    <<: *mongo
    image: ${MONGO1_IMAGE:-mongodb/mongodb-community-server:8.0.34-ubuntu2204}
    volumes:
      - mongo1-data:/data/db

Each member has its own image variable, which defaults to MongoDB 8.0.34. That means we can start everything on 8.0 now, and later upgrade a single member by setting just that member's variable. Each member also has its own named volume, so the data survives when a container is replaced with a different image.

Start the three members:

docker compose -f docker/compose.yml up -d

The first run pulls the image, which takes a moment. Even after the containers are running, each mongod needs a few seconds to start accepting connections, so wait a few seconds before running the next command. If you see a connection refused error, wait a moment and run it again. The members also don't know about each other yet, so we need to initiate the replica set:

docker compose -f docker/compose.yml exec mongo1 mongosh --quiet --eval '
  rs.initiate({
    _id: "rs0",
    members: [
      { _id: 0, host: "mongo1:27017", priority: 2 },
      { _id: 1, host: "mongo2:27017" },
      { _id: 2, host: "mongo3:27017" }
    ]
  })
'

The priority: 2 on the first member makes mongo1 the preferred primary, which keeps the walkthrough predictable. The hostnames only resolve inside the Docker network, which is why the sample application will run in a container later.

Give the replica set a few seconds to elect a primary, then load the sample data. Create docker/restore.sh with the following:

#!/bin/sh
set -eu

cd "$(dirname "$0")/.."

docker run --rm \
  --network mongo9blog_default \
  -v "$PWD/data:/data-in:ro" \
  mongo:8.0 \
  mongorestore \
    --uri "mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0" \
    --archive=/data-in/sampledata.archive \
    --nsInclude 'sample_mflix.*'

The script runs mongorestore from a throwaway container on the same network, so you don't have to install the MongoDB Database Tools. The --nsInclude flag restores only the sample_mflix database from the archive. Run it with:

sh docker/restore.sh

When it finishes, the last line should tell you everything was restored:

67661 document(s) restored successfully. 0 document(s) failed to restore.

That's 21,349 movies, 41,079 comments, and the rest of the sample_mflix collections, and our starting point is ready.

Create an Application to Test the Upgrade

An upgrade that only proves the server starts isn't much of a test, so we'll use a small application that talks to the replica set before, during, and after the upgrade. It's written in Node.js, but the specifics matter less than the habit of running something real against the deployment at each stage.

Create app/package.json with the following:

{
  "name": "mongodb-9-upgrade-demo",
  "version": "0.1.0",
  "private": true,
  "type": "module",
  "scripts": {
    "baseline": "node src/baseline.js",
    "utc-demo": "node src/utcDemo.js"
  },
  "dependencies": {
    "mongodb": "^7.8.0"
  }
}

The two scripts correspond to the two files we're about to write. Next, create app/src/db.js, which is a small helper that connects, runs a function, and always closes the client:

import { MongoClient } from "mongodb";

const uri = process.env.MONGODB_URI ?? "mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0";

export async function withDb(fn) {
  const client = new MongoClient(uri);
  try {
    await client.connect();
    return await fn(client.db("sample_mflix"), client);
  } finally {
    await client.close();
  }
}

The connection string lists all three members and names the replica set. That's what lets the driver find whichever member is primary, and it's also what lets it keep working while members restart during a rolling upgrade.

The first script, app/src/baseline.js, reports what the application sees and does a read and a write:

import { withDb } from "./db.js";

await withDb(async (db) => {
  const admin = db.admin();
  const { version } = await admin.serverInfo();
  const fcv = await admin.command({ getParameter: 1, featureCompatibilityVersion: 1 });
  console.log("version:", version);
  console.log("fcv:", fcv.featureCompatibilityVersion.version);

  const movies = db.collection("movies");
  console.log("movies:", await movies.countDocuments());

  const one = await movies.findOne({ title: "The Matrix" }, { projection: { title: 1, year: 1 } });
  console.log("find-one:", one);

  const update = await movies.updateOne({ title: "The Matrix" }, { $inc: { demoViews: 1 } });
  console.log("update-one:", { matched: update.matchedCount, modified: update.modifiedCount });
});

The script prints the server version and feature compatibility version, counts the movies, runs a findOne, and runs an updateOne. Because it does both a read and a write, it shows that the application keeps working through each stage. It is not a benchmark.

To run it, we need a Node.js container on the same network as the replica set. Create scripts/run-app.sh with the following:

#!/bin/sh
set -eu

cd "$(dirname "$0")/.."

docker run --rm \
  --network mongo9blog_default \
  -v "$PWD/app:/app" \
  -w /app \
  -e MONGODB_URI="mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0" \
  node:22 \
  sh -c "npm install --silent --no-audit --no-fund && npm run --silent $1"

The script mounts the app directory into a node:22 container, installs the dependencies, and runs whichever npm script you pass in. Run the baseline:

sh scripts/run-app.sh baseline

You should see output similar to the following:

version: 8.0.34
fcv: 8.0
movies: 21349
find-one: {
  _id: new ObjectId('573a139bf29313caabcf3d23'),
  year: 1999,
  title: 'The Matrix'
}
update-one: { matched: 1, modified: 1 }

This is our "before" picture: MongoDB 8.0.34, feature compatibility version 8.0, and an application that reads and writes without any trouble.

Add a Check for a Behavior Change

A clean upgrade tells you that the server runs, not that your application behaves the same. One change in 9.0 is easy to miss because nothing errors: server-side JavaScript now evaluates Date operations in UTC instead of the time zone of the mongod host. We'll capture a "before" result now so we can compare it after the upgrade.

Create app/src/utcDemo.js with the following:

import { withDb } from "./db.js";

await withDb(async (db) => {
  const comments = db.collection("comments");

  const total = await comments.countDocuments();

  const matchesNewYorkHour = await comments.countDocuments({
    $expr: {
      $eq: [
        {
          $function: {
            body: "function (d) { return d.getHours(); }",
            args: ["$date"],
            lang: "js",
          },
        },
        { $hour: { date: "$date", timezone: "America/New_York" } },
      ],
    },
  });

  console.log(`getHours() equals the New York hour: ${matchesNewYorkHour} of ${total}`);
});

The query counts the comments where the hour returned by JavaScript's getHours() equals the hour calculated by $hour with an explicit America/New_York time zone. Our servers run in the New York time zone, so on 8.0 these should always agree. Run it against the 8.0 replica set:

sh scripts/run-app.sh utc-demo

Every comment agrees:

getHours() equals the New York hour: 41079 of 41079

Keep that number in mind. We'll run the same query again after the upgrade.

Check the Health of the Deployment Before You Upgrade

Before changing anything, confirm that what you have is healthy. The upgrade documentation calls out that no member should be in the ROLLBACK or RECOVERING state, and you also want to know how far each secondary is behind the primary.

Run the following to get a summary of the replica set:

docker compose -f docker/compose.yml exec mongo1 mongosh --quiet --eval '
  const status = rs.status();
  const primary = status.members.find((m) => m.stateStr === "PRIMARY");
  printjson({
    members: status.members.map((m) => ({
      name: m.name,
      state: m.stateStr,
      health: m.health,
      lagSecs: (primary.optimeDate - m.optimeDate) / 1000
    })),
    version: db.version(),
    fcv: db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 }).featureCompatibilityVersion.version
  })
'

The command lists each member with its state, health, and replication lag in seconds, plus the server version and feature compatibility version. A healthy set looks like this:

{
  members: [
    { name: 'mongo1:27017', state: 'PRIMARY', health: 1, lagSecs: 0 },
    { name: 'mongo2:27017', state: 'SECONDARY', health: 1, lagSecs: 0 },
    { name: 'mongo3:27017', state: 'SECONDARY', health: 1, lagSecs: 0 }
  ],
  version: '8.0.34',
  fcv: '8.0'
}

All three members are healthy with no lag, and the feature compatibility version is 8.0. If you see a member in RECOVERING or ROLLBACK, or lag that isn't shrinking, stop here and fix that first.

Two more checks belong in a real pre-flight and can't be shown with a sample dataset. First, you can only upgrade to 9.0 from 8.0 or 8.3, so if you're on 7.0 or earlier, upgrade to 8.0 first. If you're on 8.3, its feature compatibility version must be 8.3. Second, confirm that your drivers are compatible with 9.0. The upgrade documentation warns that incompatible drivers can produce unexpected or undefined behavior. These are the first driver versions that are fully compatible with MongoDB 9.0, according to the client library compatibility tables:

• Node.js: 7.6
• PyMongo: 4.18
• Java (sync and reactive streams), Kotlin, and Scala: 5.12
• .NET/C#: 3.10
• Go: 2.9
• Rust: 3.9
• Ruby: 2.26
• PHP library: 2.5
• C: 2.5
• C++: 4.6

Older versions listed as partially compatible will work with 9.0, but they don't support every new 9.0 feature, and the tables recommend moving to the versions above. The sample application in this tutorial uses version 7.8 of the Node.js driver, so it meets the requirement. Check the compatibility page for any driver or framework that isn't listed here.

Take a Backup and Prove That It Restores

If something goes wrong during an upgrade, a backup is the one safety net that doesn't depend on the upgrade itself working. It's also the step people skip, so we'll make it a real step and verify it.

For this tutorial, we'll use mongodump, which is a logical backup that most readers can run. Create backup/backup.sh with the following:

#!/bin/sh
set -eu

cd "$(dirname "$0")/.."

FILE="backup-$(date +%Y%m%d-%H%M%S).archive.gz"

docker run --rm \
  --network mongo9blog_default \
  -v "$PWD/backup:/backup" \
  mongo:8.0 \
  mongodump \
    --uri "mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0" \
    --oplog \
    --gzip \
    --archive="/backup/$FILE"

echo "Wrote backup/$FILE"

The script is run from your own machine, not from inside a container. The container names in the connection string, mongo1, mongo2, and mongo3, only resolve inside the Docker network, so the script uses docker run to start a short-lived mongo:8.0 container on that network (mongo9blog_default) and runs mongodump there. The -v flag mounts your local backup directory into that container, which is how the archive ends up on your machine. The --rm flag removes the container when the dump finishes, so there's nothing to clean up.

The --oplog flag is the important one for a replica set. While the dump runs, writes keep arriving, and --oplog captures those changes so the archive reflects a single point in time. The --gzip flag compresses the archive, and the timestamp in the file name keeps backups from overwriting each other. From the root of the project, in a terminal on your host machine, run it:

sh backup/backup.sh

On a small dataset, this finishes quickly and ends with the name of the file it wrote:

Wrote backup/backup-20261103-075154.archive.gz

Our archive is about 73 MB. Having a file isn't the same as having a backup, though, because you don't know a backup works until you've restored it. Create backup/verify.sh, which restores an archive into a throwaway MongoDB container and compares document counts against the live replica set:

#!/bin/sh
set -eu

cd "$(dirname "$0")/.."

ARCHIVE="$(basename "$1")"
SOURCE="mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0"
COUNT='JSON.stringify({movies: db.getSiblingDB("sample_mflix").movies.countDocuments(), comments: db.getSiblingDB("sample_mflix").comments.countDocuments()})'

docker rm -f mongo9blog-scratch >/dev/null 2>&1 || true
docker run -d --name mongo9blog-scratch --network mongo9blog_default mongo:8.0 >/dev/null

until docker exec mongo9blog-scratch mongosh --quiet --eval 'db.runCommand({ping: 1}).ok' >/dev/null 2>&1; do
  sleep 1
done

docker run --rm \
  --network mongo9blog_default \
  -v "$PWD/backup:/backup:ro" \
  mongo:8.0 \
  mongorestore \
    --uri "mongodb://mongo9blog-scratch:27017" \
    --oplogReplay \
    --gzip \
    --archive="/backup/$ARCHIVE" \
    --quiet

LIVE="$(docker run --rm --network mongo9blog_default mongo:8.0 mongosh "$SOURCE" --quiet --eval "$COUNT")"
RESTORED="$(docker exec mongo9blog-scratch mongosh --quiet --eval "$COUNT")"
docker rm -f mongo9blog-scratch >/dev/null

echo "live:     $LIVE"
echo "restored: $RESTORED"

if [ "$LIVE" = "$RESTORED" ]; then
  echo "Backup verified."
else
  echo "Counts do not match." >&2
  exit 1
fi

That's a longer script, so it helps to look at the parts. The first part starts a scratch mongod and waits until it answers a ping:

docker rm -f mongo9blog-scratch >/dev/null 2>&1 || true
docker run -d --name mongo9blog-scratch --network mongo9blog_default mongo:8.0 >/dev/null

until docker exec mongo9blog-scratch mongosh --quiet --eval 'db.runCommand({ping: 1}).ok' >/dev/null 2>&1; do
  sleep 1
done

The scratch container is completely separate from the replica set, so restoring into it can't affect your data. The second part restores the archive into it with --oplogReplay, which applies the oplog entries that --oplog captured:

docker run --rm \
  --network mongo9blog_default \
  -v "$PWD/backup:/backup:ro" \
  mongo:8.0 \
  mongorestore \
    --uri "mongodb://mongo9blog-scratch:27017" \
    --oplogReplay \
    --gzip \
    --archive="/backup/$ARCHIVE" \
    --quiet

The final part compares counts between the live replica set and the restored copy, and exits with an error if they differ. Run it against the backup you just took, using your own file name:

sh backup/verify.sh backup/backup-20261103-075154.archive.gz

If the backup is good, the counts match and the script says so:

live:     {"movies":21349,"comments":41079}
restored: {"movies":21349,"comments":41079}
Backup verified.

Comparing two document counts is a deliberately small check. In a real environment, you'd also restore into something that resembles production and run a representative query or two against the restored copy.

Atlas handles backups differently. On a paid cluster, snapshots run automatically once backups are enabled. Before you upgrade, confirm that backups are enabled and that your most recent snapshot is recent. Then take an on-demand snapshot so you have a restore point from just before the change, which matters here because Atlas can't downgrade a major version. And just like on Docker, a snapshot you haven't restored is a snapshot you haven't tested, so restore one to a separate cluster and check the result.

If your cluster doesn't include cloud backups, take the backup the same way we did here, by connecting to the cluster with mongodump and using the connection string from Atlas.

Decide Whether to Go Ahead

At this point, you have everything you need to make a decision, and it's worth making that decision deliberately instead of just continuing because the next step is on the page. Before moving forward, make sure the following are true:

• The replica set is healthy, with no member in ROLLBACK or RECOVERING and no growing lag.
• You have a backup, and you've restored it and checked the result.
• You've read the compatibility changes and none of them break your application, or you have a plan for the ones that do.
• Your drivers support 9.0.
• You're on 8.0 or 8.3, and you can afford to run for a while with mixed versions in the replica set.

If any of those aren't true, stop here. Nothing has changed yet, and nothing needs to be undone. The remaining steps are the ones that touch your servers.

Upgrade the Secondaries First

A rolling upgrade replaces one member at a time so the replica set stays available throughout. You upgrade the secondaries first, then step down the primary and upgrade it last. The application keeps working while this happens because the replica set always has a majority of members online.

We'll start with mongo2. Setting the image variable for just that member and bringing it up again replaces its container with one running MongoDB 9.0.4, while keeping its data volume:

MONGO2_IMAGE=mongodb/mongodb-community-server:9.0.4-ubuntu2204 \
  docker compose -f docker/compose.yml up -d mongo2

Compose recreates only mongo2. The member shuts down, starts again with the 9.0 binaries against the same data files, and rejoins the replica set. It takes a few seconds to start, so if you see a connection error, wait a moment and try again. Start by asking mongo2 itself which version it's running:

docker compose -f docker/compose.yml exec mongo2 mongosh --quiet --eval 'db.version()'

Once it answers, you should see the new version:

9.0.4

Another member's view of the replica set can briefly show a stale state right after a restart, so don't rely on it alone. Once mongo2 answers, ask the primary for each member's state and how far behind it is:

docker compose -f docker/compose.yml exec mongo1 mongosh --quiet --eval '
  const status = rs.status();
  const primary = status.members.find((m) => m.stateStr === "PRIMARY");
  status.members.map((m) => m.name + " " + m.stateStr + " lag: " + (primary.optimeDate - m.optimeDate) / 1000 + "s").join("\n")
'

Wait until mongo2 is a SECONDARY with no lag:

mongo1:27017 PRIMARY lag: 0s
mongo2:27017 SECONDARY lag: 0s
mongo3:27017 SECONDARY lag: 0s

The replica set is now running mixed versions. The primary and mongo3 are on 8.0.34, and mongo2 is on 9.0.4, which is normal and expected during a rolling upgrade. The feature compatibility version is still 8.0 on all members, so the data is still readable by 8.0.

Upgrade mongo3 the same way. Each docker compose command only knows the variables you pass it, so keep passing the one for mongo2. We name only mongo3 here, so mongo2 isn't touched either way, but the habit protects you if you ever run up -d without a service name, which would put mongo2 back on the 8.0 image:

MONGO2_IMAGE=mongodb/mongodb-community-server:9.0.4-ubuntu2204 \
MONGO3_IMAGE=mongodb/mongodb-community-server:9.0.4-ubuntu2204 \
  docker compose -f docker/compose.yml up -d mongo3

Once mongo3 reports SECONDARY again, run the application to confirm that it still works while the primary is the only member left on 8.0:

sh scripts/run-app.sh baseline

The output still shows version: 8.0.34 because the driver is talking to the primary, and the read and write both succeed:

version: 8.0.34
fcv: 8.0
movies: 21349
find-one: {
  _id: new ObjectId('573a139bf29313caabcf3d23'),
  year: 1999,
  title: 'The Matrix'
}
update-one: { matched: 1, modified: 1 }

The application never noticed that two of the three members changed underneath it.

Step Down the Primary and Upgrade It

The primary is the last member to upgrade. You can't just restart it without causing a brief interruption, so first ask it to step down, which triggers an election and lets one of the 9.0 secondaries take over. Run the following against mongo1:

docker compose -f docker/compose.yml exec mongo1 mongosh --quiet --eval 'rs.stepDown(60)'

The 60 tells the old primary not to seek re-election for 60 seconds. The connection from mongosh may drop as the node changes roles, which is normal. Check who the primary is now:

docker compose -f docker/compose.yml exec mongo2 mongosh --quiet --eval '
  rs.status().members.map((m) => m.name + " " + m.stateStr).join(" | ")
'

One of the 9.0 members has taken over as primary:

mongo1:27017 SECONDARY | mongo2:27017 PRIMARY | mongo3:27017 SECONDARY

Now mongo1 is a secondary, and it's safe to upgrade it with the same approach as the others. This time all three variables are set:

MONGO1_IMAGE=mongodb/mongodb-community-server:9.0.4-ubuntu2204 \
MONGO2_IMAGE=mongodb/mongodb-community-server:9.0.4-ubuntu2204 \
MONGO3_IMAGE=mongodb/mongodb-community-server:9.0.4-ubuntu2204 \
  docker compose -f docker/compose.yml up -d mongo1

Once mongo1 rejoins, confirm that every member is running 9.0.4:

for m in mongo1 mongo2 mongo3; do
  echo -n "$m: "
  docker compose -f docker/compose.yml exec $m mongosh --quiet --eval 'db.version()'
done

The loop asks each member for its version:

mongo1: 9.0.4
mongo2: 9.0.4
mongo3: 9.0.4

Every binary is now 9.0.4. Because mongo1 has the highest priority, it will likely have called an election and become primary again once it caught up. That detail matters in the next step.

Burn In Before You Raise the Feature Compatibility Version

The replica set is on 9.0 binaries, but the feature compatibility version (FCV) is still 8.0. This is the window where you let the new version prove itself. In this state, the data files are still compatible with 8.0, so rolling back is as simple as reversing the steps you just took.

Run the application again:

sh scripts/run-app.sh baseline

The output shows the new binary version with the old FCV:

version: 9.0.4
fcv: 8.0
movies: 21349
find-one: {
  _id: new ObjectId('573a139bf29313caabcf3d23'),
  year: 1999,
  title: 'The Matrix'
}
update-one: { matched: 1, modified: 1 }

In a tutorial, this takes seconds. In production, the burn-in period is days or weeks, depending on how comfortable you are with your own monitoring and how much traffic you can observe. Use it to watch error rates, latency, and the application behaviors listed earlier, and don't skip it, because the next step is the one that makes rollback harder.

Raise the Feature Compatibility Version

Raising the FCV to 9.0 enables 9.0 features and makes the replica set commit to the 9.0 data format. The documentation notes that it must run on the primary, and that it shouldn't run while an initial sync is in progress. The command is idempotent, so retrying it is safe if it doesn't complete.

Because the primary can change, find it first instead of assuming:

for m in mongo1 mongo2 mongo3; do
  docker compose -f docker/compose.yml exec $m mongosh --quiet --eval '
    if (db.hello().isWritablePrimary) print("primary: '$m'")
  '
done

The loop prints a line for whichever member is primary:

primary: mongo1

Right after the upgrade, mongo1 can take 10 to 20 seconds to win the election and become primary again, so if the loop printed a different member, wait a few seconds and run it again. Now run the command on whichever member is primary. The example below uses mongo1, so substitute the name the loop printed if it's different. If you run it on a secondary, you'll get a not primary error:

docker compose -f docker/compose.yml exec mongo1 mongosh --quiet --eval '
  db.adminCommand({ setFeatureCompatibilityVersion: "9.0", confirm: true })
'

The command waits until a majority of members have the new configuration, and then returns ok: 1. Verify the result on every member:

for m in mongo1 mongo2 mongo3; do
  echo -n "$m fcv: "
  docker compose -f docker/compose.yml exec $m mongosh --quiet --eval '
    db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 }).featureCompatibilityVersion.version
  '
done

Each member reports the new version:

mongo1 fcv: 9.0
mongo2 fcv: 9.0
mongo3 fcv: 9.0

The upgrade is complete. This is also the point where a downgrade is no longer a simple binary swap, which we'll cover shortly.

See the Behavior Change After the Upgrade

Now that the replica set is on 9.0, we can run the time zone check from earlier a second time against the upgraded replica set:

sh scripts/run-app.sh utc-demo

On MongoDB 9.0.4, the output is:

getHours() equals the New York hour: 0 of 41079

Before the upgrade, the same query against the same data returned 41079 of 41079. Every row agreed, and now none of them do.

Nothing failed, and nothing logged a warning. The query returned a different answer, which is exactly why this kind of change deserves a search through your code for $function, $accumulator, $where, and mapReduce before you upgrade.

If you do find a case that depends on the host time zone, the fix is to be explicit about the time zone. An aggregation operator such as $hour accepts a timezone option and behaves the same on every version:

{ $hour: { date: "$date", timezone: "America/New_York" } }

That's the same $hour expression we used as the comparison value, and it's the reliable half of the query.

Measure Your Own Workload with \$queryStats

The numbers we've published for 9.0 are "up to" figures measured against 8.0, and the announcement ties part of the gains to Atlas Gen2 compute. Results like these depend on the hardware, the workload, and the shape of your data, so treat them as a reason to test, not as a prediction.

Running a fair test on your own workload means using the same data and indexes on both versions, warming up first, running several times, and using realistic concurrency. It also means measuring reads and writes separately and recording the hardware you used. A laptop running a three-member replica set in Docker containers can't tell you anything about a production cluster, so we won't publish numbers from this one.

What 9.0 does give you is better visibility out of the box. $queryStats is on by default, samples 1% of operations, and now covers writes. For a short test, you can sample everything. Create scripts/queryStats.js with the following:

const admin = db.getSiblingDB("admin");

admin.runCommand({
  setParameter: 1,
  internalQueryStatsSampleRate: 1,
  internalQueryStatsWriteCmdSampleRate: 1,
});

const stats = admin
  .aggregate([
    { $queryStats: {} },
    { $match: { "key.queryShape.cmdNs.db": "sample_mflix" } },
    {
      $project: {
        _id: 0,
        command: "$key.queryShape.command",
        collection: "$key.queryShape.cmdNs.coll",
        execCount: "$metrics.execCount",
        totalExecMicros: "$metrics.totalExecMicros.sum",
      },
    },
    { $sort: { execCount: -1 } },
    { $limit: 10 },
  ])
  .toArray();

printjson(stats);

The script first turns sampling up to 100% for reads and for writes. It then reads $queryStats, keeps only operations against sample_mflix, and reports the command type, the collection, how many times it ran, and the total execution time in microseconds. Sampling is configured and reported per node, so run it against the primary. The commands below use mongo1, which is the primary at this point in the tutorial, so substitute a different name if your primary changed. Run it once to turn sampling on, run the workload, and then run it again to read the stats:

docker compose -f docker/compose.yml exec -T mongo1 mongosh --quiet --eval "$(cat scripts/queryStats.js)"
sh scripts/run-app.sh baseline
docker compose -f docker/compose.yml exec -T mongo1 mongosh --quiet --eval "$(cat scripts/queryStats.js)"

The first command turns sampling on, the second generates a workload, and the third prints what was recorded. The final result lists each kind of operation that ran:

[
  { command: 'update', collection: 'movies', execCount: Long('1'), totalExecMicros: Long('1666') },
  { command: 'aggregate', collection: 'movies', execCount: Long('1'), totalExecMicros: Long('4047') },
  { command: 'find', collection: 'movies', execCount: Long('1'), totalExecMicros: Long('1467') }
]

The update entry is there because starting in 9.0, $queryStats collects statistics for update, insert, and delete commands.

With realistic traffic, this gives you the number of times each query shape ran and how long it took in total, which is exactly the data you need to compare a workload before and after an upgrade.

If You Need to Roll Back

How hard a rollback is depends on how far along you are, so it helps to think about it in terms of the steps we took.

Before you raise the feature compatibility version, the data files are still compatible with 8.0. Rolling back means reversing the rolling upgrade, replacing one member at a time with the 8.0 image, stepping down the primary before you replace it. It's the same procedure you followed to upgrade, run in the other direction, with the 8.0 image in place of the 9.0 one. Check that each member is back to SECONDARY and caught up before you move to the next, just like you did on the way up.

After you raise the feature compatibility version, a downgrade is still supported to the immediately previous major version, but it takes more work. The documentation says you have to remove any persisted features that 9.0 introduced before you downgrade. One example is a view that uses $convert to convert an object to binData, which returns errors after a downgrade. The full procedure is on the Downgrade 9.0 to 8.0 page, which also says to downgrade to the latest patch release of 8.0. This tutorial doesn't walk through that procedure.

That's the reason the backup came first and the burn-in period comes before the feature compatibility version. If you're unsure, wait. The backup is your real safety net, and a longer burn-in is cheap.

Upgrading on MongoDB Atlas

If you run on MongoDB Atlas, most of what we did by hand is managed for you. MongoDB 9.0 is rolling out incrementally to clusters that use the "Latest Version With Auto Upgrades" option. Atlas performs the rolling upgrade for you, one member at a time, so there's no binary swap or feature compatibility version step for you to run. What it doesn't do is let you go back, because an Atlas cluster can't be downgraded from one major version to the previous one. That makes the backup and compatibility checks from this tutorial matter just as much on Atlas as they do anywhere else.

Before You Do This in Production

What we built is good for rehearsing the process, but a production upgrade needs more care. Our deployment has no authentication or TLS, our dataset is tiny, and a three-member replica set of Docker containers on one machine behaves very differently from a loaded production cluster. A real dataset also takes much longer to back up and verify, so plan for that, and for the extra load that a dump puts on a member.

Run the procedure on a staging environment that resembles production first, with your real application and a copy of real data, and write down what you observe. Plan the upgrade for a quiet window, and make sure your monitoring is in place before you start. And if you run sharded clusters, read the release notes carefully, because the known issues for sharded deployments are different from the ones that apply here.

Conclusion

You just saw how to upgrade a replica set from MongoDB 8.0 to 9.0 in a way that resembles the real thing. We checked the health of the deployment, took a backup and proved that it restores, decided whether to go ahead, upgraded the secondaries and then the primary, burned in on the new binaries, and finally raised the feature compatibility version. We also looked at a behavior change that doesn't produce an error, and at $queryStats as a way to measure your own workload.

To take this further, try the other behavior changes, such as the per-operation memory limit or change streams after an election, against the same replica set. You could also load a larger dataset and use $queryStats to compare a real workload between versions.