DEV Community: Abdelfattah Radwan

Deploy MongoDB Replica Sets with Ease Using Docker Compose

Abdelfattah Radwan — Sat, 01 Feb 2025 17:45:39 +0000

Hello everyone, and welcome.

A while back, I wrote a blog post that showed how to quickly get a MongoDB cluster up and running without too much hassle. Since then, I have learned a lot more about Docker and MongoDB. This post will show you how to do the same thing more easily and streamlined. Let's get started!

First, let's create the required files. Those are:
.env: An environment file that contains our MongoDB root username, password, and replica set key.
keyfile.sh: A bash script which will put our replica set key in a file and set the correct permission for that file.
docker-compose.yml: The Docker compose file in which we define our services.
Our project directory now should look something like this:

mongodb
├── .env
├── docker-compose.yml
└── keyfile.sh

Now, let's go ahead and set up our environment variables. Which are:

MONGODB_ROOT_USERNAME: The username of the root MongoDB user.
MONGODB_ROOT_PASSWORD: The password of the root MongoDB user.
MONGO_REPLICA_SET_KEY: The replica set key, which we will put in the key file later.

I will use "root" for the username, and for the password, I will use the openssl command to generate 32 random bytes in hex format to use as the password. The command is as follows:

openssl rand -hex 32

The output will look something like this: 4c1adead789af1c3bc7a164a6ede1a4398a74b61a9f5b10aa5db5ef19334962c

Some requirements must be met for the replica set key to be accepted by the replica set nodes. You can find them on MongoDB's documentation website here.

To generate the key, we will use the following command:

openssl rand -base64 756

That will generate a long string of base64 characters, which represents our key.

With all of that done, you should have an environment file that looks something like this:

MONGODB_ROOT_USERNAME="root"
MONGODB_ROOT_PASSWORD="e771096f5659c043bf5117b64eae4001e0ac6fd1344e469cc2ceb0fdb4453620"

MONGO_REPLICA_SET_KEY="y5yYpzyuzGYjK8xMsY+rjm7B/PfmFm/DSBoR/FlYAnoug/5Xyj2bHP7h3D5vFJrAOC9hrIfG/nmj1WgBOqxa2m/NX4aSfbu3jVVR4Oq1YG8od6PhY2cSfHoiHGZiw0WannwuQ1TFiU8EF6R0jxksTZJayAi1sPhEwwvEvHccC2gaPHA30Q4biiUazBnh9W2O9xV5oFSnCzvGRkQk9rhZQcf4X4kFR78Xa7wGtX0Qc3PudLgvTTMz5RlYuCY+r17zHFUohtPM4OdADx0SLB/D74NTasRF4BKlsOrjp8rRUcCTD7pAQ8qCtUukwYArCV0/ryb67oTfazGei/G81mkhhpPCBhxgzsNgjUyAXXupiVNnqQZjNQJBfP8iqYM7u1SYE7FLTZG7OXWDtM1iwxk7F3LS2HRfG8BQSGOAbOvegXccVRSKzilFd8gZkirGvI5ZvVfMZ9PNgSk0VGfukYeiWU+j8EJZlnxkBhMHS17fGhmfoC39E2sdMWacG0JcER+UKyWRT9qpKJw8NH07BcuPFMIYfrFt0PXw3RnVmzPIvmVjYQVL6uIoygJs4qnMBPk9r1fYm+5vklz4v2pZI6SsfcrdWsg+8K4CIiVF+J2wdBmM1zSmG3T95iPxmhCfgkmRs9dz7GI2/axA++jox60DGTtsVTxHQca2JB2pkEG+jwZAtSJXMNkjhlFZVfDyp4MNAj4i3awsHD8vW6v79vyBtxTC6aS03+j2y/0JteI+twt3ASNAwVHoERIfwuNd7hFRTDl//n/7KcVJuiyriRXL+X/JCbg+5s0qZmVqnESOVI4Vi0KLL4Lx/Zir7RfhQ2s2woW2gMULJ+6zfIGY+U7XGYJd/N8IP+HKK2fGaQrYbS0xhqcVzdg3lEvMu3ky7WfxJtRnlqt/TarOod88VzcYlmoRAWmlkNMEjL0/xq5rrQ+u0nYOk3Mgc+MS3P/4X19ITtgm6H5yDb1jRDD9H/C4J7+PuzU8uf9G3zuCcUDCW6zz6AW1"

Now, let's create the keyfile.sh script. As I mentioned, it sets up our keyfile with the correct replica set key and correct permissions. Here's how it should be:

#!/bin/bash

# Echo MONGO_REPLICA_SET_KEY to the keyfile.

echo $MONGO_REPLICA_SET_KEY > /data/configdb/keyfile

# Change the permissions of the keyfile.

chmod 400 /data/configdb/keyfile

# Change the ownership of the keyfile.

chown mongodb:mongodb /data/configdb/keyfile

Now that our env and keyfile.sh files are ready, let's move on to our docker-compose.yml file. It will contain three services:

primary: Our primary replica set node.
replica_1: Our first secondary node in the replica set.
replica_2: Our second secondary node in the replica set.
init: Will be used to initialise the replica set.

Beginning with our primary service:

  primary:
    image: mongo:latest
    command: mongod --port 27017 --replSet rs0 --bind_ip_all --keyFile /data/configdb/keyfile
    ports:
      - "27017:27017"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped
    environment:
      - MONGO_INITDB_ROOT_USERNAME=root
      - MONGO_INITDB_ROOT_PASSWORD=root
      - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
      - ./keyfile.sh:/docker-entrypoint-initdb.d/keyfile.sh
      - primary_data_db:/data
      - primary_data_configdb:/data/configdb
    healthcheck:
      test: 'mongosh --quiet --port 27017 --eval "db.runCommand({ ping: 1 }).ok" | grep 1'
      interval: 5s

As you can see, it's a straightforward service definition with a couple of interesting things:
Under volumes, we mount our keyfile.sh file to docker-entrypoint-initdb.d/keyfile.sh, making Docker execute it during the container's initialisation process.
We defined a healthcheck element, which will ping the MongoDB instance running inside the container, in this case, every 5 seconds, to check if it's "healthy" (started and ready) or not.

Moving on to our replica service definitions. Here's the first replica service definition:

  replica_1:
    image: mongo:latest
    command: mongod --port 27018 --replSet rs0 --bind_ip_all --keyFile /data/configdb/keyfile
    ports:
      - "27018:27018"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped
    environment:
      - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
      - ./keyfile.sh:/docker-entrypoint-initdb.d/keyfile.sh
      - replica_1_data_db:/data
      - replica_1_data_configdb:/data/configdb
    healthcheck:
      test: 'mongosh --quiet --port 27018 --eval "db.runCommand({ ping: 1 }).ok" | grep 1'

And the second:

  replica_2:
    image: mongo:latest
    command: mongod --port 27019 --replSet rs0 --bind_ip_all --keyFile /data/configdb/keyfile
    ports:
 - "27019:27019"
    extra_hosts:
 - "host.docker.internal:host-gateway"
    restart: unless-stopped
    environment:
 - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
 - ./keyfile.sh:/docker-entrypoint-initdb.d/keyfile.sh
 - replica_2_data_db:/data
 - replica_2_data_configdb:/data/configdb
    healthcheck:
      test: 'mongosh --quiet --port 27019 --eval "db.runCommand({ ping: 1 }).ok" | grep 1'

They are very similar to the primary service definition, except we don't give them a root username and password, only a replica set key.

Finally, let's define our init service:

  init:
    image: mongo:latest
    restart: no
    extra_hosts:
 - "host.docker.internal:host-gateway"
    depends_on:
      primary:
        condition: service_healthy
      replica_1:
        condition: service_healthy
      replica_2:
        condition: service_healthy
    command: >
 mongosh -u root -p root --host host.docker.internal --port 27017 --eval '
 rs.initiate({
 _id: "rs0",
 members: [
 { _id: 0, host: "host.docker.internal:27017" },
 { _id: 1, host: "host.docker.internal:27018" },
 { _id: 2, host: "host.docker.internal:27019" }
 ]
 })
 '

As I mentioned before, this service will initialise the replica set. It will do that by:
Waiting until the primary and all secondary nodes are ready.
Execute a command using mongosh to initialise the replica set.

Unlike the previous version of this post, we no longer need a bash script and a while loop to check if the nodes are ready. We use healthcheck elements and the depends_on element to emulate that behaviour more straightforwardly.

The final docker-compose.yml file should look something like this:

name: mongodb_replica_set

services:
  primary:
    image: mongo:latest
    command: mongod --port 27017 --replSet rs0 --bind_ip_all --keyFile /data/configdb/keyfile
    ports:
 - "27017:27017"
    extra_hosts:
 - "host.docker.internal:host-gateway"
    restart: unless-stopped
    environment:
 - MONGO_INITDB_ROOT_USERNAME=root
 - MONGO_INITDB_ROOT_PASSWORD=root
 - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
 - ./keyfile.sh:/docker-entrypoint-initdb.d/keyfile.sh
 - primary_data_db:/data
 - primary_data_configdb:/data/configdb
    healthcheck:
      test: 'mongosh --quiet --port 27017 --eval "db.runCommand({ ping: 1 }).ok" | grep 1'
      interval: 5s

  replica_1:
    image: mongo:latest
    command: mongod --port 27018 --replSet rs0 --bind_ip_all --keyFile /data/configdb/keyfile
    ports:
 - "27018:27018"
    extra_hosts:
 - "host.docker.internal:host-gateway"
    restart: unless-stopped
    environment:
 - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
 - ./keyfile.sh:/docker-entrypoint-initdb.d/keyfile.sh
 - replica_1_data_db:/data
 - replica_1_data_configdb:/data/configdb
    healthcheck:
      test: 'mongosh --quiet --port 27018 --eval "db.runCommand({ ping: 1 }).ok" | grep 1'

  replica_2:
    image: mongo:latest
    command: mongod --port 27019 --replSet rs0 --bind_ip_all --keyFile /data/configdb/keyfile
    ports:
 - "27019:27019"
    extra_hosts:
 - "host.docker.internal:host-gateway"
    restart: unless-stopped
    environment:
 - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
 - ./keyfile.sh:/docker-entrypoint-initdb.d/keyfile.sh
 - replica_2_data_db:/data
 - replica_2_data_configdb:/data/configdb
    healthcheck:
      test: 'mongosh --quiet --port 27019 --eval "db.runCommand({ ping: 1 }).ok" | grep 1'

  init:
    image: mongo:latest
    restart: no
    extra_hosts:
 - "host.docker.internal:host-gateway"
    depends_on:
      primary:
        condition: service_healthy
      replica_1:
        condition: service_healthy
      replica_2:
        condition: service_healthy
    command: >
 mongosh -u root -p root --host host.docker.internal --port 27017 --eval '
 rs.initiate({
 _id: "rs0",
 members: [
 { _id: 0, host: "host.docker.internal:27017" },
 { _id: 1, host: "host.docker.internal:27018" },
 { _id: 2, host: "host.docker.internal:27019" }
 ]
 })
 '

volumes:
  primary_data_db:
  primary_data_configdb:
  replica_1_data_db:
  replica_1_data_configdb:
  replica_2_data_db:
  replica_2_data_configdb:

networks:
  default:
    name: mongodb_replica_set_network

Now, we execute the following command inside the directory where our docker-compose.yml file is:

docker compose up -d

That will pull the mongo:latest image from a container registry (if you don't already have it pulled locally) and create our volumes and networks.

You might have to wait a little before everything is up and running, but it usually takes no more than 7 seconds for the primary node and ~31 seconds for the secondaries to be ready.

Once everything is ready, open MongoDB Compass and connect to the replica set!

To do that, once MongoDB Compass is open, click on "Add new connection".

Expand the "Advanced Connection Options" dropdown and click the "Authentication" tab.

Input root username and password.

Click on "Advanced" and input the replica set name.

Click on "General" and add hosts for each replica set node.

Finally, click on "Save & Connect," and voila! You've successfully connected to your very own locally hosted MongoDB replica set!

You can now start adding and querying data in MongoDB Compass or using one of the many drivers available for many languages.

I hope you found this helpful.

Thanks for reading!

Quick and Easy MongoDB Replica Set Deployment with Docker Compose

Abdelfattah Radwan — Mon, 15 Jul 2024 16:17:50 +0000

(I published an update version of this post here)

For the past few weeks, in my free time, I have wanted to learn how to set up a MongoDB replica set that I could use for development or production, but I have been frustrated by the resources available online.

Most were outdated or didn’t explain how to set up authentication using key files, which is required when deploying replica sets.

So, without further ado, let’s get started!

First, you will need to download and install Docker. I downloaded and installed Docker Desktop for Windows.

Next, open a Windows Terminal (or CMD.exe) window and run the following command: docker image pull mongo:latest. This will pull the latest version of the mongo:latest image.

You can also use Docker Desktop to do that from the GUI. Open up the Docker Desktop application, search for mongo:latest, select "latest" from the "Tag" dropdown, and click "Pull".

Once the image is pulled successfully, create a new directory named "mongodb" anywhere on your machine.

Inside that directory, create a new file named docker-compose.yml and open it up in your preferred text/source code editor. I will be using VSCode.

Let's begin by specifying the name of our Docker Compose project.

name: mongodb-replica-set

Then, let’s define our first MongoDB replica set instance:

services:
  mongodb-1:
    image: mongo:latest
    container_name: mongodb-1
    command:
      [
        "mongod",
        "--port",
        "27017",
        "--replSet",
        "rs0",
        "--bind_ip_all",
        "--keyFile",
        "/data/configdb/keyfile",
      ]
    ports:
      - "27017:27017"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: always
    environment:
      - MONGO_INITDB_ROOT_USERNAME=root
      - MONGO_INITDB_ROOT_PASSWORD=root
      - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
      - "./init-keyfile.sh:/docker-entrypoint-initdb.d/init-keyfile.sh"
      - "./data/mongodb-1:/data/db"
      - "./data/mongodb-1/configdb:/data/configdb"

You might be a little intimidated by the size of this “simple” config, but fear not, let’s go through it:

Image and Container Name: We use the latest MongoDB image and name this container mongodb-1.
Command: This starts the MongoDB server (mongod) with specific options:
- port 27017: Sets the port to 27017.
- replSet rs0: Specifies the replica set name as rs0.
- bind_ip_all: Binds MongoDB to all available IP addresses.
- keyFile /data/configdb/keyfile: Specifies the key file for authentication.
Ports: Maps port 27017 of the container to port 27017 on the host machine.
Extra Hosts: Adds an entry to the container’s /etc/hosts file.
Restart Policy: Ensures the container always restarts.
Environment Variables: Sets the root username, password, and the replica set key.
Volumes: Mounts necessary files and directories from the host to the container.

Although not required, on Windows, I suggest you add the "host.docker.internal:host-gateway" to the extra_hosts attribute. This allows drivers and apps like MongoDB Compass to resolve the addresses of other replica set instances. As far as I know, that is not required on Linux.

You might have also noticed that we have a custom environment variable called MONGO_REPLICA_SET_KEY. It holds the key that will be put inside our instances’ key files. The key file is created using the init-keyfile.sh script. The script copies the value of MONGO_REPLICA_SET_KEY into a file named "keyfile" located at /data/configdb and sets the correct permissions and owner of the file.

#!/bin/bash

# Echo MONGO_REPLICA_SET_KEY to the keyfile.

echo $MONGO_REPLICA_SET_KEY > /data/configdb/keyfile

# Change the permissions of the keyfile.

chmod 400 /data/configdb/keyfile

# Change the ownership of the keyfile.

chown mongodb:mongodb /data/configdb/keyfile

You can generate a key using the following command (taken from the MongoDB docs):

openssl rand -a -base64 756

Copy and paste the generated key into a .env file:

MONGO_REPLICA_SET_KEY="YOUR_REPLICA_SET_KEY"

Now, let’s define a couple more MongoDB instances for our replica set:

mongodb-2:
    image: mongo:latest
    container_name: mongodb-2
    command:
      [
        "mongod",
        "--port",
        "27018",
        "--replSet",
        "rs0",
        "--bind_ip_all",
        "--keyFile",
        "/data/configdb/keyfile",
      ]
    ports:
      - "27018:27018"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: always
    environment:
      - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
      - "./init-keyfile.sh:/docker-entrypoint-initdb.d/init-keyfile.sh"
      - "./data/mongodb-2:/data/db"
      - "./data/mongodb-2/configdb:/data/configdb"

mongodb-3:
    image: mongo:latest
    container_name: mongodb-3
    command:
      [
        "mongod",
        "--port",
        "27019",
        "--replSet",
        "rs0",
        "--bind_ip_all",
        "--keyFile",
        "/data/configdb/keyfile",
      ]
    ports:
      - "27019:27019"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: always
    environment:
      - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
      - "./init-keyfile.sh:/docker-entrypoint-initdb.d/init-keyfile.sh"
      - "./data/mongodb-3:/data/db"
      - "./data/mongodb-3/configdb:/data/configdb"

These two instances are the same as the first, with only minor differences.

Let’s define a fourth MongoDB instance that will wait for the first instance to start before initiating the replica set.

mongodb-init:
    image: mongo:latest
    container_name: mongodb-init
    depends_on:
      - mongodb-1
      - mongodb-2
      - mongodb-3
    extra_hosts:
      - "host.docker.internal:host-gateway"
    command: bash -c "chmod +x init-replica-set.sh && ./init-replica-set.sh"
    volumes:
      - "./init-replica-set.sh:/init-replica-set.sh"
    restart: no

Pretty simple, eh? This instance is configured to wait for the other three instances’ containers to start before it starts and executes a script called init-replica-set.sh. Here is the script:

#!/bin/bash

echo "Waiting for MongoDB to start on host.docker.internal:27017..."

until mongosh --quiet --host host.docker.internal --port 27017 --eval "db.runCommand({ ping: 1 }).ok" | grep 1 &>/dev/null; do
  sleep 1
done

echo "MongoDB has started successfully"

echo "Initiating MongoDB replica set..."

mongosh -u root -p root --host host.docker.internal --port 27017 --eval "
  rs.initiate({
    _id: 'rs0',
    members: [
      {
        _id: 0,
        host: 'host.docker.internal:27017'
      },
      {
        _id: 1,
        host: 'host.docker.internal:27018'
      },
      {
        _id: 2,
        host: 'host.docker.internal:27019'
      }
    ]
  })
"

Again, don’t worry if it looks intimidating, let’s go through it:

echo "Waiting for MongoDB to start on host.docker.internal:27017..."

until mongosh --quiet --host host.docker.internal --port 27017 --eval "db.runCommand({ ping: 1 }).ok" | grep 1 &>/dev/null; do
  sleep 1
done

echo "MongoDB has started successfully"

This block waits until the MongoDB instance on port 27017 is up and running. It checks by pinging the MongoDB server and waiting for a successful response.
The loop continues to check every second (sleep 1) until a successful ping response is received.

echo "Initiating MongoDB replica set..."

mongosh -u root -p root --host host.docker.internal --port 27017 --eval "
  rs.initiate({
    _id: 'rs0',
    members: [
      {
        _id: 0,
        host: 'host.docker.internal:27017'
      },
      {
        _id: 1,
        host: 'host.docker.internal:27018'
      },
      {
        _id: 2,
        host: 'host.docker.internal:27019'
      }
    ]
  })
"

This block initiates the replica set using the mongosh command.
It connects to the MongoDB instance on port 27017 using the root credentials.
The rs.initiate command sets up the replica set with the specified members (host.docker.internal:27017, host.docker.internal:27018, and host.docker.internal:27019).

Note that we used host.docker.internal and NOT the container name as you’d usually do when working with containers. This again is only required on Windows.

The complete docker-compose.yml file should look something like this:

name: mongodb-replica-set

services:
  mongodb-1:
    image: mongo:latest
    container_name: mongodb-1
    command:
      [
        "mongod",
        "--port",
        "27017",
        "--replSet",
        "rs0",
        "--bind_ip_all",
        "--keyFile",
        "/data/configdb/keyfile",
      ]
    ports:
      - "27017:27017"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped
    environment:
      - MONGO_INITDB_ROOT_USERNAME=root
      - MONGO_INITDB_ROOT_PASSWORD=root
      - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
      - "./init-keyfile.sh:/docker-entrypoint-initdb.d/init-keyfile.sh"
      - "./data/mongodb-1:/data/db"
      - "./data/mongodb-1/configdb:/data/configdb"

  mongodb-2:
    image: mongo:latest
    container_name: mongodb-2
    command:
      [
        "mongod",
        "--port",
        "27018",
        "--replSet",
        "rs0",
        "--bind_ip_all",
        "--keyFile",
        "/data/configdb/keyfile",
      ]
    ports:
      - "27018:27018"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: always
    environment:
      - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
      - "./init-keyfile.sh:/docker-entrypoint-initdb.d/init-keyfile.sh"
      - "./data/mongodb-2:/data/db"
      - "./data/mongodb-2/configdb:/data/configdb"

  mongodb-3:
    image: mongo:latest
    container_name: mongodb-3
    command:
      [
        "mongod",
        "--port",
        "27019",
        "--replSet",
        "rs0",
        "--bind_ip_all",
        "--keyFile",
        "/data/configdb/keyfile",
      ]
    ports:
      - "27019:27019"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: always
    environment:
      - MONGO_REPLICA_SET_KEY=${MONGO_REPLICA_SET_KEY}
    volumes:
      - "./init-keyfile.sh:/docker-entrypoint-initdb.d/init-keyfile.sh"
      - "./data/mongodb-3:/data/db"
      - "./data/mongodb-3/configdb:/data/configdb"

  mongodb-init:
    image: mongo:latest
    container_name: mongodb-init
    depends_on:
      - mongodb-1
      - mongodb-2
      - mongodb-3
    extra_hosts:
      - "host.docker.internal:host-gateway"
    command: bash -c "chmod +x init-replica-set.sh && ./init-replica-set.sh"
    volumes:
      - "./init-replica-set.sh:/init-replica-set.sh"
    restart: no

networks:
  default:
    name: mongodb-replica-set-network

Finally, run the following command in the directory containing the docker-compose.yml file: docker compose up -d

You might have to wait a bit before everything starts up, but afterwards, you can connect to your replica set via one of the drivers or MongoDB Compass. I will use MongoDB Compass to demonstrate the final result.

Open MongoDB Compass and enter the following connection string: mongodb://root:root@localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0

Click on “Connect” and voila! MongoDB Compass will present you with a screen like this:

That’s it! Now, you have a fully functional MongoDB replica set! However, running all instances of the replica set on the same machine defeats the purpose of replica sets: redundancy and data availability. So, I advise you to spread your instances across different machines or create a single-instance replica set for local development.