How to Configure a 3-Node MongoDB 6 Replica Set: Tested Step-by-Step Guide

Prerequisites

Before setting up the MongoDB 6.0.10 replica set, ensure you have completed the following prerequisites:

Step 1: Install MongoDB 6.0.10

Install MongoDB 6.0.10 using the provided script:

# Install required packages
sudo apt-get install gnupg curl -y

# Import the MongoDB public GPG Key
curl -fsSL <https://pgp.mongodb.com/server-6.0.asc> | \\
   sudo gpg -o /usr/share/keyrings/mongodb-server-6.0.gpg \\
   --dearmor

# Create a list file for MongoDB
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-6.0.gpg ] <https://repo.mongodb.org/apt/ubuntu> jammy/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list

# Update package list and install MongoDB
sudo apt-get update
sudo apt-get install mongodb-org -y

Step 2: Add Hosts and Configure Server Hostnames

Add the hostnames and IP addresses of all three servers to the /etc/hosts file on each server. Use this example as a reference:

192.168.1.21 primary.mongo
192.168.1.22 secondary1.mongo
192.168.1.23 secondary2.mongo

Step 3: Test Hostname Resolution

Ping each server from every other server using their hostnames to verify hostname resolution.

Example:

ping primary.mongo
ping secondary1.mongo
ping secondary2.mongo

Step 4: Create Admin User (If Not Enabled)

If you didn't enable authentication during MongoDB installation, create an admin user.

Example:

# Connect to MongoDB using the MongoDB shell
mongosh --host <Hostname or IP of each>:27017

# Switch to the admin database
use admin

# Create an admin user with appropriate roles
db.createUser({
    user: "admin",
    pwd: "your_password",
    roles: [
        "userAdminAnyDatabase",
        "dbAdminAnyDatabase",
        "readWriteAnyDatabase",
        "root"
    ]
})

Replace "your_password" with a secure password for the admin user.

Step 5: Enable Security with Keyfile

Generate a key and store it at /etc/mongodb/keys/mongo-key,
Just copy & paste the below command on all three servers:

sudo openssl rand -base64 756 > /etc/mongodb/keys/mongo-key
sudo chmod 600 /etc/mongodb/keys/mongo-key

Step 6: Restart MongoDB and Check Status

Restart MongoDB on all servers and check the status:

sudo service mongod restart
sudo service mongod status

Step 7: Test Connectivity (Optional)

Log in to MongoDB and check if it's working:

mongosh -u "admin_username"

Let's begin with ReplicaSet Configuration

Now proceed with the main setup steps:

Step 1: Update MongoDB Configuration Files

On each server, update the MongoDB configuration file /etc/mongod.conf with the provided example:

storage:
  dbPath: /opt/mongodb

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

net:
  port: 27017
  bindIp: 0.0.0.0

processManagement:
  timeZoneInfo: /usr/share/zoneinfo
  fork: true

security:
  authorization: enabled
  keyFile: /etc/mongodb/keys/mongo-key
  transitionToAuth: true

replication:
  replSetName: rs111

Step 2: Start MongoDB on Each Server

Start MongoDB on each server using the following command:

sudo service mongod start
sudo service mongod status

Step 3: Connect to Primary's Mongo Shell (mongosh or mongo)

Connect to one of the MongoDB servers using the MongoDB shell (mongosh).

Example:

mongosh --host server_ip:27017

Replace server_ip with the IP address of the server you want to connect to.

Step 4: Initialize the Replica Set

Inside the MongoDB shell, initialize the replica set with the chosen name (rs111). Run this command on the server you want as the primary.

Example:

rs.initiate({
  _id: "rs111",
  members: [
    { _id: 0, host: "primary.mongo:27017" },
    { _id: 1, host: "secondary1.mongo:27017" },
    { _id: 2, host: "secondary2.mongo:27017" }
  ]
})

Replace primary_hostname, secondary1_hostname, and secondary2_hostname with the hostnames of your servers. The primary should be the server where you run this command.

Step 5: Verify Replica Set Status

Verify the status of the replica set by running the following command in the MongoDB shell:

rs.status()

To check the status of replicaSet

Troubleshooting Common Issues & Notes

Issue 1: "no reachable/healthy members" or "STARTUP" state

• Solution: If you encounter these issues, it means that one or more replica set members are not reachable or are still in the startup process. Check network connectivity, MongoDB status, and logs.
  

  sudo less /var/log/mongodb/mongod.log
  

• Solution: Check /etc/hosts in each server, are all three hostnames added and are they using correct IP Addresses.
  

  192.168.1.21 primary.mongo
  192.168.1.22 secondary1.mongo
  192.168.1.23 secondary2.mongo
  

• Solution: If nothing works reconfigure replicaSet using correct IP Addresses.

1. Connect to the primary node of your replica set using the mongo shell:

       mongo --host primary_ip:27017

Replace primary_ip with the IP address of your primary node.

1. In the mongo shell, execute the following command to obtain the current replica set configuration:

        var cfg = rs.conf()

This command stores the current replica set configuration in the cfg variable.

1. Edit the cfg variable to update the hostnames with their respective IP addresses. You will need to modify the host field for each member in the configuration.

For example:

        cfg.members[0].host = "192.168.1.21:27017";
        cfg.members[1].host = "192.168.1.22:27017";
        cfg.members[2].host = "192.168.1.23:27017";

Replace primary_ip, secondary1_ip, and secondary2_ip with the actual IP addresses and port numbers for your servers.

1. After making the necessary changes, reconfigure the replica set using the updated cfg variable:

        rs.reconfig(cfg)     

This command will apply the modified configuration to the replica set.

1. To ensure that the changes are applied successfully, you can check the replica set status using:

        rs.status()

Verify that the hostnames have been updated with the correct IP addresses in the replica set configuration.

Make sure to execute these commands carefully and replace the placeholders with your actual server IP addresses and port numbers.

Issue 2: Authentication Errors

• Solution: If you encounter authentication errors, ensure valid credentials are provided when connecting to MongoDB. Verify user privileges, or check if you’ve created an admin user in secondary servers.

Additional Notes:

• Use 0.0.0.0 for bindIp only if you've enabled authentication by adding an admin user in the admin database of all MongoDB instances.
• You can change default “dbPath: /var/lib/mongodb” or choose a different path like “dbPath: /opt/mongodb” in mongod.conf file.