How to migrate a Mongo Database with Ansible Playbook

Recently, a client tasked me with migrating MongoDB from MongoDB Atlas to Digital Ocean managed MongoDB service. He had 2 reasons, one was to manage everything in Digital Ocean since the application server was in Digital Ocean, and two was to minimize the network latency, the client wanted to have the MongoDB in the same VPC as the application server.

Prerequisites

• Database Tools (The package includes mongodump and mongorestore)
• Ansible

Migration Concept

To migrate the mongo database, we will use the backup (mongodump) and restore (mongorestore) utilities. In my case, I ran both commands from my application server in Digital Ocean. We will then use the Ansible Playbook to create a template to run it as many times as we need. In my case, the client wanted to run it twice. First, he wanted me to migrate the DB to a staging environment, and once everything worked correctly, he asked me to migrate the DB to a production environment. Ansible Playbook also provides time calculation, giving you an idea of how long the live Mongodb migration will take.

Backup

We will back up the MongoDB from MongoDB Atlas using the mongodump utility.

Syntax and Example

To backup your existing database, the mongodump syntax will be like this:

mongodump --username DB_USERNAME --out=TARGET_FOLDER --config=mongodump.cfg

Note that:

• username is the username we use to authenticate to our mongo DB.
• out is the target folder to store our mongo DB backup.
• config is the config file that will be used by mongodump.

The mongodump.cfg contains these configurations:

password: PASSWORD
uri: mongodb+srv://HOSTNAME.mongodb.net/DATABASE_NAME?retryWrites=true&w=majority

Change PASSWORD, HOSTNAME, and DATABASE_NAME accordingly. retryWrites=true&w=majority are connection parameters used by mongo DB Atlas, so leave them as they are.

Restore

We will restore the MongoDB to Digital Ocean managed mongo DB service using the mongorestore utility.

Syntax and Example

To restore to a database, the mongorestore syntax will be like this:

mongorestore --ssl --tlsInsecure --username USERNAME --config=mongorestore.cfg --nsInclude=DATABASE_NAME TARGET_FOLDER/DATABASE_NAME/

Note that:

• ssl means we will connect to mongo DB via TLS/SSL.
• tlsInsecure will disable the certificate validations.
• config is the config file that will be used by mongorestore.
• nsInclude specifies the namespace pattern. Since we will restore the whole database, we will use DATABASE_NAME (change this with your database name).

The mongorestore.cfg contains these configurations:

uri: mongodb+srv://HOSTNAME.mongo.ondigitalocean.com/DATABASE_NAME?authSource=admin
password: PASSWORD

Change PASSWORD, HOSTNAME, and DATABASE_NAME accordingly.
authSource=admin is a connection parameter used by Digital Ocean managed mongo DB service, so leave it as it is.

Ansible Playbook

We will now use Ansible Playbook. First create a file named playbook.yml, and then put these in the file:

# How to run: ansible-playbook playbook.yaml
-
  name: Migrate mongodb
  hosts: localhost
  tasks:
    - name: Remove backup folder if exists
      ansible.builtin.file:
        path: /root/mongorestore/backup
        state: absent
    - name: Mongodump from mongodb atlas
      shell: mongodump --username USERNAME --out=backup/mongodb --config=mongodump.cfg
    - name: Mongorestore to Digital Ocean managed database - mongodb
      shell: mongorestore --ssl --tlsInsecure --username USERNAME --config=mongorestore.cfg --nsInclude=DATABASE_NAME backup/mongodb/DATABASE_NAME/

Change USERNAME and DATABASE_NAME accordingly.
The ansible playbook contains 3 tasks, and they are:

1. The first task will remove the backup folder if it exists. Since we will run the ansible playbook many times, we need to do this to ensure there is no trace of the previous backup
2. The 2nd task will dump mongo DB
3. The last task will restore mongo DB

Next, we need to create a file named mongodump.cfg, and then put these in the file:

password: PASSWORD
uri: mongodb+srv://HOSTNAME.mongodb.net/DATABASE_NAME?retryWrites=true&w=majority

Remember to change PASSWORD, HOSTNAME, and DATABASE_NAME accordingly.

Then create a file named mongorestore.cfg, and then put these in the file:

uri: mongodb+srv://HOSTNAME.mongo.ondigitalocean.com/DATABASE_NAME?authSource=admin
password: PASSWORD

Lastly, we will configure Ansible to display the time calculation to know how much time we need to run all the 3 tasks. According to the doc here, there are 4 ways that we can configure (pick one), and they are:

• The ANSIBLE_CONFIG environment variable
• Create a file named ansible.cfg and put it in the same folder as the playbook.yml resides
• ~/.ansible.cfg, if you are logged in as root, the file will be stored in /root. If you are logged in another user, e.g. ubuntu, it will be stored in /home/ubuntu
• Modify the file in /etc/ansible/ansible.cfg

Let's say we create a file named ansible.cfg. Once created, put these in the file:

[defaults]
callback_whitelist = profile_tasks, profile_roles, timer

We use callback_whitelist to allow some plugins to run during the playbook execution. For ansible version older than 2.0, refer to here to find out how to display the time calculation. profile_tasks, profile_roles, and timer will display a detailed information of the executed tasks.

Before we run the ansible playbook, we need to create a new terminal session so that when our SSH connection is accidentally closed, the command will continue to run in the background. Run this command:

screen -S mongodb_migration

To connect to the terminated terminal session, run this command:

screen -R

Once everything is fully setup, run the ansible playbook with this command:

ansible-playbook playbook.yml

The output will look like this:

root@docker-ubuntu:~/mongodb-migration# ansible-playbook playbook.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Migrate mongodb] ****************************************************************************************************************************************************

TASK [Gathering Facts] ****************************************************************************************************************************************************
Tuesday 31 October 2023  03:00:31 +0000 (0:00:00.042)       0:00:00.042 *******
Tuesday 31 October 2023  03:00:31 +0000 (0:00:00.041)       0:00:00.041 *******
ok: [localhost]

TASK [Remove backup folder if exists] *************************************************************************************************************************************
Tuesday 31 October 2023  03:00:32 +0000 (0:00:01.770)       0:00:01.812 *******
Tuesday 31 October 2023  03:00:32 +0000 (0:00:01.770)       0:00:01.812 *******
changed: [localhost]

TASK [Mongodump from mongodb atlas] ***************************************************************************************************************************************
Tuesday 31 October 2023  03:00:33 +0000 (0:00:00.608)       0:00:02.420 *******
Tuesday 31 October 2023  03:00:33 +0000 (0:00:00.607)       0:00:02.420 *******
changed: [localhost]

TASK [Mongorestore to Digital Ocean managed database - mongodb] ***********************************************************************************************************
Tuesday 31 October 2023  03:04:34 +0000 (0:04:01.372)       0:04:03.793 *******
Tuesday 31 October 2023  03:04:34 +0000 (0:04:01.373)       0:04:03.793 *******
changed: [localhost]

PLAY RECAP ****************************************************************************************************************************************************************
localhost                  : ok=4    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Tuesday 31 October 2023  03:09:43 +0000 (0:05:08.507)       0:09:12.300 *******
===============================================================================
shell ----------------------------------------------------------------- 549.88s
gather_facts ------------------------------------------------------------ 1.77s
ansible.builtin.file ---------------------------------------------------- 0.61s
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
total ----------------------------------------------------------------- 552.26s
Tuesday 31 October 2023  03:09:43 +0000 (0:05:08.508)       0:09:12.301 *******
===============================================================================
Mongorestore to Digital Ocean managed database - mongodb --------------------------------------------------------------------------------------------------------- 308.51s
Mongodump from mongodb atlas ------------------------------------------------------------------------------------------------------------------------------------- 241.37s
Gathering Facts ---------------------------------------------------------------------------------------------------------------------------------------------------- 1.77s
Remove backup folder if exists ------------------------------------------------------------------------------------------------------------------------------------- 0.61s

I have uploaded the ansible files to a repository here.

Feel free to connect with me on LinkedIn:
https://www.linkedin.com/in/budiantoip/