DEV Community: Jagannath Shingne

Automated MongoDB Backups — Production‑Ready Guide

Jagannath Shingne — Fri, 06 Feb 2026 14:16:04 +0000

Why Backups Matter (More Than You Think)

Production databases will fail — human error, cloud outages, bad deployments, or accidental deletes. The only real safety net is a reliable, automated backup system that you trust enough to restore from at 3 AM.

In this guide, we’ll walk through a battle‑tested MongoDB backup system built using:

mongodump / mongorestore
Node.js + node-cron
Amazon S3
MongoDB (as backup metadata store)

By the end, you’ll understand how to safely back up MongoDB, store it in S3, clean old backups, and fully restore your database if disaster strikes.

System Goals

We wanted a system that:

Backs up multiple MongoDB databases
Runs daily, weekly, and monthly automatically
Stores backups safely in S3
Tracks everything in MongoDB
Deletes expired backups automatically
Can fully restore a deleted database

No manual steps. No risky shortcuts.

Core Concept: MongoDB Dumps

MongoDB provides a native backup tool called mongodump.

We use it like this:

mongodump --uri="<MONGO_URI>" --archive --gzip --oplog

What this does

--archive → outputs a single file (easy to store)
--gzip → compresses the backup
--oplog → ensures consistent snapshots for replica sets

Result: one .archive.gz file per backup.

This backup is read‑only — your original database is never modified.

Supporting Multiple Databases

Instead of hard‑coding one database, we define a generic list of MongoDB connections:

const databases = [
  { name: "app-db", uri: process.env.MONGO_URI_1 },
  { name: "analytics-db", uri: process.env.MONGO_URI_2 }
];

These names are examples only and can represent any MongoDB database in any environment.

Each database:

Uses the same backup logic
Generates its own backup file
Stores metadata separately

This approach keeps the system *generic and reusable*.

Automated Cron Jobs

All backups run inside the Node.js app using node-cron.

Job Type	Schedule
Daily	Every day
Weekly	Every Sunday
Monthly	1st day of month
Cleanup	Every day

Each cron:

Runs mongodump
Uploads the file to S3
Saves metadata in MongoDB

No external cron servers required.

S3 Storage Structure

Backups are uploaded as .archive.gz files to an S3 bucket.

A generic and recommended folder structure looks like this:

s3://mongo-backups/
  {dbName}/
    daily/
    weekly/
    monthly/

Where:

{dbName} → sanitized MongoDB database name
daily | weekly | monthly → backup frequency

Production Safety Rules

Database names are sanitized (lowercase, no spaces)
Files are uploaded only after dump succeeds
Local files are deleted only after S3 upload succeeds

This structure works for any project or organization and scales cleanly.

Backup Metadata in MongoDB

MongoDB stores generic backup metadata, independent of any company or product:

{
  dbName: "app-db",
  backupType: "daily",
  storageProvider: "s3",
  storageKey: "mongo-backups/app-db/daily/backup.archive.gz",
  createdAt: ISODate(),
  expireAt: ISODate()
}

Why this matters:

MongoDB becomes the source of truth
Every backup is auditable
Cleanup logic is safe and deterministic

Nothing is deleted unless it is explicitly tracked.

Automatic Cleanup of Expired Backups

A separate daily cron job:

Finds backups where expireAt < now
Deletes the file from S3
Deletes the record from MongoDB

If it’s not in MongoDB → it’s not deleted.

This prevents:

Accidental data loss
Unlimited storage growth

Disaster Recovery (Restore Process)

If any MongoDB database is deleted or corrupted:

Step 1: Download the latest backup

aws s3 cp s3://mongo-backups/{dbName}/daily/latest.archive.gz .

Step 2: Restore using `mongorestore`

mongorestore --gzip --archive=latest.archive.gz --drop

--drop clears existing data
The database is rebuilt from the archive

This process works for any MongoDB deployment.

Safety Guarantees

✔ Original DB is never modified
✔ Backups are full & consistent
✔ Storage growth is controlled
✔ Every backup is tracked
✔ Restore is always possible

This is production‑grade, not a demo script.

Final Thought

This guide demonstrated a generic, production‑safe MongoDB backup strategy using mongodump, scheduled jobs, S3 storage, MongoDB‑tracked metadata, automated cleanup, and a reliable restore flow — adaptable to any project or organization.

Happy building 🚀

From Synchronous to Scalable: Redesigning Video Processing with RabbitMQ

Jagannath Shingne — Wed, 04 Feb 2026 14:08:48 +0000

Introduction

Handling video at scale is never trivial. What starts as a simple upload-and-process flow can quickly turn into a performance bottleneck as video length, user traffic, and processing complexity grow.

In our case, we faced a major challenge when increasing video durations began to slow down our APIs and degrade the user experience.

This blog walks through the problem we faced with synchronous video processing, why it didn’t scale, and how we redesigned the system into a RabbitMQ-driven asynchronous pipeline that significantly improved performance and reliability.

The Original Problem

Initially, our backend followed a fully synchronous video processing approach:

The user recorded a video on the client.
The full video was sent to the backend API.
The API handled everything:
- Uploading the video to S3
- Processing the video
- Generating subtitles
- Uploading the subtitle .vtt file to S3
- Updating the database with both URLs
Only after all these steps completed did the API return a success response.

This approach worked fine when video lengths were limited to 5–6 minutes.

However, as interview durations gradually increased, this design started showing serious limitations.

Why This Didn’t Scale

As video length grew, so did the problems:

Long API response times due to heavy video processing
APIs blocked until FFmpeg and subtitle generation finished
Poor user experience, as users had to wait until everything completed
Limited scalability as traffic increased

The API was doing too much. Video processing, cloud uploads, and database updates were tightly coupled with user-facing requests.

We needed a better architecture.

The New Scalable Architecture

To overcome these limitations, we redesigned the system with asynchronous processing and background workers.

1. Chunked Video Upload with Presigned URLs

Instead of sending the full video to the API:

The client records the video in small chunks.
Each chunk is uploaded directly to S3 using presigned URLs.
This removes the API entirely from the upload path.

Once the last chunk is uploaded, the client sends a lightweight request to mark the interview as completed.

2. Asynchronous Processing with RabbitMQ

After interview completion:

The API publishes a message to RabbitMQ.
The request immediately returns success to the user.
All heavy processing happens in the background.

We introduced two dedicated RabbitMQ workers.

Video Merge Worker

Fetches all uploaded chunks from S3
Uses FFmpeg to merge them into a single video
Uploads the final video back to S3
Updates the database with the merged video URL

Subtitle Worker

Downloads the final merged video from S3
Extracts audio from the video
Converts audio into subtitles (.vtt)
Uploads the .vtt file to S3
Updates the database with the subtitle URL

Each worker is isolated, scalable, and focused on a single responsibility.

Key Benefits

This redesign brought immediate and measurable improvements:

80–90% reduction in API response time
Heavy CPU tasks moved out of the API layer
Fully asynchronous, fault-tolerant processing
Easily scalable by adding more workers
No waiting time for users after finishing the interview
Support for more than 30-minute videos with zero downtime

Users can now finish their interview and move on instantly, while processing continues quietly in the background.

Final Thoughts

This shift from a synchronous to an asynchronous architecture was a turning point.

By decoupling video uploads and processing from the API layer and leveraging RabbitMQ workers, we built a system that is faster, more scalable, and far more resilient.

Redesigned synchronous video processing into a RabbitMQ-driven asynchronous pipeline, cutting API response time by ~70–80% while enabling scalable background processing for video merging, subtitle generation, and cloud storage.

If you’re dealing with long-running tasks like video processing, this pattern can dramatically improve both performance and user experience.