DEV Community: Ziaul Hoque

How I Built a Bulletproof MongoDB Backup System That Saved My Startup

Ziaul Hoque — Mon, 01 Sep 2025 14:32:25 +0000

A developer's journey from backup anxiety to peaceful sleep

Picture this: It's 2 AM, and you're lying awake wondering, "What if our database crashes tomorrow?" If you're running a MongoDB instance in production, this scenario has probably crossed your mind more than once.

As developers, we know backups are crucial, but let's be honest – manual backups are a recipe for disaster. We get busy, we forget, and suddenly we're one hardware failure away from losing everything our users trust us with.

That's exactly where I found myself six months ago. Our startup was growing, our MongoDB database was becoming more critical by the day, and I realized we needed a backup solution that worked even when I was on vacation (or simply human and forgot to run it manually).

The Solution That Changed Everything

After researching various approaches, I built an automated backup system that creates daily snapshots of our MongoDB database and stores them safely in Google Drive. The best part? It runs completely hands-off, and I sleep better knowing our data is protected.

Here's how you can build the same peace of mind into your infrastructure:

What You'll Need

Before we dive in, make sure you have:

A MongoDB instance running in Docker with exposed ports
MongoDB Database Tools installed on your Ubuntu server
A Google account for cloud storage
About 30 minutes to set this up once

Step 1: Setting Up rclone (Your Cloud Storage Bridge)

The magic happens with a tool called rclone – think of it as the Swiss Army knife for cloud storage management. Installation is surprisingly simple:

curl https://rclone.org/install.sh | sudo bash

Step 2: Connecting to Google Drive (The Tricky Part Made Simple)

This step used to intimidate me, but it's actually straightforward once you know the process. Since most servers don't have web browsers, we'll use what's called "headless" authentication:

Start the configuration:
Follow these exact steps:
The authentication dance:
Confirm and you're done!

Pro tip: This one-time setup might feel tedious, but you'll thank yourself later when backups are running silently in the background.

Step 3: The Backup Script That Does the Heavy Lifting

Here's where the real magic happens. I created a script that handles everything: dumping the database, uploading to the cloud, and cleaning up afterwards.

Create your script file:

nano ~/backup_mongo.sh

Here's the complete script (remember to update the credentials):

#!/bin/bash

# --- Configuration ---
# Database credentials (replace with your actual values)
MONGO_USER="your_admin_user"
MONGO_PASS="your_super_strong_password"

# Local backup directory
BACKUP_DIR="/home/mongo_backups"

# rclone Configuration
RCLONE_REMOTE="gdrive"
DRIVE_BACKUP_FOLDER="MongoDB_Backups"

# --- The Magic Happens Here ---
echo "Starting MongoDB backup..."

mkdir -p $BACKUP_DIR

TIMESTAMP=$(date +"%Y-%m-%d_%H-%M-%S")
BACKUP_FILE="$BACKUP_DIR/mongodb_backup_$TIMESTAMP.gz"

# Create the database dump
echo "Creating database dump: $BACKUP_FILE"
mongodump \
  --host=localhost \
  --port=27017 \
  --username=$MONGO_USER \
  --password=$MONGO_PASS \
  --authenticationDatabase=admin \
  --archive="$BACKUP_FILE" \
  --gzip

# Check if the dump worked
if [ $? -eq 0 ]; then
  echo "Database dump successful."

  # Upload to Google Drive
  echo "Uploading to Google Drive..."
  rclone copy "$BACKUP_FILE" "${RCLONE_REMOTE}:${DRIVE_BACKUP_FOLDER}" --progress

  if [ $? -eq 0 ]; then
    echo "Upload successful."

    # Clean up local file to save space
    rm "$BACKUP_FILE"
    echo "Backup complete!"
  else
    echo "rclone upload failed!"
  fi
else
  echo "mongodump failed!"
fi

Make it executable:

chmod +x ~/backup_mongo.sh

Step 4: Set It and Forget It with Cron

The final piece is automation. I set mine to run at 3 AM Bangladesh time (when our traffic is lowest):

crontab -e

Add this line:

0 21 * * * /home/revuers/backup_mongo.sh > /home/revuers/mongo_backup.log 2>&1

Why 21:00 UTC? That's 3:00 AM in Bangladesh – perfect timing for minimal disruption.

The Peace of Mind Payoff

Six months later, this system has:

Created 180+ automated backups without me lifting a finger
Survived server maintenance, power outages, and my vacation to Cox's Bazar
Saved me countless hours of manual backup procedures

Lessons Learned

Start simple, then optimize: My first version was much more complex. This stripped-down approach has proven more reliable.
Monitor but don't micromanage: The log file lets me spot-check without obsessing over every backup.
Test your restores: Having backups means nothing if you can't restore from them. Test this process before you need it.

What's Next?

I'm considering adding Slack notifications for backup failures and implementing a retention policy to automatically delete old backups. But honestly? Sometimes the simple solution that just works is perfect as-is.

Have you implemented automated backups for your databases? What challenges did you face, and what solutions worked for you? I'd love to hear about your experiences in the comments below.

If this helped you sleep better at night knowing your data is safe, give it a 👍 and share it with your developer friends who might need this peace of mind too.

Why Our "Perfect" MongoDB Setup Failed in Production (And How We Fixed It)

Ziaul Hoque — Sun, 31 Aug 2025 08:37:11 +0000

Three weeks before our product launch, everything seemed perfect. Our app was running smoothly in development, Prisma was handling our database transactions beautifully, and we were confident about going live.

Then we deployed to production.

"Transaction failed: not in a replica set" – six words that turned our launch week into a debugging marathon. Sound familiar?

If you've ever attempted to run MongoDB transactions in production, you're familiar with this pain. What works perfectly in development suddenly breaks when you need it most. Today, I'm sharing the production-ready MongoDB setup that finally solved this puzzle for us.

The Problem That Blindsided Us

Here's what happened: Our development environment used a simple MongoDB instance. But when we tried to use Prisma transactions in production, everything crashed. The issue? MongoDB requires a replica set for transactions to function, even in a single-node setup.

This isn't clearly documented anywhere, and it caught our entire team off guard. We weren't alone – I've since learned this trips up countless development teams during their first production deployment.

The Solution That Actually Works
After three days of research, testing, and some very late nights, we built a bulletproof MongoDB 8.0 setup that handles transactions, provides proper security, and persists data reliably. Here's exactly how we did it:

Step 1: Getting Organized (The Foundation)
First, we created a clean project structure with secure credential management:

mkdir -p ~/mongodb_prod/mongo-key
cd ~/mongodb_prod

Then we set up environment variables for our sensitive data:

nano .env

In the .env file:

# .env
MONGO_ROOT_USER=your_admin_user
MONGO_ROOT_PASS=your_super_strong_and_secret_password

Pro tip: Never hardcode these credentials in your Docker files. I learned this the hard way when I accidentally committed passwords to our repo.

Step 2: The Security KeyFile (Critical for Production)
This step stumped us initially. Replica sets need a security keyFile for internal communication, even in single-node setups:

# Generate the security key
openssl rand -base64 756 > ./mongo-key/mongodb.key

## Set proper permissions (this is crucial!)
chmod 400 ./mongo-key/mongodb.key
sudo chown 999:999 ./mongo-key/mongodb.key

Why user ID 999? That's the MongoDB user inside the Docker container. Getting this wrong will cause authentication failures that are incredibly frustrating to debug.

Step 3: The Docker Compose Configuration That Works
After multiple failed attempts, here's the configuration that finally worked:

services:
  mongodb:
    image: mongo:8.0
    container_name: mongodb_prod
    hostname: mongodb_prod
    restart: always
    ports:
      - "27017:27017"
    environment:
    # Reads credentials from the .env file
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_ROOT_USER}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_ROOT_PASS}
    # The magic command that enables replica set
      command: ["--replSet", "rs0", "--keyFile", "/etc/secrets/mongodb.key"]
    volumes:
      # Persistent database data
      - mongo-data:/data/db
      # Persistent config data
      - mongo-config:/data/configdb
      # Security keyFile mount
      - ./mongo-key/mongodb.key:/etc/secrets/mongodb.key:ro

volumes:
  mongo-data:
    driver: local
  mongo-config:
    driver: local

Key insights:

The --replSet rs0 parameter is what enables transactions

The hostname must match what you use in the replica set configuration

Named volumes ensure your data survives container restarts

Step 4: The One-Time Initialization
This is where many tutorials fail you. After starting the container, you need to initialize the replica set:

# Start the container
docker compose up -d

# Wait 20-30 seconds for initialization
sleep 30

# Initialize the replica set (replace with your actual credentials)
docker exec mongodb_prod mongosh \
  --username your_admin_user \
  --password your_super_strong_and_secret_password \
  --authenticationDatabase admin \
  --eval "rs.initiate({ _id: 'rs0', members: [{ _id: 0, host: 'mongodb_prod:27017' }]})"

Look for "ok": 1 in the output – that's your confirmation that everything worked.

Step 5: The Connection String That Changes Everything
Here's the connection string format that finally made our Prisma transactions work:

DATABASE_URL="mongodb://your_admin_user:your_super_strong_password@mongodb_prod:27017/your_db_name?authSource=admin&replicaSet=rs0"

The crucial parts:

authSource=admin – tells MongoDB where to authenticate
replicaSet=rs0 – enables transaction support

Lessons Learned the Hard Way

Test your production setup early: Don't wait until launch week to discover replica set requirements
Document your deployment process: Future you (and your team) will thank you
Monitor your setup: Use docker compose logs -f mongodb_prod to catch issues early
Backup from day one: Even with persistent volumes, regular backups saved us during a server migration

What's Next?

This setup forms the foundation of our current infrastructure. We've since added monitoring with Prometheus, automated backups (remember my previous article?), and performance optimization. But this core configuration remains unchanged – it just works.

Have you faced similar MongoDB production challenges? What setup works best for your team? I'd love to hear about your experiences and any optimizations you've discovered.

If this saved you from deployment headaches, give it a 👍 and share it with other developers who might be struggling with MongoDB transactions in production.

The Critical Gap in Next.js SEO: Implementing Sitemap Index for Enterprise Applications

Ziaul Hoque — Sat, 30 Aug 2025 20:09:08 +0000

Next.js has revolutionized how we build modern web applications, providing sophisticated built-in sitemap generation capabilities through the generateSitemaps function. However, even the most comprehensive frameworks have limitations, and Next.js has one significant oversight that can impact large-scale applications: the absence of sitemap index functionality.

For enterprise applications managing thousands or millions of pages, this limitation creates substantial SEO management challenges. While Next.js excels at generating multiple sitemaps, it leaves developers to manually coordinate these sitemaps with search engines—a process that becomes increasingly complex as applications scale.

Understanding Next.js Sitemap Capabilities

Before addressing the limitation, it's important to recognize what Next.js does exceptionally well. The framework provides robust sitemap generation through two primary mechanisms:

Static Sitemap Generation allows developers to create sitemaps for relatively stable content using the standard sitemap.js file convention. This approach works perfectly for pages like home, about, contact, and other foundational site content.

Dynamic Multiple Sitemaps represent Next.js's most powerful sitemap feature. Using the generateSitemaps function, developers can programmatically create multiple sitemaps by returning an array of objects containing sitemap identifiers. This functionality enables applications to split large datasets across multiple sitemaps, staying within Google's recommended limits of 50,000 URLs per sitemap.

The implementation follows a clean pattern: the generateSitemaps function determines how many sitemaps are needed based on data volume, while the default sitemap function generates content for each individual sitemap using the provided ID parameter.

The Missing Piece: Sitemap Index

Despite these powerful capabilities, Next.js notably lacks built-in support for sitemap index files. A sitemap index serves as a master directory that references all individual sitemaps on a website, providing search engines with a single entry point to discover and crawl all site content.

This omission creates several operational challenges:

Manual Search Console Management becomes necessary when dealing with multiple sitemaps. Each individual sitemap must be manually submitted to Google Search Console, creating administrative overhead that scales poorly.

Discovery Complexity increases as applications grow. Search engines cannot automatically discover new sitemaps when content expands, requiring ongoing manual intervention.

Maintenance Overhead compounds over time. As applications add new content types or expand existing ones, developers must continuously track and manage additional sitemaps.

Scalability Constraints emerge in enterprise environments where applications may generate dozens or hundreds of individual sitemaps across different content categories.

Architectural Solution: Custom Sitemap Index Implementation

The solution involves creating a custom sitemap index that bridges Next.js's native capabilities with proper SEO management. This approach leverages Next.js's existing strengths while addressing the framework's limitation.

The architecture follows a hierarchical structure where a single sitemap index file references all individual sitemaps generated through Next.js's native functionality. This creates a unified entry point for search engines while maintaining the performance benefits of Next.js's built-in sitemap generation.

Implementation Foundation

The implementation begins with standard Next.js sitemap setup. Static pages utilize the conventional sitemap.js file, returning an array of URL objects with appropriate metadata including last modification dates, change frequencies, and priority values.

Dynamic content leverages Next.js's generateSitemaps functionality. The generateSitemaps function calculates the total number of required sitemaps based on content volume, returning an array of identifier objects. The accompanying sitemap function then generates individual sitemaps using these identifiers to fetch and format the appropriate content subsets.

// Static sitemap following Next.js conventions
export default function sitemap(): MetadataRoute.Sitemap {
  return [
    {
      url: 'https://yoursite.com',
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 1,
    },
    {
      url: 'https://yoursite.com/about',
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 0.8,
    },
  ]
}

// Dynamic sitemaps using Next.js generateSitemaps
export async function generateSitemaps() {
  const totalCompanies = await getCompanyCount() // Custom count function
  const companiesPerSitemap = 45000
  const numberOfSitemaps = Math.ceil(totalCompanies / companiesPerSitemap)

  return Array.from({ length: numberOfSitemaps }, (_, i) => ({ id: i }))
}

export default async function sitemap({ id }: { id: number }): Promise<MetadataRoute.Sitemap> {
  const start = id * 45000
  const companies = await getCompanies(start, 45000) // Custom fetch function

  return companies.map((company) => ({
    url: `https://yoursite.com/company/${company.slug}`,
    lastModified: company.updated_at,
    changeFrequency: 'weekly',
    priority: 0.8,
  }))
}

Custom Sitemap Index Route

The critical component is a custom route that generates the sitemap index XML. This route calculates the total number of existing sitemaps and creates a properly formatted sitemap index that references all individual sitemaps.

// Custom sitemap index route
async function getCompanyCount() {
  // Implementation to retrieve total company count
}

export async function GET() {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || "https://yoursite.com"
  const companiesPerSitemap = 45000

  try {
    const totalCompanies = await getCompanyCount()
    const numberOfCompanySitemaps = Math.ceil(totalCompanies / companiesPerSitemap)
    const currentDate = new Date().toISOString().split("T")[0]

    let sitemapIndexXml = `<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">`

    // Reference the static sitemap
    sitemapIndexXml += `
  <sitemap>
    <loc>${baseUrl}/sitemap.xml</loc>
    <lastmod>${currentDate}</lastmod>
  </sitemap>`

    // Reference all dynamic company sitemaps
    for (let i = 0; i < numberOfCompanySitemaps; i++) {
      sitemapIndexXml += `
  <sitemap>
    <loc>${baseUrl}/company/sitemap/${i}.xml</loc>
    <lastmod>${currentDate}</lastmod>
  </sitemap>`
    }

    sitemapIndexXml += `
</sitemapindex>`

    return new Response(sitemapIndexXml, {
      headers: {
        "Content-Type": "application/xml",
        "Cache-Control": "public, max-age=3600, s-maxage=3600",
      },
    })
  } catch (error) {
    console.error("Error generating sitemap index:", error)

    // Provide fallback with minimal sitemap index
    const fallbackXml = `<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>${baseUrl}/sitemap.xml</loc>
    <lastmod>${new Date().toISOString().split("T")[0]}</lastmod>
  </sitemap>
</sitemapindex>`

    return new Response(fallbackXml, {
      headers: {
        "Content-Type": "application/xml",
        "Cache-Control": "public, max-age=300",
      },
    })
  }
}

Operational Benefits

This implementation delivers significant operational improvements over manual sitemap management approaches.

Unified Management replaces the complexity of tracking multiple individual sitemaps with a single sitemap index submission to Google Search Console. This dramatically reduces administrative overhead while ensuring comprehensive search engine coverage.

Automatic Scaling eliminates manual intervention when content volume changes. As applications add new companies, products, or other dynamic content, the sitemap index automatically reflects these additions without requiring developer intervention or additional search console submissions.

Enhanced Discovery provides search engines with immediate access to all site content through a single entry point. This improves crawl efficiency and ensures that new content is discovered promptly as it becomes available.

Future Extensibility creates a foundation for adding additional content types without architectural changes. New sitemap categories can be seamlessly integrated into the existing index structure.

Search Engine Integration

The robots.txt file requires a single modification to reference the sitemap index:

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap-index.xml

Google Search Console integration becomes remarkably straightforward, requiring only the submission of the single sitemap index URL. Search engines then automatically discover and process all referenced individual sitemaps, providing comprehensive coverage with minimal configuration.

Performance Considerations

The implementation maintains optimal performance characteristics through several design decisions. Individual sitemaps remain within recommended size limits, preventing memory issues and ensuring rapid generation times. The sitemap index itself is lightweight, consisting primarily of URL references rather than content data.

Caching strategies at both the application and CDN levels ensure that sitemap index generation doesn't impact site performance, while the modular architecture allows for efficient updates when content changes occur.

Extending the Architecture

The foundation readily accommodates additional content types. Blog posts, product catalogs, or other dynamic content can be integrated by adding corresponding sitemap calculations and references to the index generation logic.

// Example extension for blog content
const totalBlogs = await getBlogCount()
const blogSitemaps = Math.ceil(totalBlogs / blogsPerSitemap)

for (let i = 0; i < blogSitemaps; i++) {
  sitemapIndexXml += `
  <sitemap>
    <loc>${baseUrl}/blog/sitemap/${i}.xml</loc>
    <lastmod>${currentDate}</lastmod>
  </sitemap>`
}

This extensibility ensures that the architecture scales naturally as applications evolve and expand their content offerings.

Conclusion

Next.js provides exceptional sitemap generation capabilities through its native generateSitemaps functionality, enabling developers to create sophisticated, scalable sitemap solutions. However, the framework's lack of built-in sitemap index support creates a significant gap for enterprise applications managing large volumes of content.

The custom sitemap index implementation presented here bridges this gap effectively, combining Next.js's native strengths with proper SEO management practices. By creating a unified entry point for search engine discovery, this approach eliminates manual sitemap management overhead while ensuring optimal search engine coverage.

For development teams building large-scale Next.js applications, implementing this sitemap index solution transforms SEO management from a complex, ongoing maintenance task into an automated, scalable system. The result is improved search engine visibility, reduced operational overhead, and a foundation that grows seamlessly with application content.

The investment in implementing this functionality pays dividends immediately through simplified search console management and continues to provide value as applications scale, making it an essential component of any enterprise Next.js SEO strategy.