Archiving MongoDB Data at Scale: A Practical Guide

Rupanshi Chawda — Sun, 02 Aug 2026 14:02:07 +0000

Archival is one of those problems that looks simple until your production database starts hitting hundreds of millions of documents. At that point, the naive "fetch everything, dump it somewhere" approach falls apart fast — and you need a system designed for scale from the ground up.

This post covers the key engineering decisions we made when building a MongoDB archival system that handles large datasets without bringing production down.

The Core Problem

Archival sounds simple: identify old data, export it to cold storage, delete it from the production database. But doing this at scale introduces several hard problems:

Memory pressure: You can't load millions of documents into the JVM heap.
Concurrency: Running archival in a single thread is too slow; running it unconstrained kills your database.
Correctness: You need to verify data was safely exported before deleting it from production.
Multi-tenancy: In a SaaS system, each tenant's data must be processed independently and in isolation.

Streaming With Cursors, Not Loading Into Memory

The most important decision in the whole system is this: never load all documents into memory.

Instead of:

List<Document> allDocs = mongoTemplate.find(query, Document.class); // OOM waiting to happen

We use MongoDB's streaming cursor:

query.cursorBatchSize(2000); // fetch 2000 docs at a time from the server
Stream<MyEntity> entityStream = mongoTemplate.stream(query, MyEntity.class, collectionName);

The cursor fetches batches of documents from MongoDB on demand. Combined with Java's Stream API, data flows directly from MongoDB into a CSV serializer and then into a cloud storage upload — without ever materializing the full result set in memory.

The pipeline looks like this:

MongoDB Cursor (batch=2000)
    → Stream<Entity>
    → CSV row serializer
    → InputStream (piped)
    → Cloud Storage upload

At any point in time, the JVM holds at most one cursor batch. This keeps memory usage flat regardless of how many documents you're archiving.

A State Machine for Reliability

Because archival spans multiple steps — export, verify, purge — we model each archival job as a state machine:

READY_FOR_ARCHIVAL
    → ARCHIVAL_IN_PROGRESS   (streaming export to cloud storage)
    → READY_FOR_VERIFICATION (export complete, integrity check pending)
    → READY_FOR_PURGE        (verified, safe to delete)
    → PURGED_FROM_TABLE      (deleted from production, metadata saved)

Each stage is independently retryable. If export fails halfway, the run stays in ARCHIVAL_IN_PROGRESS and gets retried without re-running discovery. If verification fails, you don't purge. The state machine makes the system resilient to partial failures without complex rollback logic.

Thread Pools Per Stage

Different stages of archival have different resource profiles. Export is I/O-heavy; purge involves many small delete operations. Rather than using a single shared executor, we run separate fixed thread pools per stage:

@Bean("archivalRunTaskExecutor")
public Executor archivalRunTaskExecutor() {
    return Executors.newFixedThreadPool(5); // concurrent exports
}

@Bean("purgeTaskExecutor")
public Executor purgeTaskExecutor() {
    return Executors.newFixedThreadPool(5); // concurrent deletes
}

When a job trigger fires, it fetches a batch of pending runs and fans them out to the appropriate executor:

pendingRuns.forEach(run ->
    archivalRunTaskExecutor.execute(() -> processArchivalRun(run))
);

This fire-and-forget pattern keeps the job trigger fast and lets the thread pool manage concurrency. You tune pool sizes independently based on observed bottlenecks.

Batched Deletes, Not Bulk Deletes

When purging documents from MongoDB after a verified export, deleting millions of records in one operation is dangerous — it holds locks and can spike replication lag. We delete in small batches:

int DELETE_BATCH_SIZE = 1000;
for (int i = 0; i < ids.size(); i += DELETE_BATCH_SIZE) {
    List<?> batch = ids.subList(i, Math.min(i + DELETE_BATCH_SIZE, ids.size()));
    collection.deleteMany(Criteria.where("_id").in(batch));
}

Each batch is a separate operation. This gives the database breathing room between deletes and keeps replication healthy.

Quota-Based Throttling

In a multi-tenant system, a single large tenant shouldn't be able to exhaust all archival capacity. We track the total number of IDs currently "in-flight" (discovered but not yet purged) and gate new discovery behind a quota check:

long currentInFlight = getInFlightIdsCount();
long available = maxInFlightIds - currentInFlight;

if (available <= 0) {
    return; // backpressure: pause discovery until prior runs complete
}

This acts as natural backpressure. If the system falls behind, discovery pauses until the pipeline drains. It prevents the archival metadata store from ballooning while export/purge catch up.

Verification Before Purge

We never purge based on the assumption that export succeeded. Before any document is deleted, the exported file is read back and IDs are verified against what's in the archival metadata store. To avoid loading the entire CSV into memory, we use a chunked iterator:

try (CsvChunkIterator iterator = getCsvColumnIterator(fileUri, primaryKeyColumn, chunkSizeBytes)) {
    while (iterator.hasNext()) {
        List<String> chunk = iterator.nextChunk();
        verifyIdsExist(runId, chunk); // batch DB lookup
    }
}

Only after all IDs are verified does the run transition to READY_FOR_PURGE. This gives you a hard guarantee: data is only deleted from production once confirmed safe in cold storage.

Key Takeaways

Problem	Solution
Memory pressure from large datasets	Cursor-based streaming, never load full result set
Partial failure recovery	State machine with idempotent, retryable stages
Concurrency without overloading DB	Fixed thread pools per stage, tuned independently
Safe deletes at scale	Batched deletes with configurable batch size
Runaway discovery	In-flight ID quota with backpressure
Correctness guarantees	Verify export before purge, chunk-streamed verification

The common theme: never do work in bulk that can cascade. Whether it's memory allocation, database deletes, or thread scheduling — bounding the work at each step is what keeps the system stable under load.

If you're building something similar, the most important thing to get right first is the streaming pipeline. Once you've eliminated in-memory loading, the rest of the system — state machines, thread pools, batching — falls into place naturally.

Flutter PDF Viewer - Automated using GitHubActions

Rupanshi Chawda — Tue, 15 Sep 2020 17:27:01 +0000

View PDF directly into your Flutter App using - flutter_plugin_pdf_viewer .

You can implement this using the flutter plugin and load a PDF directly into your Flutter App. It works offline too as it downloads the pdf and then loads it into the app.

My Workflow

The GitHub Action PDF_Flutter CI is designed to automate certain tasks for the developers making their life less complicated.

Configuring Flutter version
Installing dependencies
Run Test

Submission Category:

Maintainer Must-Haves and Wacky Wildcards

Yaml File and Link to Code

name: PDF_Flutter CI

# This workflow is triggered on pushes to the repository.

on:
  push:
    branches:
    - master

# on: push    # Default will running for every branch.

jobs:
  build:
    # This job will run on ubuntu virtual machine
    runs-on: ubuntu-latest
    steps:

    # Setup Java environment in order to build the Android app.
    - uses: actions/checkout@v1
    - uses: actions/setup-java@v1
      with:
        java-version: '12.x'

    # Setup the flutter environment.
    - uses: subosito/flutter-action@v1
      with:
        channel: 'stable' # 'dev', 'alpha', default to: 'stable'
        flutter-version: '1.17.5' # you can also specify exact version of flutter

    # Get flutter dependencies.
    - name: Install dependencies
      run: flutter pub get
    - name: run tests
      run: flutter pub run tests

rupanshi-chawda / Pdf-Viewer-in-Flutter

Flutter PDF Viewer

View PDF directly into your Flutter App using -

flutter_plugin_pdf_viewer

Implementation -

You can implement this using the flutter plugin and load a PDF directly into your Flutter App. It works offline too as it downloads the pdf and then loads it into the app.

View on GitHub

Additional Resources / Info

subosito / flutter-action

Flutter environment for use in actions. It works on Linux, Windows, and macOS.

flutter-action

This action sets up a flutter environment for use in actions. It works on Linux, Windows, and macOS.

Usage

steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
  with:
    distribution: 'zulu'
    java-version: '11'
- uses: subosito/flutter-action@v1
  with:
    flutter-version: '2.0.5'
- run: flutter pub get
- run: flutter test

Use latest release for particular channel:

steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
  with:
    distribution: 'zulu'
    java-version: '11'
- uses: subosito/flutter-action@v1
  with:
    channel: 'stable' # or: 'beta', 'dev' or 'master'
- run: flutter pub get
- run: flutter test
- run: flutter build apk

Use latest release for particular version and/or channel:

steps
- uses: actions/checkout@v2
- uses: actions/setup-java@v2
  with:
    distribution: 'zulu

…