DEV Community

Cover image for Archiving MongoDB Data at Scale: A Practical Guide
Rupanshi Chawda
Rupanshi Chawda

Posted on

Archiving MongoDB Data at Scale: A Practical Guide

#mongodb #microservices #java #backend

Archival is one of those problems that looks simple until your production database starts hitting hundreds of millions of documents. At that point, the naive "fetch everything, dump it somewhere" approach falls apart fast — and you need a system designed for scale from the ground up.

This post covers the key engineering decisions we made when building a MongoDB archival system that handles large datasets without bringing production down.

The Core Problem

Archival sounds simple: identify old data, export it to cold storage, delete it from the production database. But doing this at scale introduces several hard problems:

  • Memory pressure: You can't load millions of documents into the JVM heap.
  • Concurrency: Running archival in a single thread is too slow; running it unconstrained kills your database.
  • Correctness: You need to verify data was safely exported before deleting it from production.
  • Multi-tenancy: In a SaaS system, each tenant's data must be processed independently and in isolation.

Streaming With Cursors, Not Loading Into Memory

The most important decision in the whole system is this: never load all documents into memory.

Instead of:

List<Document> allDocs = mongoTemplate.find(query, Document.class); // OOM waiting to happen
Enter fullscreen mode Exit fullscreen mode

We use MongoDB's streaming cursor:

query.cursorBatchSize(2000); // fetch 2000 docs at a time from the server
Stream<MyEntity> entityStream = mongoTemplate.stream(query, MyEntity.class, collectionName);
Enter fullscreen mode Exit fullscreen mode

The cursor fetches batches of documents from MongoDB on demand. Combined with Java's Stream API, data flows directly from MongoDB into a CSV serializer and then into a cloud storage upload — without ever materializing the full result set in memory.

The pipeline looks like this:

MongoDB Cursor (batch=2000)
    → Stream<Entity>
    → CSV row serializer
    → InputStream (piped)
    → Cloud Storage upload
Enter fullscreen mode Exit fullscreen mode

At any point in time, the JVM holds at most one cursor batch. This keeps memory usage flat regardless of how many documents you're archiving.

A State Machine for Reliability

Because archival spans multiple steps — export, verify, purge — we model each archival job as a state machine:

READY_FOR_ARCHIVAL
    → ARCHIVAL_IN_PROGRESS   (streaming export to cloud storage)
    → READY_FOR_VERIFICATION (export complete, integrity check pending)
    → READY_FOR_PURGE        (verified, safe to delete)
    → PURGED_FROM_TABLE      (deleted from production, metadata saved)
Enter fullscreen mode Exit fullscreen mode

Each stage is independently retryable. If export fails halfway, the run stays in ARCHIVAL_IN_PROGRESS and gets retried without re-running discovery. If verification fails, you don't purge. The state machine makes the system resilient to partial failures without complex rollback logic.

Thread Pools Per Stage

Different stages of archival have different resource profiles. Export is I/O-heavy; purge involves many small delete operations. Rather than using a single shared executor, we run separate fixed thread pools per stage:

@Bean("archivalRunTaskExecutor")
public Executor archivalRunTaskExecutor() {
    return Executors.newFixedThreadPool(5); // concurrent exports
}

@Bean("purgeTaskExecutor")
public Executor purgeTaskExecutor() {
    return Executors.newFixedThreadPool(5); // concurrent deletes
}
Enter fullscreen mode Exit fullscreen mode

When a job trigger fires, it fetches a batch of pending runs and fans them out to the appropriate executor:

pendingRuns.forEach(run ->
    archivalRunTaskExecutor.execute(() -> processArchivalRun(run))
);
Enter fullscreen mode Exit fullscreen mode

This fire-and-forget pattern keeps the job trigger fast and lets the thread pool manage concurrency. You tune pool sizes independently based on observed bottlenecks.

Batched Deletes, Not Bulk Deletes

When purging documents from MongoDB after a verified export, deleting millions of records in one operation is dangerous — it holds locks and can spike replication lag. We delete in small batches:

int DELETE_BATCH_SIZE = 1000;
for (int i = 0; i < ids.size(); i += DELETE_BATCH_SIZE) {
    List<?> batch = ids.subList(i, Math.min(i + DELETE_BATCH_SIZE, ids.size()));
    collection.deleteMany(Criteria.where("_id").in(batch));
}
Enter fullscreen mode Exit fullscreen mode

Each batch is a separate operation. This gives the database breathing room between deletes and keeps replication healthy.

Quota-Based Throttling

In a multi-tenant system, a single large tenant shouldn't be able to exhaust all archival capacity. We track the total number of IDs currently "in-flight" (discovered but not yet purged) and gate new discovery behind a quota check:

long currentInFlight = getInFlightIdsCount();
long available = maxInFlightIds - currentInFlight;

if (available <= 0) {
    return; // backpressure: pause discovery until prior runs complete
}
Enter fullscreen mode Exit fullscreen mode

This acts as natural backpressure. If the system falls behind, discovery pauses until the pipeline drains. It prevents the archival metadata store from ballooning while export/purge catch up.

Verification Before Purge

We never purge based on the assumption that export succeeded. Before any document is deleted, the exported file is read back and IDs are verified against what's in the archival metadata store. To avoid loading the entire CSV into memory, we use a chunked iterator:

try (CsvChunkIterator iterator = getCsvColumnIterator(fileUri, primaryKeyColumn, chunkSizeBytes)) {
    while (iterator.hasNext()) {
        List<String> chunk = iterator.nextChunk();
        verifyIdsExist(runId, chunk); // batch DB lookup
    }
}
Enter fullscreen mode Exit fullscreen mode

Only after all IDs are verified does the run transition to READY_FOR_PURGE. This gives you a hard guarantee: data is only deleted from production once confirmed safe in cold storage.

Key Takeaways

Problem Solution
Memory pressure from large datasets Cursor-based streaming, never load full result set
Partial failure recovery State machine with idempotent, retryable stages
Concurrency without overloading DB Fixed thread pools per stage, tuned independently
Safe deletes at scale Batched deletes with configurable batch size
Runaway discovery In-flight ID quota with backpressure
Correctness guarantees Verify export before purge, chunk-streamed verification

The common theme: never do work in bulk that can cascade. Whether it's memory allocation, database deletes, or thread scheduling — bounding the work at each step is what keeps the system stable under load.

If you're building something similar, the most important thing to get right first is the streaming pipeline. Once you've eliminated in-memory loading, the rest of the system — state machines, thread pools, batching — falls into place naturally.

Top comments (0)