The Problem Nobody Audits Until It Hurts
Data expiry is infrastructure, not cleanup. The choice between MongoDB's built-in TTL mechanism and application-managed expiry determines your write amplification budget, index contention profile, oplog pressure, and what happens to your query plans at 3 AM when the TTL thread wakes up hungry.
This article dissects both approaches at the mechanical level—how TTL indexes actually execute, where they contend with your hot write path, and what application-layer expiry costs you in exchange for control. The tradeoffs are not symmetric and the defaults will bite you in high-throughput services.
How MongoDB TTL Actually Executes
TTL deletion is driven by the TTLMonitor background thread, which runs on a fixed 60-second interval by default (ttlMonitorSleepSecs). Each pass performs a collection scan over every TTL-indexed collection, issues range deletes for documents where the indexed field is older than expireAfterSeconds, and writes those deletes to the oplog.
The key mechanics:
-
TTL uses a regular BSON Date index. Under the hood it is a single-field ascending index with a
expireAfterSecondsproperty. Reads against that index compete with TTL's own range scans. - Deletes are not batched as a single operation. MongoDB issues individual document deletes per expired document internally—each one acquires a document-level write lock, writes a delete to the oplog, and updates all secondary indexes on that collection.
- Oplog amplification is proportional to secondary indexes. A collection with five secondary indexes generates six oplog entries per TTL delete (one delete plus five index key removals). At scale, this taxes replication bandwidth and secondary apply throughput.
-
The TTL thread is not throttled by default before MongoDB 7.0. On 7.0+,
ttlMonitorBatchDeletesand the associated batch size controls allow the thread to yield between batches, which reduces lock contention but extends the actual expiry window beyond the configuredexpireAfterSeconds.
What Contention Looks Like
Imagine a session store: a sessions collection with a TTL index on createdAt (expire after 3600 seconds), plus secondary indexes on userId and deviceId. Write throughput is 8,000 inserts/second during peak.
At second 3600, the TTL thread wakes and begins deleting the cohort from an hour ago. If that cohort represents 500,000 documents, the thread issues 500,000 individual deletes, each touching three index structures. Your oplog rate spikes. Secondary apply lag creeps up. Replica read preferences that route to secondaries start returning stale session data—not expired data, just old data—which can masquerade as authentication bugs.
Meanwhile, your Go service is hammering the same userId index with session lookups. WiredTiger's document-level locking prevents a hard bottleneck there, but index-level operations still share B-tree page locks. At high concurrency, TTL-driven index page splits on the createdAt index cause insert latency spikes precisely when write load is highest: when sessions are being created because users are active.
Application-Layer Expiry: What You Buy and What You Pay
Application-layer expiry means your Go service is responsible for identifying and deleting expired documents, typically via a background goroutine with controlled concurrency.
func runExpiryWorker(ctx context.Context, coll *mongo.Collection) {
ticker := time.NewTicker(30 * time.Second)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if err := deleteExpiredBatch(ctx, coll); err != nil {
log.Printf("expiry batch error: %v", err)
}
}
}
}
func deleteExpiredBatch(ctx context.Context, coll *mongo.Collection) error {
cutoff := time.Now().UTC().Add(-1 * time.Hour)
filter := bson.D{{Key: "createdAt", Value: bson.D{{Key: "$lt", Value: cutoff}}}}
opts := options.Delete().SetHint(bson.D{{Key: "createdAt", Value: 1}})
result, err := coll.DeleteMany(ctx, filter, opts)
if err != nil {
return fmt.Errorf("deleteMany: %w", err)
}
metrics.ExpiredDocumentsDeleted.Add(float64(result.DeletedCount))
return nil
}
This is not simply "the same thing in Go." The behavioral differences are significant:
Scheduling control. You decide when deletion runs and can suspend it during peak traffic windows—something the TTL thread cannot do without changing a global server parameter.
Batching semantics. DeleteMany with a query is still multiple document-level deletes internally, but you control how often you call it. You can implement cursor-based pagination over the expired cohort and delete in capped batches with time.Sleep between iterations, giving WiredTiger checkpoints time to flush and replication time to catch up.
Partial expiry signals. Before deletion, your Go worker can emit metrics, publish events to an SQS queue for audit, or write to an archive collection. TTL gives you nothing between "exists" and "gone."
The cost you pay: you now own the liveness of this goroutine. If the service restarts during a large expiry batch and does not complete it, documents accumulate until the next run. You need observability: a counter on DeletedCount, a gauge on the oldest document age, and an alert when that gauge exceeds 2 * expireAfterSeconds.
Index Design Pressure from Both Approaches
The createdAt index required for TTL (or for application-layer range queries) sits in your hot index set. Consider what this means for insert throughput:
Every insert must update the createdAt B-tree. If createdAt values are monotonically increasing (they almost always are for timestamps), the rightmost leaf of the B-tree is the only page being written to. WiredTiger handles this reasonably with its MVCC design, but at very high insert rates you still observe right-edge contention on that leaf page.
A compound index {userId: 1, createdAt: 1} may serve reads better but is ineligible as a TTL index (TTL requires single-field). This forces you to maintain a separate single-field {createdAt: 1} TTL index alongside your compound read index—two index writes per insert instead of one. Application-layer expiry can use the compound index for its batch deletes, eliminating the redundant single-field index entirely.
Oplog and Replication: The Underestimated Factor
In replica sets with cross-region secondaries (common in AWS multi-AZ or multi-region deployments), oplog apply rate is your replication ceiling. TTL-driven deletes that generate millions of oplog entries in a short window can push secondary apply lag into the tens of seconds. If your service reads from secondaries with readPreference: secondary for scale, those reads will operate on a meaningfully stale view of the data—stale not just by seconds but by the depth of the oplog backlog.
This is a correctness issue when expired documents are security-relevant (sessions, tokens, capability grants). A secondary that is 15 seconds behind oplog apply will serve session documents that the TTL thread already deleted on the primary. Your Go service reading from that secondary considers the session valid.
Application-layer expiry using w: majority write concern for deletes ensures all nodes acknowledge the delete before your worker proceeds—at the cost of latency per batch. The tradeoff: slower expiry, stronger consistency guarantees on cross-node reads.
Decision Framework
Use this matrix when evaluating which approach fits your workload:
| Dimension | TTL Index | Application-Layer Expiry |
|---|---|---|
| Operational overhead | Low — fire and forget | High — goroutine lifecycle, alerting |
| Scheduling control | None (60s global thread) | Full — pause during peak, batch size tuning |
| Secondary index count | Forces extra createdAt index |
Can reuse compound read index |
| Oplog pressure | High — uncontrolled burst | Controlled — sized by batch frequency |
| Pre-delete actions | None | Audit, archive, event emission |
| Replication lag risk | High at scale | Low with batching and sleep |
| Correctness under secondary reads | Risk of stale-valid documents | Controllable with write concern |
Choose TTL when: the collection has at most two secondary indexes, write throughput is below 1,000 documents/second, no pre-delete processing is needed, and replication lag on secondaries is not a correctness concern.
Choose application-layer expiry when: the collection is high-write, carries security-relevant data (sessions, tokens), has three or more secondary indexes, serves reads from secondaries, or requires any audit trail before deletion.
Hybrid approach: Use the TTL index as a safety net with a very long expireAfterSeconds (e.g., 7 days) to prevent unbounded collection growth in case the application worker fails. Set the application worker to delete at the true business expiry window (1 hour). The TTL index almost never fires, eliminating its contention profile while preserving a backstop against worker failure.
Observability Requirements for Application-Layer Expiry
If you own the expiry loop, you own its failure modes. Minimum instrumentation:
-
Gauge: oldest document age in the collection (sampled, not a full scan—use
find({}).sort({createdAt: 1}).limit(1)). - Counter: documents deleted per batch run. A sudden drop to zero means the query plan changed or the index was dropped.
- Histogram: batch duration. Spikes indicate lock contention or oplog backpressure.
- Alert: oldest document age > 2× expiry window. This is your canary for a stalled worker.
Without these, application-layer expiry fails silently and you discover the problem when your collection has grown by an order of magnitude and your index scans have regressed.
Data lifecycle decisions compound. A TTL index that looks free on day one has a compounding cost profile as collection size, secondary index count, and write throughput grow. Model the oplog amplification factor before you ship to production. The hybrid backstop pattern costs almost nothing and eliminates the worst failure mode of application-managed expiry. That is the baseline.
Top comments (0)