TTL Indexes Are Not Records Management

A TTL index deletes documents silently when a date passes. Retention of records needs holds, proof, content deletion and a stub. Here is the difference, with code.

Every few months I meet a document platform where "retention" is implemented like this:

db.documents.createIndex({ "lifecycle.retention.dispositionDue": 1 }, { expireAfterSeconds: 0 })

One line, done. Documents disappear when their disposition date passes. It even sounds like records management.

It is not, and the gap between the two is the gap between a platform that can answer an auditor and one that cannot. This post goes through what a TTL index actually does, operationally, and what a disposition run must do instead. The code is MongoDB, because that is where I work, but the argument is about any time-to-live mechanism in any store.

What a TTL index does, precisely

A TTL index is a single-field index on a date (or an array of dates, in which case the earliest wins) with an expireAfterSeconds option. A background thread on the primary wakes up every 60 seconds, scans each TTL index for documents whose date plus the offset is in the past, and deletes them. The thread is single-threaded, and per index per sub-pass it stops at 50,000 deletions or after one second, whichever comes first, then moves to the next index and comes back later. Since 6.1 the deletes are batched and the monitor is fair across indexes. Secondaries do nothing: they replicate the primary's deletes through the oplog.

Four consequences follow directly from the documentation, and each one matters for records:

No timing guarantee. The manual says it plainly: expired data is not guaranteed to be deleted immediately, and under load may persist well past expiry. For caches that is fine. For a document whose retention expired, "destroyed within the retention schedule" cannot be proven.
No conditions. The monitor evaluates one thing, the date. It cannot check that the document is not under legal hold, that its dossier is not under hold, that the content object has no other references, or that a retention schedule was not revised last week.
No record. The delete leaves nothing behind. db.serverStatus().metrics.ttl gives you deletedDocuments, passes and subPasses as counters, not a log of which documents went and why. A change stream sees a plain delete event, indistinguishable from a user's.
No reach. The index deletes the metadata document. The PDF in object storage, the OCR text in the search projection, the chunks in the vector collection, the rendition cache and the export package someone produced last year are untouched.

There is a fifth, subtler one: documents whose date field is missing or is not a BSON date never expire. A migration that wrote dispositionDue as a string on 3% of records has silently created 3% of records that will live forever, and nothing will tell you.

What records management asks for

Strip the regulatory vocabulary and a records function asks a document platform for four things at disposition time:

Precondition checks that are evaluated at the moment of destruction, not when the date was computed: holds at document and dossier level, schedule still in force, approval where the schedule requires it.
An inventory of everything that constitutes the record: every version, every rendition, every derived copy, with an outcome for each.
Proof: a disposition record stating what was destroyed, when, under which schedule and by which run, that survives the destruction itself.
A stub: the metadata skeleton of the document stays, so that a request five years later gets "destroyed on this date under this schedule" rather than silence.

None of these can be expressed as an index option. They are a job.

A disposition run that holds up

Below is the shape I use in the reference design of my field guide on document management. It is a state machine in two phases. Phase one runs inside a transaction and makes the decision; phase two, outside any transaction and fully retryable, destroys content and finalizes.

javascript
// Phase 1 (transaction): re-check holds at document and dossier level inside the
// transaction, move the record to "disposing", inventory every object of every version,
// decrement reference counts, and write the disposition record.
const due = db.documents.find({
  "lifecycle.retention.dispositionDue": { $lte: new Date() },
  "lifecycle.state": { $in: ["registered", "closed"] },
  "lifecycle.holds": { $size: 0 }
}, { projection: { _id: 1, dossier: 1, "lifecycle.retention": 1 } }).limit(1000);

for await (const d of due) {
  await session.withTransaction(async () => {
    const dossier = await db.collection("dossiers").findOne(
      { _id: d.dossier.id }, { projection: { holds: 1 }, session });
    if (dossier && dossier.holds.length > 0) throw new SkipError("dossier under hold");

    const r = await db.collection("documents").updateOne(
      { _id: d._id, "lifecycle.state": { $in: ["registered", "closed"] },
        "lifecycle.holds": { $size: 0 } },
      { $set: { "lifecycle.state": "disposing", "current.object": null,
                "current.renditions": [] },
        $inc: { recordVersion: 1 } }, { session });
    if (r.matchedCount !== 1) throw new SkipError("state changed concurrently");

    const versions = await db.collection("versions")
      .find({ documentId: d._id }, { projection: { object: 1, renditions: 1 }, session })
      .toArray();
    const keys = versions.flatMap(v => [v.object.key,
                                        ...v.renditions.map(x => x.key).filter(Boolean)]);
    for (const v of versions) {
      await db.collection("objects").updateOne(
        { _id: { tenant: tenantId, sha256: v.object.sha256 } },
        { $inc: { refs: -1 } }, { session });
    }
    await db.collection("dispositions").insertOne({ _id: d._id, at: new Date(), runId,
      scheduleId: d.lifecycle.retention.scheduleId, action: d.lifecycle.retention.action,
      objects: keys.map(k => ({ key: k, deletedAt: null })) }, { session });
  }).catch(e => { if (!(e instanceof SkipError)) throw e; });
}

// Phase 2 (outside any transaction, retryable, run once the longest signed-URL lifetime
// has elapsed): delete objects whose reference count is zero, then finalize the record.
const pending = db.collection("dispositions").find({ "objects.deletedAt": null,
                                                     at: { $lte: urlCutoff } });
for await (const disp of pending) {
  for (const o of disp.objects.filter(o => !o.deletedAt)) {
    const obj = await db.collection("objects").findOne({ key: o.key }, { projection: { refs: 1 } });
    const shared = !!(obj && obj.refs > 0);
    if (!shared) await contentStore.deleteObject(o.key);               // idempotent, logged
    await db.collection("dispositions").updateOne({ _id: disp._id, "objects.key": o.key },
      { $set: { "objects.$.deletedAt": new Date(), "objects.$.shared": shared } });
  }
  await db.collection("documents").updateOne({ _id: disp._id, "lifecycle.state": "disposing" },
    { $set: { "lifecycle.state": "disposed", "lifecycle.disposedAt": new Date() } });
}

Six things in there that a TTL index cannot do:

The hold check runs inside the transaction, on both document and dossier, so a hold placed one second before the run wins.
The state transition is conditional on the current state, so a concurrent run or a concurrent re-registration makes the update match zero documents and the transaction aborts cleanly.
Every version's objects are inventoried, not only the current one.
Reference counts: content is keyed by hash, so two documents can share one object. The object is deleted only when the last reference goes. A TTL index on documents would happily orphan or, worse, keep content that another document's disposition later finds already gone.
Content deletion waits for the longest signed-URL lifetime the platform issues, so a URL handed out just before the run cannot fetch destroyed content; from disposing onwards the document API refuses to issue new ones.
The record ends in disposed with pointers removed, the disposition record attached, class and registration number intact. That is the stub.

The index that serves the run is a partial one, cheap and honest about what it is:

javascript
db.documents.createIndex({ "lifecycle.retention.dispositionDue": 1 },
  { partialFilterExpression: { "lifecycle.state": { $in: ["registered", "closed"] } } })

No expireAfterSeconds. The index finds what is due; the job decides.

Where TTL indexes belong in the same platform

I am not arguing against TTL indexes. In the same reference design they do real work, exactly where their semantics fit:

upload sessions and multipart state, expiring after a few hours;
cached renditions and thumbnails that are regenerated on demand;
signed-URL issuance records, which exist only to be revocable for minutes;
dead-letter entries already reprocessed, after a grace period.

The rule is simple: a TTL index is right when losing the document early, late or silently costs nothing. A record is the definition of a document for which that is false.

A test you can run

If you want to see the difference rather than take it on faith, this is the smallest experiment. Create a collection with a TTL index set to expire immediately, insert a document "under hold", and watch what happens.

javascript
db.ttl_demo.drop();
db.ttl_demo.createIndex({ due: 1 }, { expireAfterSeconds: 0 });
db.ttl_demo.insertMany([
  { _id: "A", due: new Date(Date.now() - 1000), holds: ["LITIGATION-2026-14"] },
  { _id: "B", due: new Date(Date.now() - 1000), holds: [] },
  { _id: "C", due: "2026-10-01T00:00:00Z", holds: [] }        // string, not a date
]);
// wait up to ~60 s, then:
db.ttl_demo.find({}, { _id: 1 });
db.serverStatus().metrics.ttl;

A and B are gone, hold or no hold. C is still there and will be forever. The metrics counter went up by two, and nothing in the database can tell you that A was under a litigation hold when it was destroyed. That is the whole post in six lines.

This is an extract and expansion of Chapter 8 of Document Management: Metadata-Driven Search over Documents and Records, Volume 2 of Real-World Architectures, the free field guides I write on the architectural patterns that decide whether data platforms survive production. The guide covers the rest: registration with an anchored hash chain, legal hold propagation, export packages, the search layer with one permission model, and a readiness checklist. Free PDF, 51 pages: marionoioso.com/real-world-architectures/document-management. Archived with DOI 10.5281/zenodo.23691439.

Views my own. The guide is an independent, non-commercial publication and is not sponsored, reviewed or endorsed by MongoDB, Inc. Code reviewed statically against MongoDB 8.0 documentation; test it in your own environment.