Delete expired documents automatically with MongoDB (TTL index)

Recently I needed to delete some documents that I save in MongoDB after some time. I can think of a few examples why we would want to delete data after some time:

• logs / events
• api keys / access tokens
• non-active users
• etc

We could do it by running a cronjob that deletes the data, delete the data every time we insert new data or any other solution.

Luckily for me, my wife told me that MongoDB already have that mechanism built-in.

TTL index

TTL (Time-To-Live) indexes are special single-field indexes that MongoDB can use to automatically remove documents from a collection after a certain amount of time.

A background thread in mongod reads the values in the index and removes expired documents from the collection (usually every minute).

For example, to create a TTL index on the lastModifiedDate field of the eventlog collection, use the following operation in the mongo shell:

db.eventlog.createIndex( { "lastModifiedDate": 1 }, { expireAfterSeconds: 3600 } )

• The indexed field must be BSON date type or an array of BSON dates
• If the indexed field in a document is not a date or an array that holds a date value(s), the document will not expire.
• If a document does not contain the indexed field, the document will not expire.

Conditional delete

As of MongoDB 3.2, a collection can be partially indexed using a specified filter expression, partialFilterExpression. TTL index can also be used with partial indexes.

For example:

Delete documents that created 1 hour ago and their state is TMP

db.eventlog.createIndex(
  { created_at: 1 },
  { expireAfterSeconds: 3600, partialFilterExpression: { state: 'TMP' } }
);

Delete documents that created 1 day ago and their count is lower than 5

db.eventlog.createIndex(
  { created_at: 1 },
  { expireAfterSeconds: 86400, partialFilterExpression: { count: { $lt: 5 } } }
);

Additional info on partial indexes

Real world example

Recently I finished developing BundleMon, which is a free tool that helps you monitor your apps bundle size.

One of the components in BundleMon is a service that saves historic reports in order to compare bundle size between branches, So when you open a PR BundleMon saves a record with the current bundle size report.

No need to save the report more than 30 days, so I just added a TTL index:

db.reports.createIndex(
  { creationDate: 1 },
  { expireAfterSeconds: 2592000, partialFilterExpression: { prNumber: { $exists: true } } }
);

Comments

[faizan shaikh]

Hi, Liron I have a doubt.

1. For the real-world example do we have to run the query after every 30 days?
2. Can you please tell me how to configure the removal for a specific client? eg if in my collection I had data for different clients but I want to configure the removal query for specific client_id.

[LironEr]

Hey

1. You don't need to run it every 30 days, just once to create the index. A background thread in mongod reads the values in the index and removes expired documents from the collection (usually every minute).
2. If you just want to remove your client from your clients collection:

db.clients.remove({ clientId: "123456"})

You can read more about the remove method here

If you want to auto remove specific client data that is older than X time then you can use TTL index:

db.client_logs.createIndex(
  { creationDate: 1 },
  { expireAfterSeconds: 2592000, partialFilterExpression: { client_id: { $eq: "123456" } } }
);

[amittras-pal]

How can we achieve this with mongoose ??

[LironEr]

stackoverflow.com/a/14597622/2602426