Time to Live (TTL) in Azure Cosmos DB

#azure #cosmosdb

TTL is a powerful feature that automatically deletes items from a container after a specified time period. This feature helps manage storage, reduce costs, and keep data fresh. Works at both the container and individual item level.

How to set it up

Azure Portal

Open the Azure Portal and navigate to your Cosmos DB account.
In the left-hand menu, go to the Data Explorer.
Expand your database and select the container where you want to apply TTL
Click on Scale & Settings under the selected container.
Go to the Settings tab (as shown in the screenshot).
Under the Time to Live section:
- Select On to enable TTL for the container.
- Optionally, you can:
  - Select "On (no default)" to require TTL per item.
  - Select "On" and enter a value in seconds in the box below to set a default TTL duration for all items.
Click Save at the top to apply your changes.

Azure CLI

az cosmosdb sql container update \
    --account-name <your-account-name> \
    --database-name <your-database-name> \
    --name <your-container-name> \
    --resource-group <your-resource-group> \
    --default-ttl <new-ttl-value>

Replace <your-account-name>, <your-database-name>, <your-container-name>, <your-resource-group>, and <new-ttl-value> with your specific values. Setting default-ttl to -1 enables TTL without a default expiration, 0 disables TTL, and any positive integer sets the default TTL in seconds.

Bicep file

resource cosmosDbAccount 'Microsoft.DocumentDB/databaseAccounts@2021-04-15' = {
  name: 'myCosmosDbAccount'
  location: 'West US'
  kind: 'GlobalDocumentDB'
  properties: {
    consistencyPolicy: {
      defaultConsistencyLevel: 'Session'
    }
    locations: [
      {
        locationName: 'West US'
        failoverPriority: 0
      }
    ]
  }
}

resource cosmosDbDatabase 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases@2021-04-15' = {
  name: '${cosmosDbAccount.name}/myDatabase'
  properties: {
    resource: {
      id: 'myDatabase'
    }
  }
}

resource cosmosDbContainer 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers@2021-04-15' = {
  name: '${cosmosDbDatabase.name}/myContainer'
  properties: {
    resource: {
      id: 'myContainer'
      partitionKey: {
        paths: ['/myPartitionKey']
        kind: 'Hash'
      }
      defaultTtl: -1  // TTL set to "On (no default)"
    }
  }
}

The defaultTtl attribute for Azure Cosmos DB containers can have the following values:

-1: This value enables TTL without a default expiration time. You can then set the TTL for individual documents within the container.
0: This value disables TTL for the container.
Positive Integer: Any positive integer value sets the default TTL in seconds for all documents in the container. For example, a value of 3600 sets the default TTL to 1 hour.

How ttl and _ts attribute interact

_ts attribute is a system-generated timestamp that records the last modification time of a document in seconds since the Unix epoch (January 1, 1970). It helps track when a document was last updated.

When a document is created or modified, the _ts value is updated to the current time.
The ttl value is set to define how long the document should be retained.
Cosmos DB periodically checks the _ts value against the ttl value. If the difference between the current time and the _ts value exceeds the ttl value, the document is marked for deletion.

This mechanism ensures that documents are automatically cleaned up based on their age and the specified ttl value, helping manage storage and maintain data freshness.

There is no guarantee that items will be deleted exactly at the TTL expiration time. Cosmos DB performs background deletion of expired items. This process is asynchronous, meaning the items are not deleted immediately after the TTL expires. The actual deletion time can vary based on factors like container throughput and the volume of expired items.

Summary

Time to Live (TTL) in Azure Cosmos DB is a valuable feature for automating data lifecycle management. By enabling TTL, you can ensure that outdated or unnecessary documents are automatically deleted after a specified period, helping to optimize storage costs and maintain performance.