Terraform Fundamentals: DocumentDB Elastic

DocumentDB Elastic with Terraform: A Production Deep Dive

The relentless pressure to scale applications while maintaining data consistency and low latency is a constant challenge. Traditional database provisioning often becomes a bottleneck, requiring manual intervention and slowing down development cycles. Modern infrastructure as code (IaC) workflows, particularly those leveraging Terraform, demand a database solution that can be rapidly provisioned, scaled, and managed. DocumentDB Elastic, a feature within Azure Cosmos DB, addresses this need by providing serverless, on-demand scalability for MongoDB-compatible workloads. This post details how to effectively manage DocumentDB Elastic using Terraform, focusing on production-grade considerations for engineers, SREs, and infrastructure architects. It assumes familiarity with Terraform fundamentals and Azure concepts. This service fits into IaC pipelines as a core component of application infrastructure, often deployed via CI/CD pipelines managed by Terraform Cloud or similar platforms.

What is DocumentDB Elastic in Terraform Context?

DocumentDB Elastic isn’t a standalone Terraform resource; it’s a configuration within an existing Azure Cosmos DB account. Terraform manages the Cosmos DB account and its associated Elastic capabilities through the azurerm provider. The primary resource is azurerm_cosmosdb_account, with the capabilities block enabling Elastic.

The azurerm_cosmosdb_account resource manages the Cosmos DB account itself, including location, kind (GlobalDocumentDB is required for Elastic), and consistency level. The capabilities block within this resource is where the Elastic configuration resides.

Terraform-Specific Behavior & Caveats:

• State Management: Changes to the capabilities block, particularly enabling or disabling Elastic, can be disruptive. Careful state management and planning are crucial.
• Dependencies: The Cosmos DB account must exist before attempting to configure Elastic. Terraform’s dependency graph handles this automatically when resources are correctly defined.
• Preview Feature: While generally available, new features within Elastic may still be considered preview and subject to change. Monitor Azure updates closely.
• Resource Naming: Azure resource naming conventions apply. Terraform’s name argument must adhere to these rules.

Use Cases and When to Use

DocumentDB Elastic excels in scenarios demanding unpredictable or highly variable workloads:

1. Event-Driven Architectures: Ingesting and processing streams of events (e.g., IoT data, clickstreams) where throughput fluctuates dramatically. SREs benefit from the auto-scaling capabilities, reducing operational overhead.
2. Microservices with Spiky Traffic: Individual microservices experiencing intermittent bursts of requests. DevOps teams can deploy and scale these services rapidly without over-provisioning.
3. Development/Testing Environments: Dynamically provisioning databases for short-lived development or testing environments. This reduces costs and simplifies environment management.
4. Content Management Systems (CMS): Handling peak traffic during content releases or promotional campaigns. The Elastic model ensures responsiveness without manual intervention.
5. Personalization Engines: Serving personalized content based on user behavior, where query patterns and data volumes vary significantly.

Key Terraform Resources

Here are essential Terraform resources for managing DocumentDB Elastic:

1. azurerm_resource_group: Container for all Azure resources.
   

   resource "azurerm_resource_group" "example" {
     name     = "rg-documentdb-elastic"
     location = "East US"
   }
   

2. azurerm_cosmosdb_account: The core Cosmos DB account resource.
   

   resource "azurerm_cosmosdb_account" "example" {
     name                = "cosmosdb-elastic-account"
     location            = azurerm_resource_group.example.location
     resource_group_name = azurerm_resource_group.example.name
     kind                = "GlobalDocumentDB"
     consistency_level   = "Session"
   
     capabilities {
       enable_elastic = true
     }
   }
   

3. azurerm_cosmosdb_sql_container: A container (collection) within the Cosmos DB account.
   

   resource "azurerm_cosmosdb_sql_container" "example" {
     name                = "my-container"
     account_name        = azurerm_cosmosdb_account.example.name
     resource_group_name = azurerm_resource_group.example.name
     partition_key_path  = "/id"
   }
   

4. azurerm_cosmosdb_sql_role_definition: Predefined roles for access control.
   

   resource "azurerm_cosmosdb_sql_role_definition" "example" {
     name                = "DocumentReader"
     account_name        = azurerm_cosmosdb_account.example.name
     resource_group_name = azurerm_resource_group.example.name
     role_definition     = jsonencode({
       permissions = [
         {
           permission = "ReadDocument"
         }
       ]
     })
   }
   

5. azurerm_cosmosdb_sql_user: User accounts for database access.
   

   resource "azurerm_cosmosdb_sql_user" "example" {
     name                = "dbuser"
     account_name        = azurerm_cosmosdb_account.example.name
     resource_group_name = azurerm_resource_group.example.name
     password            = "StrongPassword123!"
   }
   

6. azurerm_cosmosdb_sql_role_assignment: Assigns roles to users.
   

   resource "azurerm_cosmosdb_sql_role_assignment" "example" {
     name                = "dbuser-reader"
     account_name        = azurerm_cosmosdb_account.example.name
     resource_group_name = azurerm_resource_group.example.name
     role_definition_id  = azurerm_cosmosdb_sql_role_definition.example.id
     assignee_id         = azurerm_cosmosdb_sql_user.example.id
   }
   

7. azurerm_diagnostic_setting: Enables diagnostic logging for monitoring.
   

   resource "azurerm_diagnostic_setting" "example" {
     name                = "cosmosdb-diagnostics"
     target_resource_id  = azurerm_cosmosdb_account.example.id
     storage_account_id  = "/subscriptions/<subscription_id>/resourceGroups/<rg_name>/providers/Microsoft.Storage/storageAccounts/<storage_account_name>"
   }
   

8. azurerm_monitor_diagnostic_setting: (Alternative to azurerm_diagnostic_setting) - More flexible diagnostic settings.

Common Patterns & Modules

• Remote Backend: Always use a remote backend (e.g., Azure Blob Storage, Terraform Cloud) for state management, especially in team environments.
• Dynamic Blocks: Use dynamic blocks within azurerm_cosmosdb_account to configure multiple capabilities if needed.
• for_each: Employ for_each to create multiple containers or users based on a map or list.
• Layered Architecture: Structure your Terraform code into layers: base (resource group, Cosmos DB account), data (containers, users), and application (role assignments).
• Monorepo: A monorepo structure is recommended for larger projects, allowing for better code reuse and dependency management.

Public modules for Cosmos DB exist, but often lack specific Elastic configuration options. Building a custom module is often preferable for production deployments.

Hands-On Tutorial

This example creates a Cosmos DB account with Elastic enabled and a single container.

Provider Setup:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

Resource Configuration:

resource "azurerm_resource_group" "example" {
  name     = "rg-cosmosdb-elastic-demo"
  location = "East US"
}

resource "azurerm_cosmosdb_account" "example" {
  name                = "cosmosdb-elastic-demo"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  kind                = "GlobalDocumentDB"
  consistency_level   = "Session"

  capabilities {
    enable_elastic = true
  }
}

resource "azurerm_cosmosdb_sql_container" "example" {
  name                = "my-demo-container"
  account_name        = azurerm_cosmosdb_account.example.name
  resource_group_name = azurerm_resource_group.example.name
  partition_key_path  = "/id"
}

Apply & Destroy:

terraform init
terraform plan
terraform apply
terraform destroy

terraform plan will show the resources to be created. terraform apply will provision the resources in Azure. terraform destroy will remove them.

Enterprise Considerations

Large organizations typically leverage Terraform Cloud/Enterprise for state management, remote operations, and collaboration. Sentinel or Open Policy Agent (OPA) are used for policy-as-code, enforcing compliance rules (e.g., allowed locations, consistency levels).

• IAM Design: Implement least privilege using Azure RBAC and Terraform’s azurerm_role_assignment resource.
• State Locking: Terraform Cloud/Enterprise automatically handles state locking, preventing concurrent modifications.
• Secure Workspaces: Use separate workspaces for different environments (dev, staging, production).
• Costs: Elastic pricing is based on consumed RU/s. Monitor usage closely to avoid unexpected costs.
• Scaling: Elastic automatically scales, but consider regional availability and potential latency implications.
• Multi-Region: For global applications, configure Cosmos DB with multiple regions for redundancy and low latency.

Security and Compliance

• Least Privilege: Grant only the necessary permissions to users and applications.
• RBAC: Use Azure RBAC to control access to Cosmos DB resources.
• Policy Constraints: Enforce policies using Sentinel or OPA to prevent misconfigurations.
• Drift Detection: Regularly run terraform plan to detect drift between the Terraform configuration and the actual Azure state.
• Tagging Policies: Implement tagging policies to categorize and manage resources effectively.
• Auditability: Enable diagnostic logging and integrate with Azure Monitor for auditing.

Example IAM policy:

resource "azurerm_role_assignment" "example" {
  scope              = azurerm_resource_group.example.id
  role_definition_id = "/subscriptions/<subscription_id>/providers/Microsoft.Authorization/roleDefinitions/4e95e578-5d9c-43a9-84f6-89988c8994d5" # Contributor Role

  principal_id       = "<service_principal_id>"
}

Integration with Other Services

1. Azure Functions: Triggered by Cosmos DB changes.
2. Azure Event Hubs: Ingesting data into Cosmos DB.
3. Azure Logic Apps: Automating workflows based on Cosmos DB data.
4. Azure Synapse Analytics: Analyzing Cosmos DB data.
5. Azure API Management: Exposing Cosmos DB data through APIs.

graph LR
    A[Azure Functions] --> B(Cosmos DB Elastic);
    C[Azure Event Hubs] --> B;
    D[Azure Logic Apps] --> B;
    E[Azure Synapse Analytics] --> B;
    F[Azure API Management] --> B;

Module Design Best Practices

• Abstraction: Encapsulate the Cosmos DB account and Elastic configuration into a reusable module.
• Input/Output Variables: Define clear input variables for customization (e.g., location, consistency level, container names). Output variables should expose relevant information (e.g., Cosmos DB endpoint, keys).
• Locals: Use locals to simplify complex expressions and improve readability.
• Backends: Configure a remote backend for state management.
• Documentation: Provide comprehensive documentation for the module, including usage examples and parameter descriptions.

CI/CD Automation

# .github/workflows/cosmosdb-elastic.yml

name: Deploy Cosmos DB Elastic

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: hashicorp/setup-terraform@v2
      - run: terraform fmt
      - run: terraform validate
      - run: terraform plan -out=tfplan
      - run: terraform apply tfplan

Pitfalls & Troubleshooting

1. State Corruption: Avoid concurrent Terraform operations without proper state locking.
2. Resource Naming Conflicts: Ensure unique resource names across environments.
3. Incorrect Partition Key: Choosing a poor partition key can lead to performance bottlenecks.
4. Insufficient RU/s: Underestimating the required RU/s can cause throttling.
5. IAM Permissions: Incorrect IAM permissions can prevent Terraform from creating or modifying resources.
6. Elastic Not Enabled: Forgetting to set enable_elastic = true in the capabilities block.

Pros and Cons

Pros:

• Scalability: Automatic scaling based on demand.
• Cost-Effectiveness: Pay-per-use pricing.
• Simplified Management: Reduced operational overhead.
• MongoDB Compatibility: Easy migration for existing MongoDB applications.

Cons:

• Complexity: Requires understanding of Cosmos DB and Elastic concepts.
• Vendor Lock-in: Tightly coupled to Azure Cosmos DB.
• Monitoring: Requires careful monitoring of RU/s consumption and costs.
• Potential Latency: Scaling up can introduce some latency.

Conclusion

DocumentDB Elastic, when managed with Terraform, provides a powerful and flexible solution for building scalable and cost-effective applications. By embracing the patterns and best practices outlined in this post, infrastructure engineers can streamline database provisioning, reduce operational overhead, and accelerate development cycles. Start by incorporating this service into a proof-of-concept project, evaluating existing modules, and setting up a CI/CD pipeline to automate deployments. The strategic value lies in its ability to adapt to dynamic workloads and empower teams to focus on innovation rather than infrastructure management.