Cold Storage, Hot Savings 🧊 Automate Azure Blob Tiering with Terraform and Cut Storage Costs by 70%

#azure #terraform #devops #cloud

Your Hot tier is full of data nobody has touched in months. Lifecycle policies automatically move cold blobs to cheaper tiers. Here's how to deploy them with Terraform and stop paying premium rates for forgotten files.

A company stores 10 TB of application logs, backups, and reports in Azure Blob Storage. All of it sits in the Hot tier at $0.018/GB. Only 20% gets accessed regularly. The other 80%? Paying premium prices for the privilege of being ignored. Monthly waste: $115. Annual waste: $1,382. And that's just one storage account. 🧊

Here's the Azure Blob Storage pricing reality:

Storage per GB/month (LRS, East US):
  Hot tier:     $0.018/GB
  Cool tier:    $0.010/GB    (44% cheaper)
  Cold tier:    $0.0036/GB   (80% cheaper)
  Archive tier: $0.002/GB    (89% cheaper)

The math for 10 TB with typical access patterns (20% active, 30% occasional, 50% rarely/never):

All Hot:     10,240 GB x $0.018      = $184.32/month

Optimized:
  Hot (2 TB):      2,048 GB x $0.018  = $36.86
  Cool (3 TB):     3,072 GB x $0.010  = $30.72
  Cold (3 TB):     3,072 GB x $0.0036 = $11.06
  Archive (2 TB):  2,048 GB x $0.002  = $4.10
  Total:                              = $82.74/month

Monthly savings: $101.58
Annual savings:  $1,218.96 per storage account 🤯

And most orgs have dozens of storage accounts. The best part? Lifecycle management policies are free to configure. You only pay the standard Set Blob Tier API cost when blobs actually move. Let's automate this with Terraform. ⚡

🎯 How Azure Blob Lifecycle Policies Work

Azure lifecycle management runs once daily in the background. You define rules based on blob age (days since last modification, last access, or creation), and Azure automatically moves blobs between tiers or deletes them.

Key concepts to know:

Hot to Cool/Cold/Archive: Instant transitions, no delay
Archive to Hot/Cool: Requires rehydration (up to 15 hours standard, ~1 hour high priority)
Early deletion penalties: Cool = 30 days minimum, Cold = 90 days, Archive = 180 days
Lifecycle policies are free: Only the Set Blob Tier API calls are billed
Runs once daily: Not real-time, background scheduling
Block blobs only for tiering: Append and page blobs only support delete actions

⚡ Step 1: Storage Account with Access Tracking

Before setting up lifecycle rules, enable last_access_time_enabled on your storage account. This tracks when each blob was last read, giving you access-based tiering instead of just modification-based:

resource "azurerm_storage_account" "app_data" {
  name                     = "stappdata${var.environment}"
  resource_group_name      = azurerm_resource_group.main.name
  location                 = azurerm_resource_group.main.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  account_kind             = "StorageV2"  # Required for tiering

  # Enable last access time tracking for smarter tiering
  blob_properties {
    last_access_time_enabled = true  # THIS IS KEY 🔑
  }

  tags = {
    Environment = var.environment
    CostCenter  = var.cost_center
    ManagedBy   = "terraform"
  }
}

Without last_access_time_enabled, you can only tier based on when a blob was last modified or created. With it enabled, you can tier based on actual usage: "move to Cool if nobody has read this blob in 30 days." Much smarter. 🎯

🏗️ Step 2: Lifecycle Policy with Terraform

Here's a production-ready lifecycle policy that handles the common data patterns:

resource "azurerm_storage_management_policy" "tiering" {
  storage_account_id = azurerm_storage_account.app_data.id

  # ──────────────────────────────────────────────
  # Rule 1: Application logs (high volume, rarely re-read)
  # ──────────────────────────────────────────────
  rule {
    name    = "logs-tiering"
    enabled = true

    filters {
      prefix_match = ["logs/"]
      blob_types   = ["blockBlob"]
    }

    actions {
      base_blob {
        tier_to_cool_after_days_since_modification_greater_than    = 7
        tier_to_cold_after_days_since_modification_greater_than    = 30
        tier_to_archive_after_days_since_modification_greater_than = 90
        delete_after_days_since_modification_greater_than          = 365
      }
      snapshot {
        delete_after_days_since_creation_greater_than = 30
      }
    }
  }

  # ──────────────────────────────────────────────
  # Rule 2: Backups (keep longer, archive aggressively)
  # ──────────────────────────────────────────────
  rule {
    name    = "backups-tiering"
    enabled = true

    filters {
      prefix_match = ["backups/"]
      blob_types   = ["blockBlob"]
    }

    actions {
      base_blob {
        tier_to_cool_after_days_since_modification_greater_than    = 14
        tier_to_archive_after_days_since_modification_greater_than = 60
        delete_after_days_since_modification_greater_than          = 730  # 2 years
      }
    }
  }

  # ──────────────────────────────────────────────
  # Rule 3: Reports and uploads (access-based tiering)
  # ──────────────────────────────────────────────
  rule {
    name    = "reports-smart-tiering"
    enabled = true

    filters {
      prefix_match = ["reports/", "uploads/", "exports/"]
      blob_types   = ["blockBlob"]
    }

    actions {
      base_blob {
        # Uses LAST ACCESS TIME instead of modification time
        tier_to_cool_after_days_since_last_access_time_greater_than    = 30
        tier_to_archive_after_days_since_last_access_time_greater_than = 180
        delete_after_days_since_last_access_time_greater_than          = 730

        # Auto-promote back to Hot if someone reads it
        auto_tier_to_hot_from_cool_enabled = true
      }
    }
  }

  # ──────────────────────────────────────────────
  # Rule 4: Cleanup old blob versions and snapshots
  # ──────────────────────────────────────────────
  rule {
    name    = "version-cleanup"
    enabled = true

    filters {
      blob_types = ["blockBlob"]
    }

    actions {
      version {
        change_tier_to_cool_after_days_since_creation  = 30
        delete_after_days_since_creation               = 90
      }
      snapshot {
        delete_after_days_since_creation_greater_than = 90
      }
    }
  }
}

This gives you four targeted rules:

Logs: Aggressive tiering (Hot to Cool in 7 days, Archive in 90, delete at 1 year)
Backups: Archive-focused (Cool at 14 days, Archive at 60, keep 2 years)
Reports: Access-based with auto-promotion (only tiers down if nobody reads it)
Versions/Snapshots: Automatic cleanup to prevent silent accumulation

🔧 Step 3: Reusable Module for Multiple Storage Accounts

If you have many storage accounts, build a module that applies consistent lifecycle policies:

# modules/storage-lifecycle/variables.tf

variable "storage_account_id" {
  type        = string
  description = "ID of the storage account to apply lifecycle policies to"
}

variable "log_retention_days" {
  type    = number
  default = 365
}

variable "backup_retention_days" {
  type    = number
  default = 730
}

variable "cool_after_days" {
  type    = number
  default = 30
}

variable "archive_after_days" {
  type    = number
  default = 90
}

variable "enable_access_based_tiering" {
  type    = bool
  default = true
  description = "Use last access time for tiering (requires last_access_time_enabled on storage account)"
}

variable "custom_rules" {
  type = list(object({
    name           = string
    prefix_match   = list(string)
    cool_after     = optional(number)
    cold_after     = optional(number)
    archive_after  = optional(number)
    delete_after   = optional(number)
  }))
  default     = []
  description = "Additional custom lifecycle rules"
}

# modules/storage-lifecycle/main.tf

resource "azurerm_storage_management_policy" "this" {
  storage_account_id = var.storage_account_id

  # Default log tiering
  rule {
    name    = "log-lifecycle"
    enabled = true
    filters {
      prefix_match = ["logs/", "diagnostics/", "audit/"]
      blob_types   = ["blockBlob"]
    }
    actions {
      base_blob {
        tier_to_cool_after_days_since_modification_greater_than    = 7
        tier_to_cold_after_days_since_modification_greater_than    = 30
        tier_to_archive_after_days_since_modification_greater_than = var.archive_after_days
        delete_after_days_since_modification_greater_than          = var.log_retention_days
      }
    }
  }

  # Default backup tiering
  rule {
    name    = "backup-lifecycle"
    enabled = true
    filters {
      prefix_match = ["backups/", "snapshots/"]
      blob_types   = ["blockBlob"]
    }
    actions {
      base_blob {
        tier_to_cool_after_days_since_modification_greater_than    = 14
        tier_to_archive_after_days_since_modification_greater_than = 60
        delete_after_days_since_modification_greater_than          = var.backup_retention_days
      }
    }
  }

  # Custom rules via variable
  dynamic "rule" {
    for_each = var.custom_rules
    content {
      name    = rule.value.name
      enabled = true
      filters {
        prefix_match = rule.value.prefix_match
        blob_types   = ["blockBlob"]
      }
      actions {
        base_blob {
          tier_to_cool_after_days_since_modification_greater_than    = rule.value.cool_after
          tier_to_cold_after_days_since_modification_greater_than    = rule.value.cold_after
          tier_to_archive_after_days_since_modification_greater_than = rule.value.archive_after
          delete_after_days_since_modification_greater_than          = rule.value.delete_after
        }
      }
    }
  }

  # Always clean up old versions
  rule {
    name    = "version-cleanup"
    enabled = true
    filters {
      blob_types = ["blockBlob"]
    }
    actions {
      version {
        change_tier_to_cool_after_days_since_creation = 30
        delete_after_days_since_creation              = 90
      }
      snapshot {
        delete_after_days_since_creation_greater_than = 90
      }
    }
  }
}

Usage across multiple storage accounts:

module "app_storage_lifecycle" {
  source             = "./modules/storage-lifecycle"
  storage_account_id = azurerm_storage_account.app_data.id

  custom_rules = [
    {
      name          = "user-uploads"
      prefix_match  = ["uploads/", "media/"]
      cool_after    = 60
      archive_after = 180
      delete_after  = 1095  # 3 years
    }
  ]
}

module "analytics_storage_lifecycle" {
  source               = "./modules/storage-lifecycle"
  storage_account_id   = azurerm_storage_account.analytics.id
  log_retention_days   = 180  # Shorter retention for analytics
  archive_after_days   = 60   # Archive sooner
}

Every storage account gets a consistent baseline with room for per-account customization. New storage accounts automatically get lifecycle policies. No more forgotten Hot-tier data sitting untouched for months. ⚡

⚡ Quick Audit: Find Your Hot Tier Waste

# Check total capacity by tier for a storage account
az storage account show \
  --name YOUR_STORAGE_ACCOUNT \
  --resource-group YOUR_RG \
  --query "{Name:name, AccessTier:accessTier}" \
  --output table

# List all storage accounts and their default access tier
az storage account list \
  --query "[].{Name:name, RG:resourceGroup, Tier:accessTier, Kind:kind}" \
  --output table

# Check if lifecycle policy exists on a storage account
az storage account management-policy show \
  --account-name YOUR_STORAGE_ACCOUNT \
  --resource-group YOUR_RG 2>/dev/null || echo "NO LIFECYCLE POLICY! 🚨"

# Check if last access time tracking is enabled
az storage account blob-service-properties show \
  --account-name YOUR_STORAGE_ACCOUNT \
  --resource-group YOUR_RG \
  --query "lastAccessTimeTrackingPolicy.enable"

If any storage account returns "NO LIFECYCLE POLICY" - that's money sitting in the Hot tier burning. 🔥

💡 Architect Pro Tips

Enable last_access_time_enabled from day one. It only tracks access from the point you enable it, not retroactively. The sooner you turn it on, the better your data for access-based tiering decisions.
Use auto_tier_to_hot_from_cool_enabled carefully. It's great for reports and user files that might get re-accessed. But if a blob gets accessed, it auto-promotes to Hot, and if not re-accessed for 30 days, it tiers back down to Cool. The round-trip incurs early deletion fees if it happens within 30 days. Analyze your access patterns first.
Respect early deletion penalties. Cool = 30 days, Cold = 90 days, Archive = 180 days. If you delete or move a blob before the minimum, you pay the remaining days at that tier's rate. Set your policy thresholds higher than these minimums.
Archive tier is offline. Blobs in Archive cannot be read directly. Rehydration takes up to 15 hours (standard) or about 1 hour (high priority, at higher cost). Don't archive anything that might need immediate access.
Lifecycle policies apply per-account. Each azurerm_storage_management_policy replaces the entire policy for that storage account. If you manage lifecycle rules in Terraform, manage ALL rules in Terraform to avoid drift.
Prefix filters are your precision tool. Use container/folder prefixes to apply different rules to different data types. logs/ gets aggressive tiering while config/ stays Hot forever by simply not matching any rule.
Blob versions and snapshots add up silently. Without cleanup rules, every blob edit creates a version that stays in the Hot tier indefinitely. The version cleanup rule in the module above prevents this accumulation.

📊 TL;DR

Data Type	Hot to Cool	Cool to Archive	Delete	Annual Savings (10 TB)
Logs	7 days	90 days	365 days	~$800
Backups	14 days	60 days	730 days	~$900
Reports	30 days (access-based)	180 days	730 days	~$600
Versions/Snapshots	30 days	-	90 days	~$200

Bottom line: Every storage account without a lifecycle policy is paying Hot-tier prices for data that should be in Cool, Cold, or Archive. The lifecycle policy itself is free. The azurerm_storage_management_policy resource takes 10 minutes to configure and runs forever on autopilot. There's no reason not to have one on every storage account. 🧊

Check your storage accounts right now. Run the audit command above. If you see "NO LIFECYCLE POLICY" on any of them, you just found your next quick win. 😏

This is Part 6 of the "Save on Azure with Terraform" series. Next up: Right-Size or Pay the Price 📏. How to find over-provisioned VMs and build a Terraform module that maps workloads to optimal SKUs. 💬