Solved: Need to vend resource to 100+ Azure subscriptions via pipeline, but Terraform kicking off about providers

🚀 Executive Summary

TL;DR: Terraform struggles with provider configuration errors when deploying resources to numerous Azure subscriptions via for\_each loops because providers must be configured before resource evaluation. The article outlines three solutions: a quick shell script shim, the idiomatic Terraform provider aliasing, and a robust fan-out pipeline architecture for enterprise-scale deployments.

🎯 Key Takeaways

• Terraform’s dependency graph requires provider configuration *before* resource evaluation, causing conflicts when for\_each attempts to dynamically assign subscription IDs to a single provider.
• Provider aliasing enables defining distinct azurerm provider instances for each target Azure subscription, allowing resources to explicitly reference the correct provider via the provider meta-argument.
• For large-scale, complex environments, a fan-out/fan-in pipeline architecture, where an orchestrator triggers separate child pipelines for each subscription, provides superior isolation, parallelization, and scalability.
• The shell script shim offers a quick, albeit less elegant, emergency solution by running terraform apply sequentially for each subscription, bypassing the provider conflict but introducing state management challenges.

Struggling with Terraform provider errors when deploying to multiple Azure subscriptions? Learn why this happens and explore three solutions: a quick script, the native provider aliasing fix, and a robust architectural pattern.

Wrangling Terraform Providers Across 100+ Azure Subscriptions? You’re Not Alone.

I still remember the night clearly. It was 2 AM, coffee was running low, and we were responding to a critical security incident. The mandate from on high was simple: deploy a new, hardened Log Analytics workspace to every single one of our 157 Azure subscriptions. Immediately. “It’s just one resource, Vance. Should be easy with that fancy Terraform you love so much,” my director had said. Famous last words. I whipped up a quick for\_each loop over our subscription list, ran the pipeline, and watched it explode with a sea of red text about providers. Terraform was angry, my director was watching, and I was about to learn a very important lesson about how Terraform thinks about providers at scale.

The Root of the Problem: Why Terraform Gets Confused

Before we dive into the fixes, let’s get into the “why”. This isn’t a bug; it’s a fundamental aspect of how Terraform builds its dependency graph. When you run terraform plan, Terraform needs to configure all its providers *before* it evaluates the resources themselves. Think of it like a chef prepping all their ingredients before they start cooking.

When you use a for\_each loop on a resource, you’re telling Terraform, “I’ll tell you the specific subscription ID for each instance of this resource *during the loop*.” But Terraform has a problem: it already needed to configure the azurerm provider with a single, specific subscription ID *before* the loop even started. It can’t be in 100 places at once with a single provider configuration. It’s a classic chicken-and-egg scenario, and the result is that infamous error message.

Solution 1: The Quick Fix (The “Shell Script Shim”)

This is the “It’s 3 AM and this just needs to work” solution. It’s not elegant, but it’s effective. The idea is to move the loop *outside* of Terraform. You wrap your Terraform execution in a simple shell script (like Bash or PowerShell) that iterates through your list of subscription IDs.

For each subscription, the script calls terraform apply, passing the subscription ID as an input variable. This forces Terraform to run once per subscription, creating a separate, clean execution context each time. No more confusion.

Example (Bash):

#!/bin/bash

# Assume subscriptions.txt is a file with one subscription ID per line
SUBSCRIPTIONS=$(cat subscriptions.txt)

for SUB_ID in $SUBSCRIPTIONS
do
  echo "--- Deploying to Subscription: $SUB_ID ---"

  # Set the context for this run
  az account set --subscription $SUB_ID

  # Run terraform, passing the sub id as a variable
  terraform apply -auto-approve -var="target_subscription_id=$SUB_ID"

  echo "--- Deployment for $SUB_ID complete. ---"
done

Warning: This approach is simple but has drawbacks. It’s slow as it runs sequentially, and managing state can become tricky. You are essentially running 100+ separate Terraform initializations and applications, which isn’t what your pipeline was originally designed for. Use this for emergencies, not for your standard process.

Solution 2: The “Proper” Terraform Way (Provider Aliasing)

This is the solution you should strive for. It keeps the logic entirely within Terraform and is the idiomatic way to handle this scenario. The trick is to define an azurerm provider instance for *every single subscription* you want to target, giving each one a unique alias.

You can even generate these provider blocks dynamically using a for\_each loop!

Step 1: Generate your providers (e.g., in providers.tf)

First, we define a map of our subscriptions. Then, we use a for\_each loop on the provider block itself to create an aliased provider for each entry.

locals {
  # A map of friendly names to subscription IDs
  target_subscriptions = {
    "prod-finance"    = "00000000-0000-0000-0000-000000000001"
    "prod-hr"         = "00000000-0000-0000-0000-000000000002"
    "dev-webservices" = "00000000-0000-0000-0000-000000000003"
    # ... and 97 more ...
  }
}

# The default provider for the pipeline's context (e.g., to read state)
provider "azurerm" {
  features {}
}

# Now, generate a provider for EACH subscription in our map
provider "azurerm" {
  for_each = local.target_subscriptions

  alias           = each.key # e.g., "prod-finance"
  subscription_id = each.value # e.g., "00000000-....-0001"
  features {}
}

Step 2: Use the aliases in your resource

Now, in your resource block, you loop over the same map. For each resource instance, you use the provider meta-argument to explicitly tell Terraform which aliased provider to use for that specific instance.

resource "azurerm_resource_group" "monitoring_rg" {
  for_each = local.target_subscriptions

  # Here's the magic!
  # We tell this specific instance of the resource group
  # to use the provider aliased with `each.key`
  provider = azurerm[each.key]

  name     = "rg-shared-monitoring-eastus"
  location = "East US"
}

This is clean, declarative, and lets Terraform understand the entire plan at once, enabling parallel execution. This is the way.

Solution 3: The Architectural Shift (Fan-Out/Fan-In Pipelines)

For very large, complex environments, sometimes the best solution is to rethink the pipeline itself. Instead of one monolithic pipeline run trying to do 100+ things, you adopt a “fan-out” pattern.

In this model, a master orchestrator pipeline does one thing: it identifies all the target subscriptions. Then, for each subscription, it triggers a separate, standardized child or template pipeline. Each child pipeline is responsible for deploying to just *one* subscription.

How it works:

• Orchestrator Pipeline: Reads a list of target subscriptions (e.g., from a file in git, or via an API call). It then loops and triggers the “Deployment Template Pipeline” for each sub, passing the subscription ID and other parameters.
• Deployment Template Pipeline: A generic, reusable pipeline that takes a subscription ID as a parameter. It initializes Terraform and runs apply for that single scope.

This pattern is natively supported in tools like Azure DevOps (YAML templates) and GitHub Actions (matrix strategies). It provides maximum isolation (a failure in one sub doesn’t stop others), massive parallelization, and much clearer logging and reporting.

Pro Tip: This architectural approach is more complex to set up initially, but it’s the most robust and scalable solution for enterprise-grade IaC. It aligns perfectly with a GitOps model where changes to a subscription’s desired state trigger a targeted deployment pipeline for that subscription only.

Which one should you choose?

As always in DevOps, the answer is “it depends”. Here’s my take:

Solution	Best For…	Complexity
1. Shell Script Shim	Emergency fixes, one-off tasks, or when you are severely time-constrained.	Low
2. Provider Aliasing	The standard, recommended approach for most multi-subscription deployments within a single state file.	Medium
3. Fan-Out Pipeline	Large-scale enterprise environments, promoting team autonomy, and building a truly scalable, resilient IaC platform.	High

That 2 AM incident taught me that understanding the “why” behind your tools is just as important as knowing the commands. By understanding how Terraform’s dependency graph and provider model works, you can move from frustrating hacks to elegant, scalable solutions.

---

👉 Read the original article on TechResolve.blog

---

☕ Support my work

If this article helped you, you can buy me a coffee:

👉 https://buymeacoffee.com/darianvance