DEV Community

Cover image for Why terraform plan fails three modules from the cause: the map(any) trap
Muhammad Hassaan Javed for Infraforge

Posted on Edited on Originally published at infraforge.agency

Why terraform plan fails three modules from the cause: the map(any) trap

The on-call engineer pinged me at 4:42pm on a Friday with the release window open until 5:30. terraform apply against the staging workspace had failed with Error: Invalid value for module argument, anchored at a module block argument three layers down, in a subtree nobody on the team had touched in six months. They told me plan ran clean, and that they had re-run it twice. Both of those runs used their default dev var-file, which never carried the value that broke staging. The failure was not deferred to apply either: apply runs a plan first, and that is the phase that errored. plan and apply do not diverge on a static merge() of two known maps. That only happens when a value is unknown at plan time, on a data source or a resource attribute that resolves during apply, and nothing in this call graph was unknown.

Problem signals:

  • terraform plan passes against the environment your contributors run it against, and fails against the one nobody plans until deploy day
  • The error is Invalid value for module argument on a module block, or Incorrect attribute value type / Inappropriate value for attribute on a resource, and the file it names is not the file that built the value
  • The value the diagnostic rejects was assembled by a merge() or lookup() in a module that has not been edited in months, and the diagnostic never names that file
  • Your root module input list has crossed 20 variables and several are typed any or map(any)
  • There is no CI job that runs terraform plan against every environment on every PR

Three hypotheses, three dead ends, thirty minutes left in the release window

What we ruled out in the first 18 minutes

The first thing the on-call lead suggested was state drift. Someone, somewhere, had terraform import-ed a resource by hand. We checked the audit log. No import events in the past 30 days. We checked the lock table in DynamoDB. The lock had been released cleanly by the previous successful apply at 2:11pm.

The second hypothesis was provider version drift. The team had recently bumped hashicorp/aws from 5.62 to 5.71 in versions.tf. A breaking change in a resource schema can absolutely produce a type error like this one. Pinning the run to 5.71 would have proved nothing, because 5.71 is the version versions.tf already declares and re-initialising on it just reproduces the suspect. So we pinned back to the last-known-good version = "5.62.0", deleted .terraform/ and .terraform.lock.hcl, re-ran init -upgrade, then plan -var-file=envs/staging.tfvars. Same error, same module block, same line, and it came back in about a second, before a single resource had been refreshed. Persisting on 5.62 exonerated the bump, and that timing was the genuinely discriminating signal: the error was landing during variable and argument evaluation, at the very start of the plan walk.

The third hypothesis was a stale workspace. terraform workspaces sometimes diverge from the configuration if workspace select was bypassed by an engineer who exported TF_WORKSPACE and forgot. We ran terraform workspace show and verified it matched the intended target. The backend key and the resource addresses in the failing run were the right ones.

Three explanations, three dead ends, eighteen minutes burned. The release window was now thirty minutes wide and shrinking. The on-call lead asked whether we should just roll back the deploy and figure it out Monday. I asked one more question first.

The 15th map(any) input that had been silently incubating for three weeks

Where the collision actually lived

I asked the on-call lead to walk me through what had merged into the workspace in the past three weeks. There were six commits. Five were obvious changes (image tags, a new IAM policy, a security group port). The sixth was a feature flag, added as a 15th map(any) input on the root module by an engineer who had joined six weeks earlier.

That was the lead.

The root module had 28 input variables. 14 of them were any-typed or map(any) to absorb per-environment overrides accumulated over six years of feature additions. The new feature flag added a 15th map(any) input named feature_overrides. Its values flowed through a merge() chain down to the database child module, which did its own merge(local.legacy_db_flags, var.feature_overrides) inside modules/services/database/locals.tf.

The two maps had a key collision. Both contained a key named read_replica_routing. Every value in the staging map was a string, so map(any) unified to map(string) the moment the tfvars value was assigned, and the input converted without complaint. The legacy local's value for that same key was a map(object({ host = string, weight = number })). merge() resolves collisions by taking the last argument's value, and var.feature_overrides is the last argument, so wherever it carried that key it won. merge() of maps whose element types differ returns an object type, so the merge itself raised nothing either way. The key was set in envs/staging.tfvars and left out of every other environment's tfvars, which is why staging was the one place the string ever reached the module.

Nothing about the string was hidden by any. map(any) cannot even carry a mixed map: Terraform unifies every element to a single concrete type, and a map that mixes a string with a shape it cannot convert is rejected outright with Invalid value for input variable ... all map elements must have the same type during plan. Had the engineer written the object shape into feature_overrides alongside the strings, the input would have failed at the variable, which is exactly the error you want. What we got instead was a well-formed map(string) travelling three layers before anything with a real type constraint looked at it.

The diagnostic read Invalid value for module argument, and it was anchored at the argument in the module block inside modules/services/database/ that passes local.merged_flags one layer further down. Not inside the child that consumed it, and not at modules/services/database/locals.tf where the value was actually built: a locals block carries no type constraint, so it can never be the thing that rejects a value. (Had the value landed on a resource attribute rather than a module argument, the string would have been Incorrect attribute value type or Inappropriate value for attribute instead.) You read that diagnostic forwards to find the receiving type constraint, then walk backwards from the argument to the merge() that produced the value. That backwards walk was the whole investigation.

 raw `map(any)` endraw  does not move the type error past plan. It moves it away from the file that caused it.

map(any) does not move the type error past plan. It moves it away from the file that caused it.

The collision had been latent for three weeks because nobody had planned staging with the key set. terraform's planner did not collapse anything: any resolved to string at variable evaluation, module-argument conversion ran in the same plan walk, and the error was reachable from a one-second terraform plan -var-file=envs/staging.tfvars on the day the input merged. What was missing for three weeks was the plan, not the type check.

That is the part that hurts. any is a type-constraint placeholder, not a runtime type, and it defers nothing to apply. Terraform resolves it to a concrete type from whatever value you supply, during variable evaluation, at the start of every run. A genuine plan-pass/apply-fail type error needs an unknown value, a resource attribute or an apply-time data source feeding the argument, and a static merge() of two known maps out of tfvars is never that. What every map(any) input on a root module actually buys you is an error anchored three files from its cause, against an expected shape that is written down nowhere.

Three options, one open release window, two minutes to pick

What we did before running apply again

We had three options and one open release window. I walked the on-call lead through them on the bridge call.

Step What it does
1. Delete the legacy key Feels fastest, and on its own it is a no-op on the error: merge() takes the last argument's value, so the string from feature_overrides still wins and the plan fails identically. Clearing the diagnostic this way means also dropping read_replica_routing from envs/staging.tfvars and retiring the object-typed contract the legacy key fed, which three modules-of-modules three layers down still referenced. That is a far larger change than it looks, and the wrong one to attempt inside a release window.
2. Rename the new key Safe-feeling. Left the underlying any-typed contract intact. Two months later a different contributor would add another map(any) input and we would be back on a Friday afternoon with the same shape of failure.
3. Rename plus add validation Slower. Renamed the new key to feature_routing_overrides AND added a validation block on the input that rejected the colliding shape at the variable, where the diagnostic names the file a contributor can act on. Stopped the immediate reoccurrence.

Option three carried the day. The rename took four minutes. The validation block took six. The staging plan then came back with the release's actual changes in it, the image tags, the IAM policy and the security group rule, and apply succeeded at 5:14pm with sixteen minutes to spare on the release window. The release shipped on time.

The audit work behind option one (the one we did NOT take) is what stuck with me. The next morning, we grep-ed the entire terraform/ tree for read_replica_routing to map every consumer. Seven references across four modules. Three in modules/services/database/locals.tf itself. One in modules/monitoring/cloudwatch.tf. One in modules/services/cache/lookups.tf, which read the value to construct its own routing decision and would have broken silently if we had deleted the legacy key the night before. The remaining two were in a state-recovery helper module the team had forgotten existed. We had nearly fired the second shot of our own foot.

We left a tombstone comment on the legacy key and an open PR that would, the following week, replace its map(any) type with a proper object({ ... }) schema. That work landed five days later. The downstream consumers surfaced the change at plan time, and three of them needed minor patches before the type tightening could merge. None of those patches would have caught the original collision. They all caught real existing mismatches that the loose contract had let travel much further from their source than they should have.

Two policy changes and one structural fix

What we changed afterwards

Two policy changes came out of that night, and one structural fix took longer.

The first policy: no new map(any) or any-typed inputs on root modules. The team's terraform/ directory has a pre-commit hook (8 lines of grep) that fails the commit if any new variable block contains type = any or type = map(any). Existing instances are grandfathered, with a TODO list tracked against each module. Three of the original 14 have been converted to typed objects so far. The hook has fired four times in the six weeks since.

The second policy: every PR runs terraform plan against every environment, not just the one the contributor cares about. A matrix job in CI runs terraform workspace select <env> && terraform plan -input=false -var-file=envs/<env>.tfvars across all four environments and fails the PR if any of them errors. Each leg has to select the environment's state, not only its variables: swapping -var-file alone would plan all four var-files against whatever single workspace the runner happens to have selected, so the prod leg would propose rewriting staging with prod values. Setting TF_WORKSPACE=<env> per leg is the equivalent one-liner, and if your environments are separate backends rather than workspaces, the leg needs terraform init -reconfigure -backend-config=envs/<env>.backend instead. The staging leg of that matrix would have failed on the PR that introduced the input, three weeks before the release window. It also catches a different class of failure where one environment's tfvars hits an unwritten code path.

# Before: latent any-typed input
variable "feature_overrides" {
  type        = map(any)
  default     = {}
  description = "Per-environment feature flag overrides"
}

# In modules/services/database/locals.tf
locals {
  merged_flags = merge(
    local.legacy_db_flags,
    var.feature_overrides,
  )
}

# Above still fails at plan when the two maps share a key
# whose value types disagree, but the diagnostic lands on a
# module argument three files away, against a shape nobody
# declared here. `any` hides the contract, not the error.

# After: typed, explicit, errors at the variable
variable "feature_overrides" {
  type = map(object({
    enabled     = bool
    rollout_pct = optional(number, 0)
    routing     = optional(string, "default")
  }))
  default     = {}
  description = "Per-environment feature flag overrides"

  validation {
    condition = alltrue([
      for k, v in var.feature_overrides :
      v.rollout_pct >= 0 && v.rollout_pct <= 100
    ])
    error_message = "rollout_pct must be between 0 and 100."
  }
}
Enter fullscreen mode Exit fullscreen mode

The same variable, before and after. Both forms fail at plan; the lower one fails at the variable itself, naming the shape it expected.

The structural fix took longer. A 28-input root module is not a configuration problem, it is a service-boundary problem. The team running the database stack should own a database/ root module with four inputs, not a 14-input subtree of a shared 28-input root. We split the original root into three roots along ownership boundaries (network, services, observability) using a thin terragrunt overlay for the cross-cutting variables. The split took six weeks of careful state-mv work to land without downtime. We have written more on the structural fix in the Terraform and IaC debt playbook, which covers when a shared root module starts costing more than the consistency it buys.

What we tell every team now: strong types in Terraform are not bureaucracy, they are the documentation. The half-day cost to write object({ name = string, enabled = bool, ... }) instead of map(any) buys you a failure at the variable, on the PR that introduces it, instead of a failure three modules away with a release window open. We have stopped accepting map(any) inputs in any client engagement that involves an IaC audit, and we have not had a single contributor push back once they saw the cost.

If you are looking at a 28-input root with map(any) sprinkled through it

When your own root module is past 20 inputs

If you are reading this and your terraform/ directory has a root module past 20 inputs with several map(any) types in the input list, the failure you are heading toward is not a surprise. It is a scheduled event. The trigger will be a new contributor who does not know the implicit contract, plus one bad-enough Friday. The hardest part of cleaning it up is not the typing work itself; it is the audit of downstream consumers that have been silently depending on the loose contract for years. Two layers of modules-of-modules can hide a reference that breaks the moment you tighten the type, and your CI will not warn you, because it only ever plans the environment the contributor touched. The environment carrying the bad value goes unplanned until someone is standing in front of a release window with it.

We run these recovery and audit engagements every week. The map(any) collision pattern is the third-most-common shape we see in seed-to-Series-B SaaS Terraform repos, right after stale state lock holders and provider-version-drift cascades. It is one variant of the broader terraform apply fear problem we engage on most weeks. On a typical engagement we map every any-typed input in your root modules within the first day, prioritize them by blast radius, and either convert them in-place or split the root if the input count is the real problem. If you are looking at a Terraform root with map(any) sprinkled through it and a release window that does not forgive a 4pm failure, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week.


Originally published at https://infraforge.agency/insights/terraform-apply-fails-map-any-trap/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Top comments (0)