DEV Community

Honey Kiran
Honey Kiran

Posted on Originally published at honeykiran.substack.com

3AM Files: Stop. Do Not Hit Yes. Terraform Deletes Every Tenant.

#ai #devops #infrastructure #terraform

One model version string, bumped in one shared variable. The vendor shipped a new model; you point your AI platform at it and go back to bed.

Then terraform plan calmly says it will destroy and recreate every tenant's Bedrock application inference profile. Every tenant. One string. Type a tired 3 a.m. yes and every live profile is gone in one apply.

That scene is the pattern, not a confession.

Animated Terraform plan. A shared model version changes from v1 to v2, and all 12 tenant profiles turn red as the plan counts up to destroy 12, create 12, every tenant, ending on the words Stop. Do not hit yes.

The multiplication

An r/Terraform thread described a module that loops over tenants with for_each, building each key from tenant, model and tag set — something like acme-sonnet4-prod-tags. In the thread's own arithmetic, resource count = model × tenant × tag set, and a model upgrade plans a destroy/create per tenant.

  • Tenant name: almost never changes.
  • Model version: changes with each release and renames every tenant on that model.
  • Tag set: changes with any tag cleanup and renames every tenant carrying it.

Every piece multiplies the AI fleet, and every piece is a trigger.

Diagram of a Terraform for_each key built from tenant, model and tags. A model bump cracks the key and forces destroy/create. After the fix, only the tenant is the key, model and tags sit in the values, and a new model version leaves the key unchanged.

Keys are identities, not labels

Terraform identifies each instance by its key: aws_bedrock_inference_profile.tenant["acme"]. The key is who the object is.

Terraform is a literal hotel clerk: a room change means one guest gone and a stranger arriving. Bump the model in a key and, by default, the old profile is destroyed and a new one is created.

A version string is a label. The moment you glue it into a key, Terraform treats it as a name — and it will bury everyone who held the old name.

The honest half

Some replacements are real. The Bedrock API offers only Create, Get, List, Delete, Tag, Untag and ListTags for these profiles. There's no edit button in the API, so a new model means a new profile. That's physics. The problem is surprise replacement: fleet-wide, unreviewed, set off by one string.

Fix 1: separate identity from configuration

Ask every key: what is this thing, permanently? Model and tags are configuration, so they go in the map's value.

# BEFORE — the key is built from tenant + model + tags.
# Any version bump or tag cleanup renames every key.
resource "aws_bedrock_inference_profile" "tenant" {
  for_each = {
    for t in var.tenants :
    "${t.name}-${t.model}-${t.tag_set}" => t
  }
  name = each.key
  model_source {
    copy_from = each.value.model_arn
  }
  tags = each.value.tags
}

# AFTER — the key is the tenant. Model and tags are values.
resource "aws_bedrock_inference_profile" "tenant" {
  for_each = {
    for t in var.tenants : t.name => t
  }
  name = each.key
  model_source {
    copy_from = each.value.model_arn
  }
  tags = each.value.tags
}
Enter fullscreen mode Exit fullscreen mode

Now the upgrade replaces only the tenant that changed models. The rest show no changes.

Before-and-after Terraform plans. With the key built from tenant, model and tags, one version bump marks all 12 tenant profiles for destruction. With the key set to the tenant alone, the same bump replaces only acme and the other 11 show no changes.

Fix 2: moved blocks walk the fleet home

Before the next model lands, rename the fleet in state instead of rebuilding it. A moved block renames before planning: same object, new address, no destroy.

moved {
  from = aws_bedrock_inference_profile.tenant["acme-sonnet4-prod-tags"]
  to   = aws_bedrock_inference_profile.tenant["acme"]
}

moved {
  from = aws_bedrock_inference_profile.tenant["globex-sonnet4-prod-tags"]
  to   = aws_bedrock_inference_profile.tenant["globex"]
}
Enter fullscreen mode Exit fullscreen mode

Test in staging: expect moves and zero destroys. Keep the blocks; removing one later is a breaking change.

Animation of Terraform moved blocks being written one per tenant. Each old composite key slides into its new tenant-only key while a counter shows zero destroys the whole time, ending with all 12 moved and a plan with nothing to destroy.

Fix 3: rollout discipline

  • -target** is a tourniquet:** it's for exceptional cases only. Routine use breeds drift.
  • -replace** for one rebuild:** name the exact address of the tenant moving to the new model.
  • create_before_destroy: the new object comes first. It's opt-in because unique names can block overlap.
  • Escape hatches: import adopts existing profiles, and removed with destroy = false forgets one without deleting it.

15-second animation. Changing one model version string makes Terraform plan to destroy and recreate all 12 tenants. With the key set to the tenant and the version moved to a value, the same upgrade replaces only one tenant.

The key audit

  1. Forever test: the key says what the thing is.
  2. No versions in keys: no model versions, dates or tag sets.
  3. Known at plan time: never sensitive, never uuid() or timestamp().
  4. A **moved**** block per rename:** keep it until every workspace applies it.
  5. Read the destroys on every model release: an unexplained one is a key problem.
  6. Split state by churn: small states, small blast radii.
  7. Upgrades are rollouts: go tenant by tenant.

Checklist card titled The Key Audit: 7 checks before any for_each ships. 1, forever test: the key says what the thing is. 2, no versions in keys. 3, known at plan time: never sensitive, never uuid() or timestamp(). 4, a moved block per rename, kept until every workspace applies it. 5, read the destroys. 6, split state by churn. 7, upgrades are rollouts, tenant by tenant.

Terraform believes the name tag is the person. Name each tenant's AI profile after something permanent and put versions in the values. The next model release becomes a boring Tuesday change.

What's baked into your for_each keys right now? Tell me in the comments.

New here? 3AM Files is one short, useful read a day on AI doing real DevOps work. Follow along so the next one finds you.

Examples are illustrative — validate in your own environment before running in production.

Disclaimer

  • This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
  • I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
  • Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
  • Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
  • Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

I write about SRE, Kubernetes, Terraform, and platform engineering — the defaults that bite, the incidents that teach, and the guardrails that hold.

Top comments (2)

Collapse
 
price_76 profile image
Sam Price •

The "do not hit yes" part is the real story here. I have a rule now: if Terraform wants to destroy anything with a name I recognize, I close the terminal and come back in the morning. Sleep is a legitimate safety mechanism.

Collapse
 
honeykiran profile image
Honey Kiran •

Sam, that's a real safety rule. A destroy plan you read fresh in the morning beats a tired "yes" every time.
One guardrail that pairs well with it: save the plan with terraform plan -out, then read it with terraform show and count the destroys. If the count is above zero, the pipeline stops and a person has to read it before anything applies.
Your rule catches it when you're tired. The gate catches it when you're wide awake but sure you're right.
Do you hold plans an agent proposed to the same rule, or only the ones you wrote yourself?