"forces replacement": the Terraform plan line nobody reads

Line 267 of a 427-line Terraform plan:

  # aws_rds_cluster.reporting must be replaced
  -/+ resource "aws_rds_cluster" "reporting" {
      ~ arn                 = "arn:aws:rds:us-east-1:842910557412:cluster:reporting" -> (known after apply)
      ~ cluster_resource_id = "cluster-D85642F9611A" -> (known after apply)
      ~ engine_version      = "14.9" -> "15.4"
      ~ id                  = "reporting" -> (known after apply)
      ~ storage_encrypted   = false -> true # forces replacement
        # (29 unchanged attributes hidden)
    }

The merge request says "bump reporting Postgres to 15.4." The plan does exactly that. It also destroys the reporting database and creates an empty one in its place.

Underneath the known-after-apply churn, two attributes are changing. One is the version bump, the thing your MR is about. The other is storage_encrypted flipping from false to true, and it isn't yours. Someone on another team that shares this repo merged it earlier in the week.

You're just the one deploying. You review other people's Terraform MRs and have a feel for what each stack normally does; most weeks someone else shepherds the deploy. Today it's you. Your change goes out next, so you're carrying everything merged since the last deploy, including work you never reviewed and had no reason to know about. Nobody was negligent. The queue simply had someone else's change in it.

It's a good change, by the way. You want encrypted storage. But there's no in-place path from unencrypted to encrypted on an RDS cluster. Terraform's only move is destroy and create. That's what -/+ means, and the comment at the end of the line says it in plain English: forces replacement.

And the version bump alone would have failed. Going from 14 to 15 is a major version upgrade, and Aurora refuses those unless the config sets allow_major_version_upgrade = true. This one doesn't. That MR by itself would have died at apply, loudly, with an error naming the exact problem.

A replacement doesn't upgrade anything. It creates a new cluster at 15.4 from scratch, so the flag never comes up. The encryption change removed the failure that would have made someone look.

The plan is accurate and CI is green. Shipping this deploy replaces the database.

I rebuilt this scenario on real AWS to check that it behaves the way I'm about to describe. It does, and the output is further down.

The summary line won't save you

Every Terraform plan ends with a line that looks like the answer. This one shows up 153 lines after the needle:

Plan: 2 to add, 2 to change, 3 to destroy.

Read that as a reviewer. Three destroys. Alarming? You can't tell. A pipeline that tears down and rebuilds a few spot workers on every run produces the same summary as one that's about to drop a database. The summary counts actions. It says nothing about which resources, and nothing about whether this run looks like the ones before it.

And there isn't one baseline to compare against. There are as many as you have stacks.

That pipeline runs three. Here's what all of them printed on the run we've been reading:

network:        Plan: 0 to add, 1 to change, 0 to destroy.
app-services:   Plan: 3 to add, 2 to change, 0 to destroy.
data-platform:  Plan: 2 to add, 2 to change, 3 to destroy.

Five actions on app-services, seven on data-platform. Scanning that list, the difference reads as one stack doing a bit more work than another. Nothing jumps.

The histories are what separate them. app-services has printed between four and six actions on every run this month, always three adds, never a destroy. Five is the middle of its range. data-platform has printed between one and three, and has never destroyed anything. Seven is more than double its worst day, and those three destroys are the first it has ever produced.

The loud stack was normal. The quiet stack was not. You can't see that by comparing them to each other, only by comparing each to itself, which means carrying a separate sense of normal for every stack in the pipeline and noticing when the quiet one gets slightly less quiet.

It's also less precise than it looks. A replacement is one resource, but it lands in that line as one destroy and one add, so the numbers move in ways that don't map to anything you'd call a change. That counting problem is its own rabbit hole, and I have a separate post's worth of scar tissue about it.

The summary is a checksum, not a review.

"Just read the plan"

Sure. Let's be honest about what that costs.

The plan above is 427 lines for one job. The project it belongs to produced 6,726 lines of plan output across its last 30 runs. That's what one month of ordinary work asks one team to read.

Most of it is noise by construction. Of the 427 lines, 240 are state refreshes:

aws_glue_catalog_database.marketing: Refreshing state... [id=marketing]
aws_iam_role_policy.etl_runner_s3_core: Refreshing state... [id=etl-runner-s3-core:inline]
aws_sqs_queue.notifications_shared: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/842910557412/acme-notifications-shared]

Then a destroyed resource unfolds every attribute it ever had, one per line, all of them -> null:

  - resource "aws_db_instance" "airflow_meta" {
      - address                    = "airflow-meta.14d6b36f0507.us-east-1.rds.amazonaws.com" -> null
      - allocated_storage          = 100 -> null
      - arn                        = "arn:aws:rds:us-east-1:842910557412:db:airflow-meta" -> null
      - auto_minor_version_upgrade = true -> null
      ... 38 more lines

Forty-two lines of -> null for one deletion, and they look exactly like the forty-two lines of -> null for a deletion that's routine.

The needle isn't hidden. It sits in the open on line 267, correctly formatted and clearly labeled. It's just wearing the same clothes as everything else, and it's the 267th thing asking for your attention on a Thursday afternoon when the MR title already told you what the change was.

You aren't short on information in that review. You're short on attention.

I ran this against real AWS

The plan above is from the demo project, so the fair question is whether the mechanism holds up outside it. It does. I built an Aurora PostgreSQL cluster, loaded 5,001 rows into it, and shipped the same two changes as three separate runs.

The version bump on its own dies at apply, exactly as described:

InvalidParameterCombination: The AllowMajorVersionUpgrade flag must be
present when upgrading to a new major version.

The encryption flip on its own plans as must be replaced, with # forces replacement sitting on the storage_encrypted line.

Both together, the plan succeeds and never mentions allow_major_version_upgrade anywhere. Grepping the plan for it returns zero matches. The apply reports success, the new cluster has a different resource ID, and daily_revenue does not exist on it.

One thing I did not expect. The endpoint does not change:

before:  reporting.cluster-a1b2c3d4e5f6.us-east-1.rds.amazonaws.com
after:   reporting.cluster-a1b2c3d4e5f6.us-east-1.rds.amazonaws.com

Same hostname, character for character, because the cluster identifier is the same and AWS reuses the per-account DNS suffix. Anything holding that connection string reconnects, authenticates, and finds an empty database. A health check that opens a connection and runs SELECT 1 stays green through all of it. So does DNS. So does Terraform, which exits 0 and reports success. The first thing that notices is whatever reads the table, and how long that takes depends on how often something reads it.

The Terraform, the scripts, and the unedited plan and apply output are in terraform-forces-replacement. It runs in about 40 minutes, costs under a dollar, and the teardown script deletes all of it.

One footnote for anyone reproducing it. Aurora PostgreSQL 14.9 and 15.4 are both deprecated now, so AWS will not let you create a cluster on either. The repo uses 14.23 and 15.18. Crossing the 14 to 15 boundary is what matters, and that is unchanged.

What actually catches these

Reading harder doesn't fix this. Asking a different question does.

"Is this plan safe?" requires holding the blast radius of every resource in your head. "Is this plan normal?" mostly requires memory. You've watched this stack run all month. You know roughly what it does on an average Tuesday. The runs worth slowing down for are the ones that break the pattern.

Three things you can do this week without new tooling:

1. Grep for the phrase, not the symbol. Terraform announces every replacement in English:

grep -nE "forces replacement|must be replaced" plan.txt

One line in CI, and it prints the line number of every recreation in the plan. The -/+ symbol is easy to skim past. The sentence isn't.

2. Treat replacement as its own category. "3 to destroy" is not one number. A destroyed spot worker is background noise. A replaced database, volume, or anything else that holds data means the data doesn't come along. Most review checklists don't distinguish the two, which is exactly how this kind of change gets shipped.

3. Compare against the last green run, not against zero. Before you ship, ask what the same job produced last week. If this run touches four resources that job has never touched, that's the signal, whether or not any single line looks scary.

That third one is the hard one. It doesn't fit in a grep, because it needs history.

The part where I mention what I built

I kept hitting this, so I built a thing for it. DeployCompare reads Terraform plan output from your existing GitLab or GitHub CI logs and compares each stack against the last successful runs of that same stack. Every stack gets its own history instead of a shared threshold, and they're ordered by how far each has drifted from it, so the quiet stack having a loud day sorts above the loud stack having a normal one. No runner changes, no workflow rewrite, read-only access. I wrote up how it discovers pipelines and assembles a baseline in the DeployCompare case study.

The plan in this post is from the live demo, a synthetic project seeded to read like a real team's month of Terraform work. No production database was harmed in the making of this post. A throwaway one was, on purpose, in the repo above. The demo needs no signup, and you can pull up the raw 427-line log and find line 267 yourself: deploycompare.com/demo.

It's free while in beta, and I'm mostly looking for people who'll tell me where it falls over. If you review Terraform plans, or you're the one who walks them to prod, I'd like to hear what your version of line 267 was. Email me.

And if you take nothing else from this post, take the grep. It's one line, and it makes line 267 the first thing you see instead of the 267th.