Two engineers ran terraform state list against what they both called the same workspace and got 23 resources and 19. Neither laptop was broken. One was on a checkout predating the backend "s3" block, so its configuration named no backend at all, Terraform used the local one, and the committed terraform.tfstate was the state; the other was on current main and initialized against S3. Check the commit before you check the state, because the same command answers to a different authority depending on it. The terraform state commands below are the ones you reach for when that happens, and the only column that matters is which of them touch live infrastructure, which touch only the state file, and which quietly do neither. This is the reference we keep open during a state split.
Problem signals:
- terraform state list returns a different resource count on two laptops, and the checkouts are not on the same commit
- Error: Error acquiring the state lock with a Lock Info block naming a Path you do not recognise
- apply fails with Error 409: The resource ... already exists, alreadyExists on something you never removed
- git ls-files shows a tracked terraform.tfstate next to your backend block
- plan proposes to create a resource you can see running in the cloud console
What a four-resource gap in terraform state list actually means
23 resources on one laptop, 19 on the other
The first thing to establish in a state split is not what the state contains. It is which state you are reading. Terraform records the backend it resolved for a working directory in .terraform/terraform.tfstate, which is a state-shaped file with a backend key, and it is not the same file as the root terraform.tfstate that holds your resources. Be precise about when that produces a split, because Terraform does not quietly fall back. If the configuration declares backend "s3" and the cached backend is missing or records something else, it refuses to run anything at all, terraform state list included, with Initial configuration of the requested backend or Backend configuration block has changed and a demand that you run terraform init. A directory treats a committed terraform.tfstate as authoritative only when BOTH sides agree there is no backend: the configuration declares none, and .terraform/terraform.tfstate records none either. That is a checkout predating the block in a directory that never initialised against S3. Get one of the two wrong and the refusal is symmetric: a config that declares a backend the cache does not know gives you Initial configuration of the requested backend, and a cache holding a backend the config no longer declares gives you Unsetting the previously set backend "s3". Both demand an init, and both offer to copy state in the direction you may not want. Update that checkout and the same directory stops being quiet: the configuration now declares backend "s3" while the directory still has no backend cache at all, which is the first of those two refusals, and Terraform will not run until you init.
So a four-resource gap has three silent shapes, and the cheapest to rule out is the checkout. A working directory sitting on a commit that predates the backend block resolves the local backend and prints the committed file's resources without complaint, which is the case in the lede; git log -1 and grep -r 'backend "' *.tf settle it before you touch anything else. The other two need both directories properly initialised: a different selected workspace, or a different -backend-config key or bucket. Resist adding credentials to that list, however tempting, because two engineers always do have different identities and it is the entry most likely to look confirmed. On this backend credentials decide whether the read SUCCEEDS, not which object is read: the object is identified by the bucket and key in the backend configuration. Different credentials against the same configured bucket either read the same object or fail loudly on access. Their real damage is in plan, where Terraform reads a different account’s infrastructure, not in state list. The loud case is the mirror of the first. Pull that stale checkout forward past the backend block and the directory stops reading anything at all: the config now names a backend this directory has never initialised, Terraform demands an init, and the danger moves to the -migrate-state prompt below.
So the diagnostic is three commands, not one, and you run them on both machines before anyone argues about who is right. The backend cache tells you where you are pointed, and the selected workspace tells you which object you get once you are pointed there, which the cache cannot: a workspace_key_prefix setup shows the same key on both laptops while state pull reads two different objects. terraform show -json against an explicit file path tells you what a specific state file on disk contains, without going near the network and without asking Terraform to decide which backend it prefers. It does need the provider plugins installed locally, and when they are missing it fails rather than printing an empty list, which only helps if the script checks.
# Which backend did THIS working directory resolve to? An init with no backend
# block writes no backend cache at all, so ANY recorded type, local included,
# means a backend block was declared at some point. With the block still there
# the cache is expected; with it gone, Terraform answers "Unsetting the
# previously set backend" and refuses. No cache and no block is the healthy
# case this article's lede is about.
# declared() only reads *.tf here. It misses *.tf.json and a cloud {} block, so
# if either is in use, read the configuration by eye before trusting a branch.
declared() { grep -qE '^[[:space:]]*backend[[:space:]]+"' *.tf 2>/dev/null; }
cached_type=$( [ -f .terraform/terraform.tfstate ] \
&& jq -r '.backend.type // "none"' .terraform/terraform.tfstate || echo none )
if [ "$cached_type" != none ] && declared; then
jq '.backend.type, .backend.config.bucket, .backend.config.key' .terraform/terraform.tfstate
elif [ "$cached_type" != none ]; then
echo "Cache records a backend ($cached_type) the config NO LONGER declares. Terraform"
echo 'answers Unsetting the previously set backend and refuses until init. Stop.'
exit 1
elif declared; then
echo "A backend is DECLARED and none is resolved. Terraform answers Initial"
echo "configuration of the requested backend and refuses until init. Stop here."
exit 1
else
echo "no cache and no backend block: this directory resolved the LOCAL backend"
fi
# Both refusals are stops, not warnings. Past either one the pull below fails,
# leaves an empty file, and the diff reads as an empty remote, which is the
# reading that sends someone to push local state over a good remote one.
# Which workspace is selected? The backend cache does NOT record this, and with
# workspace_key_prefix the cached config.key is identical across every workspace
# of the same configuration, so two laptops can print the same bucket and key
# and still pull two different objects.
terraform workspace show 2>/dev/null || cat .terraform/environment 2>/dev/null || echo default
# Render both sides through the same function, or the diff is fiction:
# terraform show -json emits full resource ADDRESSES, and recurse() is what
# stops it silently dropping everything that lives inside a module.
# show -json needs the provider plugins installed locally, and when it fails jq
# still exits 0 on empty input, so check the exit status or a failure reads as
# a state with nothing in it.
addrs() {
local j
j=$(terraform show -json "$1") || { echo "show -json failed on $1. Stop." >&2; return 1; }
printf '%s' "$j" \
| jq -r '[.values.root_module | recurse(.child_modules[]?) | .resources[]?.address] | sort[]'
}
# What does the committed file on disk contain, independent of any backend?
addrs terraform.tfstate > /tmp/local.txt || exit 1
# And what does the CONFIGURED backend hold? Not necessarily the remote one:
# state pull reads whatever backend line 1 just printed. If that says "local",
# both sides of this diff come from the same file and it returns a false clean.
terraform state pull > /tmp/remote.tfstate || exit 1
addrs /tmp/remote.tfstate > /tmp/remote.txt || exit 1
diff /tmp/local.txt /tmp/remote.txt
Neither engineer was wrong about what they saw. They were reading two different files, and only the backend cache says which. Read line 1 before you trust the diff: state pull follows the configured backend, so on the machine whose cache says local, or has no cache at all, you are comparing a file with itself. Build both sides with the same function too: reading .values.root_module.resources[] on its own returns root-module resources only, so on any workspace with modules the diff invents a gap that was never there.
In the case that pushed us to write this down, the diff came back with five resources present only in S3 and one google_sql_database_instance present only locally, which is where the four comes from. Four of the five were ordinary additions merged since that checkout diverged. The fifth was a google_compute_firewall created by hand in the console during an outage the previous weekend and imported to the remote state by whoever fixed it. The SQL instance existed in the local file because the engineer working from the stale local backend had applied it. Both of those changes were real, and each was recorded in exactly one of the two states, which is the part a resource count cannot show you.
The committed state file only wins when the working directory never resolved any backend at all. That is the branch to rule out first.
terraform init flags: which one moves state and which one only re-points it
The five init flags that change where writes land
These five flags get confused constantly, and the confusion is expensive because three of them decide whether an existing state gets copied to the new location or abandoned there. We assume Terraform CLI open source, 1.9.x, with an S3 backend and DynamoDB locking. Current versions of the S3 backend can lock with a lockfile in the bucket itself (use_lockfile), and HashiCorp now documents DynamoDB locking as deprecated, so on a newer version the lock you are reading may be an S3 object rather than a table item. Terraform Cloud and Enterprise workspaces have their own migration surface and different prompts, so do not carry these across.
FLAG WHAT IT DOES SAFE WHEN DAMAGE WHEN NOT
------------------------- -------------------------------------------- --------------------------------- --------------------------------------
-migrate-state Offers to copy state from the previously You are moving a workspace from You accept the prompt while the source
configured backend into the newly one backend to another and want is the stale side of a split. The stale
configured one, with a yes/no prompt. the contents carried over. contents become the remote authority.
-reconfigure Re-initializes the backend and discards the You are pointing a directory at a You use it as a shortcut past a
existing backend selection. No state is backend that already holds the migration prompt. The old backend keeps
copied. state you want. the only copy of the resources.
-backend-config=KV|FILE Supplies backend settings (bucket, key, Values live outside the config, A wrong `key` adopts a different
region) at init time. e.g. per-environment .hcl files. workspace's state. From another
configuration, plan proposes to CREATE
all of yours and DESTROY all of theirs.
From another environment of the SAME
configuration the addresses match, and
plan shows updates and replacements of
their real resources instead, which is
much harder to spot.
-backend=false Initializes providers and modules only. You want a provider install or a You expect it to protect you from
Leaves the backend selection untouched. validate pass without touching writes generally. It does not gate
backend credentials. apply in a later, normal init.
-force-copy Suppresses the migration prompts and You are scripting a migration you It performs the copy that the
answers yes to all of them. Turning it have already proven by hand, and -migrate-state row tells you to stop
on also enables -migrate-state. the copy direction is settled. and read, with no prompt to read.
Check the CI init line before you
trust a pipeline: it is often there.
-migrate-state copies. -reconfigure abandons. Picking the wrong one at 2am is how the wrong side of a split becomes the official one.
The specific trap: -migrate-state prompts you, and during an incident the prompt reads like a formality. It is not. It is Terraform asking which of your two states you want to keep, and it will happily copy the four-resource-short file over a correct remote one. Read the source and destination in the prompt text out loud before typing yes. If you are unsure, answer no. That assumes there is a prompt at all: -force-copy answers yes to every one of them, so read the init line you inherited before you rely on reading anything, and take the backup outside Terraform, because terraform state pull cannot give you one here. Before init it fails with the same backend error as every other command, and once you decline the migration it reads the newly configured destination, which is empty, so you walk away holding a file you believe is the good state and is not. Copy the source directly, cp terraform.tfstate backup-$(git rev-parse --short HEAD).tfstate where the source is the local file, or run terraform state pull against the OLD backend configuration before you re-point it.
The other habit worth building is that a tracked state file is a repo problem, not a Terraform problem. git ls-files -- '*.tfstate' '*.tfstate.backup' takes half a second and answers the question that took our client's team an hour of laptop-to-laptop comparison. If it returns anything, you have a second authority in the repo regardless of what your backend block says, and every fresh clone is a coin flip. We wrote up the wider version of this failure in the Terraform state recovery playbook.
terraform state subcommands: what each one touches and the damage when misused
Ranked by what they can destroy, not by how often you type them
Here is the table itself. The column that decides everything is the middle one: whether the command touches only the state file, only live infrastructure, or both. Most state accidents come from an operator who believed a state-only command was going to change the cloud, or believed a cloud-touching command was going to be recorded.
COMMAND TOUCHES SAFE WHEN SPECIFIC DAMAGE WHEN NOT
------------------------- ------------------- ----------------------------------------- ----------------------------------------------
terraform state list nothing (read) Always, and the cheapest thing here. Run None. Note it reads the CONFIGURED backend,
it after the pull, not instead of it. so it inherits whatever split you are debugging.
terraform state show ADDR nothing (read) Always. Shows recorded attributes for one None, but the values are what state believes,
address. not what the cloud currently holds.
terraform state pull nothing (read) Always, before any of the four commands None. Take it and keep it. What any of these
below that write state. commands writes automatically depends on the
backend and the version, `state push` has no
`-backup` option at all, and none of that is worth
working out during an incident. The pull is the
copy you control, and the rule is the same for
`push`, `rm`, `mv` and `replace-provider`.
terraform state push FILE state only, and You are restoring a known-good snapshot CheckValidImport refuses three things: an
wholesale and you have read the three guards below, unrelated lineage, a serial lower than the
because the restore case trips one of destination, and an EQUAL serial whose contents
them. differ. Restoring after something wrote to
state is the second, so you meet a refusal
rather than an overwrite. The quiet danger is a
hand-edited pull with the serial bumped: that
reads as newer, passes all three, and
overwrites wholesale. -force removes the
checks.
terraform state rm ADDR state only You are about to re-import the same The resource keeps running and is now
resource under a different address in unmanaged. Use it thinking it deletes and you
the same change window. are paying for an orphan nobody plans against.
terraform state mv SRC DST state only A resource moved address because of a A typo'd destination silently creates a state
rename or a module refactor and the entry that matches no config block, and the
remote object is unchanged. next plan proposes to destroy the real object.
terraform state state only A provider source address changed, as in Rewrites EVERY resource using the from-provider
replace-provider a registry namespace move, and the in one pass, so a wrong FQN moves the whole set
resource schemas are compatible. at once. Point it at an incompatible provider
and state holds entries the new one cannot
decode. Take the pull first and do not count on
an automatic backup here.
terraform import ADDR ID state only A live object exists and a matching A wrong ID binds a real object to that address
(writes the object config block already exists for it. with no check that it is the one you meant, and
into state) the next plan reconfigures or replaces it. A
matching resource block is required before you
import, so create it first rather than treating
import as the step that finds what is missing.
terraform force-unlock ID the lock record You have confirmed the holder named in You break a lock held by a run that is still
only the Lock Info block is dead. mid-apply. Two writers, one state file, and a
serial you cannot reconstruct.
terraform apply state only Credentials and account/project resolve to Wrong-scoped credentials read EVERY resource as
-refresh-only (rewrites from live the same place the state was written from, gone, and applying that plan forgets the whole
attributes) and attributes drifted for a resource state rather than one object. It also adopts
already in state. Check that first. nothing new: a resource created outside Terraform
stays outside. And if an object in state really
was DELETED outside Terraform, applying drops it
and the next apply proposes to create it. Read
the plan; this writes state.
state rm does not delete. refresh-only does not adopt. Those two sentences are most of what goes wrong.
Two rows deserve more than a table cell. terraform state push is the only command here that can lose resources in a single keystroke, because it replaces rather than merges. Terraform does check lineage and serial and will refuse a push it considers a regression, and there is a force option that skips those checks, which means the guardrail is exactly one flag away from being off. And the checks only run against a destination that already holds state. Push into an empty one, which is exactly what declining a -migrate-state prompt leaves behind, and none of them apply, with no flag at all. Treat any push as a restore operation with a change record behind it, not as an edit.
terraform force-unlock is the one people reach for fastest and should reach for slowest. The lock error prints who holds it, and that block is the entire decision.
Error: Error acquiring the state lock
Error message: ConditionalCheckFailedException: The conditional request failed
Lock Info:
ID: 4d1f9b0c-6a37-4d2e-9c11-2f8a3e5b7d41
Path: tf-state-prod/data-layer/terraform.tfstate
Operation: OperationTypeApply
Who: ci-runner@runner-7
Version: 1.9.5
Created: 2026-08-24 15:31:07.882119 +0000 UTC
Info:
Operation: OperationTypeApply and a Created stamp two minutes old is a live writer. Force-unlocking that is how you get a state file nobody can reconstruct.
The rule we give teams: force-unlock only after you have found the holder in the other system. If Who names a CI runner, open that job and confirm it exited. If it names a person, message them. A client that reports a held lock while the DynamoDB console shows no lock item usually means the console is looking at a different table, region or account from the one the backend uses. Find the one the backend names before anything else. An IAM or region mismatch on the lock table itself fails fast, with an access or not-found error, not a lock you cannot see. And with use_lockfile there may be no table item to find at all: the lock is a .tflock object beside the state in the bucket.
Adopting a console hotfix: why refresh-only is the wrong tool and import needs config first
The refresh-only pass that adopted nothing
When we reached the reconciliation step, the first plan looked correct and was not. A firewall rule created by hand during an outage needed to come under management, and terraform apply -refresh-only is the command everyone suggests. It ran, it reported no changes to state, and the next normal plan still proposed to create a firewall rule that already existed. Refresh-only updates recorded attributes for resources Terraform already tracks. An object that has never been in state is invisible to it. That distinction cost about forty minutes.
The second wrong turn was importing before writing the config block. Import will not run at all until a resource block matches the address you name. It stops with Error: resource address "google_compute_firewall.allow_health_checks" does not exist in the configuration. and writes nothing, so what the mistake costs you is the change window, not the resource. The order is fixed for that reason: write the block, then import, then plan and expect zero changes. The destroy proposal people half-remember comes from the other direction, deleting a config block after a successful import. On 1.5 and later you can declare import {} blocks in configuration and see the adoption in the plan output before anything is written, which is what we now default to, because it makes the import reviewable in a PR instead of being a thing someone typed.
# 1. Snapshot first. Always.
terraform state pull > pre-reconcile.tfstate
# 2. Config block exists for the console-created rule, then declare the import.
# main.tf
resource "google_compute_firewall" "allow_health_checks" {
name = "allow-health-checks"
network = google_compute_network.core.id
# ... attributes matched to the live object
}
import {
to = google_compute_firewall.allow_health_checks
id = "projects/analytics-prod/global/firewalls/allow-health-checks"
}
# 3. Review the adoption in the plan BEFORE it is written to state.
terraform plan -out=reconcile.tfplan
# 4. Only then.
terraform apply reconcile.tfplan
The import block turns state adoption into something a reviewer can see in a PR diff. The CLI import command writes first and shows you afterwards.
The other half of the reconciliation was the reverse case: a database instance recorded in state at an address that no longer matched a live object, because it had been applied from the stale side of the split. That is a terraform state rm followed by an import at the correct address, and the ordering matters for a reason people miss. Removing it first is what makes the address available; importing over an occupied address fails. The instance kept serving traffic through both commands, which is the point of state rm and also exactly why it is dangerous when someone reaches for it expecting a delete.
The tell that you are done is a plan with no changes and no refresh noise. If plan still proposes to create something you can see running, read what it pairs that create with. A destroy of an address you recognise means an address mismatch, and the fix is state mv or a moved {} block, not another import. A create with no matching destroy means the object is not in state at all, and it does need the import. Teams stuck in that loop for more than an hour usually have a module refactor tangled into the same change; we cover unwinding those separately under Terraform and IaC debt.
The changes that stuck afterward were small. An 11-line pre-commit hook that rejects any staged path matching *.tfstate or *.tfstate.backup, which has fired twice since on branches nobody would have checked. A CI step that runs jq -r '.backend.type' .terraform/terraform.tfstate after init and fails the job unless it prints s3, which catches a job that initialised without the backend, or against a different one, before it plans against the wrong state. And a rule that any console change during an incident gets an import block opened in the same hour, not the same week, because state drift you know about is a ticket and state drift you forget about is an outage with a four-hour tail.
Common questions about terraform state commands and backend migration
The questions that come next
These are the follow-ups we field most often after a state split, answered for Terraform CLI open source with an S3 backend.
| Step | What it does |
|---|---|
| Does terraform state rm delete the resource? | No. It removes the entry from state and leaves the object running in the cloud, unmanaged. That is what you want immediately before re-importing it at a corrected address, and it is a bill you keep paying if you meant to destroy something. Use terraform destroy with a -target for that case, after a plan you have read. |
| Is terraform state pull safe on production? | Yes. It reads the configured backend and writes to stdout. Redirect it to a file before every other command in this reference. It is the only free insurance in the whole list, and the teams who skip it are the ones with no way back after a bad push. |
| Can I force-unlock when DynamoDB shows no lock item? | There is nothing to unlock in that case, so look at the connection instead. Check that the lock table name, region and IAM permissions in your backend config match the table you are inspecting in the console, and confirm the credentials the CLI resolved are the ones you think they are. |
| Does -migrate-state work between any two backends? | It offers to copy state from the previously configured backend to the newly configured one, including local to S3 and back. Read the source and destination in the prompt before confirming. If either side might be stale, answer no and diff them first. |
| Will apply -refresh-only pick up a resource I built in the console? | No. It refreshes attributes for resources already in state. Adopting something Terraform has never seen requires an import, and the config block for it has to exist first, or the CLI import refuses and writes nothing. |
Getting a second pair of eyes on a split state before you push anything
When both states look plausible and applies are blocked
The hard part of a state split is never the commands. It is the twenty minutes where two states each look defensible, applies are blocked, and the fastest-looking move is a state push that quietly drops four resources into being unmanaged. Nobody wants to be the person who made that call alone at the end of a long day, and the decision is genuinely hard: it depends on which side has the console hotfix, which side CI last wrote to, and whether the lock you are staring at belongs to a job that is still running.
We do this reconciliation work with platform teams often enough that we walk in with the diff commands already written and a snapshot taken before anything else happens. What we bring is mostly the discipline to not push until the two states have been diffed resource by resource against the cloud, plus enough scar tissue to recognise which flavour of drift you have from the first plan output. It is unglamorous and it is the difference between a four hour incident and a two day one.
If you are looking at two state files right now and cannot tell which one is authoritative, do not push either. Take a terraform state pull from the remote backend, keep the local file, and book an infrastructure review; we will sit on a call the same day and pick the authority with you before anything writes.
Originally published at https://infraforge.agency/insights/terraform-state-commands-safe-reference/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)