Chapter 13: Production Operations & Advanced GCP
Topic 80

State Surgery

StateOperations

State surgery is editing what Terraform thinks exists without touching what actually exists. Three operations do it: terraform state rm drops a resource from state, terraform state mv renames its address, and the -replace= plan flag forces a recreate. You reach for them when the map — your state — and the territory — the real GCP API — have diverged, and no amount of HCL editing reconciles the two.

Every one of these commands is a loaded gun. The state file has no undo, so a wrong rm orphans a live resource that still bills and a wrong mv makes the next apply quietly delete and recreate something in production. Treat surgery as the fallback, not the first move, and reach for the declarative replacements — moved blocks and import — before you ever touch the CLI.

state rm — Forget, Don't Destroy

terraform state rm drops a resource from state without destroying it on GCP. Terraform simply stops managing it; the live resource keeps running, keeps billing, and is now invisible to your configuration. The legitimate uses are narrow: handing a resource off to a different configuration that will import it, or abandoning one you have decided to manage by hand from now on.

The danger is the opposite of what people expect. rm does not delete — it forgets. Run it on a resource you meant to destroy and the Cloud SQL instance is still there, still serving, still on the bill, and Terraform will never propose to clean it up because as far as state is concerned it does not exist. A forgotten resource is an unmanaged orphan that nobody tracks until the invoice arrives.

Drop a resource from state without touching GCP
# Terraform forgets this SA; the live service account is untouched and still works
terraform state rm 'google_service_account.event_processor'

state mv — Rename the Address

terraform state mv changes a resource's address in state, which you need whenever you rename a resource in HCL or move it into a module. Done right, the plan afterward is empty — you told Terraform the new name points at the same real object. Skipped or done wrong, the next apply sees the old address gone and a new one appearing, so it destroys the old resource and creates the new.

For the Hatch event-processor service account, an un-mv'd rename is not cosmetic. Terraform destroys google_service_account.event_processor and creates a fresh google_service_account.processor with a new identity, which means every IAM binding that referenced the old account silently stops pointing at anything that has the permissions — a working pipeline broken by what looked like a rename.

Rename an address so the next plan is a no-op
# You renamed the resource in HCL from event_processor to processor
terraform state mv \
  'google_service_account.event_processor' \
  'google_service_account.processor'

-replace= — Force a Clean Recreate

-replace=ADDRESS is the modern replacement for the deprecated taint. It forces one resource to be destroyed and recreated on the next apply without editing state at all — you pass it to plan or apply and Terraform marks just that resource for replacement. Reach for it when a resource is wedged: a half-created Cloud SQL instance, a managed instance group that drifted into a state the provider cannot reconcile, anything where a clean recreate is the actual fix.

Because it triggers a real destroy-and-create, read the plan first. On a stateless resource the recreate is harmless; on a regional Cloud SQL primary it is an outage. The flag is a scalpel, not a reset button — it does exactly one recreate, but that recreate is as consequential as any other.

Force a single resource to be recreated
# Replaces the deprecated `terraform taint`; check the plan for downtime first
terraform apply -replace='google_sql_database_instance.replica'
Three state-surgery operations, three effects
state rm
Forgets a resource; the live object keeps running and billing. Does not destroy.
state mv
Renames the address so the same real object answers to a new name. Plan stays empty.
-replace=
Forces one resource to destroy and recreate on the next apply.

Why Hand-Editing State Is the Last Resort

terraform state pull and push let you fetch the raw JSON, edit it, and write it back — and that bypasses every safety check the CLI gives you. Hand-editing can corrupt the serial (the write counter) or the lineage (the UUID identifying the state's history), and against a remote GCS backend it races every other writer. Get the serial wrong and the next remote write is rejected as out of order.

If you find yourself hand-editing JSON, you have almost always skipped a tool that would have done the change safely. A rename belongs in a moved block; pulling an existing resource under management belongs in import. Hand-editing is what you do when nothing else can express the change, and that situation is rarer than it feels in the moment.

The Safer, Declarative Replacements

moved blocks turn a rename or a module-move into a declarative, committed, reviewable change. You add the block to your configuration, it ships in a PR, and everyone's next plan applies the move — no one runs a one-off CLI command on one laptop. import (and import blocks) pulls an existing resource into state without any surgery at all. Reach for both of these before state mv or rm; they leave a trace in the code, and the CLI commands leave none.

A rename as code, not a one-off command
# Committed to git, applied by everyone's next plan, reviewable in the PR
moved {
  from = google_service_account.event_processor
  to   = google_service_account.processor
}
moved block vs terraform state mv

moved block — declarative, committed to git, applied by everyone's next plan, and reviewable in a PR. It is the default for any rename or refactor within a single state file, because the change lives in the code and survives in history.

terraform state mv — an imperative CLI command run once, on one machine, leaving no trace in the configuration. Reserve it for the case a moved block cannot express: moving a resource between two separate state files.

Common Mistakes
  • Renaming a resource in HCL or moving it into a module without a moved block or state mv, so the next apply destroys the old address and recreates it — for a google_sql_database_instance that is a database wiped and rebuilt.
  • Running terraform state rm on a resource you meant to delete, then being confused that it still exists and still bills — rm forgets it, it does not destroy it.
  • Hand-editing the state JSON with state pull/push instead of using import or a moved block, and corrupting the serial so the next remote write is rejected as out of order.
  • Using -replace= on a resource that cannot recreate without downtime, like a regional Cloud SQL primary, without sequencing it — the recreate is an outage, not a fix.
  • Doing surgery against shared remote state without holding the lock, so a teammate's apply interleaves and one of you clobbers the other's write.
Best Practices
  • Reach for moved blocks and import first, and treat state mv/rm as the fallback only when a declarative form cannot express the change.
  • Run terraform plan immediately after any surgery and confirm it shows no unexpected create or destroy before you apply.
  • Use -replace=ADDRESS instead of the deprecated taint to force a recreate, and read the plan for downtime implications first.
  • Pull a fresh backup of state, or rely on GCS object versioning, before any manual operation so a mistake is recoverable.
  • Hold the state lock for the whole surgery and never edit shared remote state while another apply could be running.
Comparable tools moved · import blocks the declarative alternatives to CLI surgery Pulumi pulumi state commands for the same operations Config Connector no state file, so no surgery — the controller reconciles drift

Knowledge Check

What happens to the live GCP resource when you run terraform state rm on it?

  • Nothing — it keeps running and billing; Terraform just stops managing it and forgets it exists
  • It is destroyed on GCP, deleted through the same API call as terraform destroy
  • It is recreated from scratch on the next apply, with a fresh resource id
  • It is moved into a different state file automatically and re-keyed under the same address there, leaving the original state empty

You rename a resource in HCL and apply without a moved block or state mv. What does Terraform do?

  • Destroys the resource at the old address and creates a new one at the new address
  • Recognizes the rename by attribute fingerprint and produces an empty plan
  • Refuses to apply until you manually edit the state JSON to repoint the address
  • Keeps both the old and new resources running side by side in state

When is -replace=ADDRESS the right tool?

  • When a single resource is wedged and a clean destroy-and-recreate is the fix — after checking the plan for downtime
  • When you want to rename a resource's address in state, repointing it to a new block name without touching, recreating, or changing the underlying real object on GCP at all
  • When you want to drop a resource from state while keeping the live object running untouched on GCP
  • Whenever any apply fails for any reason, as a generic catch-all retry mechanism

Why is hand-editing the state JSON with pull/push the last resort?

  • It bypasses every safety check and can corrupt the serial or lineage, so a declarative moved or import is almost always the better path
  • It is slower than the dedicated CLI subcommands and needs a manual JSON reformat and re-indent before you push, but the result is otherwise byte-for-byte equivalent and just as safe as state mv
  • It only works against a local backend file and is rejected outright by the GCS remote backend
  • It permanently locks the state object so no one else on the team can ever apply again

You got correct