Disaster Recovery for State

State is the most critical and most fragile thing Terraform owns. Lose it or corrupt it and Terraform no longer knows what it manages — even though every resource it created is still running. The infrastructure is fine; the map to it is gone. The next plan, with an empty or wrong state, proposes to create everything from scratch, which collides with the resources that already exist.

Disaster recovery for state means versioning, backups, and a procedure you have actually rehearsed. Done right, a corrupted write or a fat-fingered state rm is a two-minute rollback to a prior version; done wrong, it is a reconstruction project where you rebuild the map by hand against live infrastructure you are afraid to touch.

The Failure Modes

State fails in a handful of recognizable ways. A corrupted write leaves the file truncated or malformed, usually from an interrupted apply. An accidental state rm or destroy removes entries that should have stayed. A deleted state object — someone cleaned up the wrong S3 key — leaves Terraform with nothing. And concurrent writes against a backend with no effective locking — an unlocked remote backend, or a local file shared by hand across machines — interleave two applies into a file that reflects neither correctly.

Every one of these shares the same shape: the infrastructure is untouched, but Terraform's record of it is wrong or missing. That is what makes state DR tractable — you are recovering a record, not rebuilding servers. The whole game is having a known-good version of that record to restore.

S3 Versioning as the Safety Net

Bucket versioning on the state bucket is the primary recovery mechanism, and the single most important thing in this topic. With it enabled, every write to the state object — every apply, every surgery command — creates a new object version, and the previous versions stay retrievable. Recovery from almost any failure becomes restoring the object version from just before the bad write.

resource "aws_s3_bucket_versioning" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
  versioning_configuration {
    status = "Enabled"
  }
}

A state bucket without versioning is an accident waiting to happen, because a corrupted write overwrites the only copy and there is no prior version to go back to. Enable versioning before the bucket holds anything that matters; it is the difference between a quick rollback and a reconstruction.

Backups

Terraform's state commands write an automatic local backup before they mutate state, timestamped so you can restore the prior file. That backup is real but limited: it lives on whoever ran the command, covers only commands run locally, and is easy to lose. It is a convenience, not a disaster-recovery plan.

For critical states, add deliberate periodic backups on top of versioning — a scheduled copy of the state object to a separate bucket or account, so a failure that takes out the primary bucket itself still leaves a recovery point. Versioning protects against bad writes; an out-of-band backup protects against losing the bucket.

Recovery Procedure

Recovery is restore, then reconcile. Restore the last known-good object version, replacing the corrupted or missing state. Then run a plan — not an apply — to compare the restored state against current reality, because anything that changed in AWS between that version and now shows up here as drift the restored state does not know about. Reconcile that drift before applying anything.

# list versions, identify the last good one, restore it as current
aws s3api list-object-versions --bucket acme-tfstate \
  --prefix apps/checkout/terraform.tfstate
aws s3api copy-object --bucket acme-tfstate \
  --key apps/checkout/terraform.tfstate \
  --copy-source acme-tfstate/apps/checkout/terraform.tfstate?versionId=GOOD_ID

# reconcile: plan against reality before applying anything
terraform plan

Skipping the reconcile step is the trap. Restoring an old version and immediately applying treats a stale snapshot as truth, so anything that changed since that version gets reverted or recreated. The plan after restore is what catches the gap; read it and reconcile before any apply.

Reconstruction as Last Resort

When there is no backup and no prior version — versioning was never on, or every copy is gone — the only path left is rebuilding state by importing each resource one at a time. This is slow, error-prone, and exactly the situation versioning exists to prevent. For an estate of any size it is days of careful work matching live resources to config addresses, with a destructive plan waiting on every resource you import imperfectly.

You never want to be here. The entire point of versioning, backups, and a rehearsed procedure is that reconstruction stays hypothetical. If you find yourself importing a production estate from scratch, the failure was not the lost state — it was never enabling the versioning that would have made the loss a rollback.