Disaster Recovery for State
The state file is the single most catastrophic thing to lose. Corrupt it or delete it and Terraform forgets every resource it manages, then on the next apply it sees an empty state against a full production org and proposes to create the whole thing from scratch — duplicate buckets, colliding names, partial chaos. Losing one resource is an inconvenience; losing state is forgetting that an entire org exists.
The GCS backend's defense is object versioning. With versioning on, hatch-tfstate keeps every prior version of the state object, so a corruption or accidental delete is a restore, not a rebuild. The rebuild-from-nothing path — when versioning failed too — is terraform import against the live org, and it is exactly as tedious as it sounds.
Why State Loss Is Catastrophic
State is the only record mapping your HCL addresses to real GCP resource ids. Lose it and apply sees an empty state against a full org, then proposes to create duplicates of everything — or worse, hits name collisions on globally-unique resources and stops halfway, leaving the org in a partially-applied mess. This is the failure the whole topic exists to prevent, and every defense below is in service of never reaching it.
GCS Object Versioning as the Recovery Mechanism
With versioning enabled on hatch-tfstate, every write to default.tfstate keeps the previous generation instead of overwriting it. Recovering a corrupted or accidentally deleted state is then listing the object's generations and restoring the last-good one — a minutes-long operation, not a rebuild. This is why versioning on the state bucket is non-negotiable: it is the difference between a restore and reconstructing the org by hand.
# Every write kept a prior generation; list them and restore the last good one gcloud storage ls --all-versions \ gs://hatch-tfstate/default.tfstate gcloud storage cp \ gs://hatch-tfstate/default.tfstate#1699999999000000 \ gs://hatch-tfstate/default.tfstate
serial and lineage
Every state carries a monotonically increasing serial — a write counter — and a lineage, a UUID identifying the state's history. A restored version carries an older serial, and a mismatched lineage is Terraform telling you two state files are not the same state at all. These two fields are how you verify you restored the right object and how the backend detects out-of-order writes: a push with a stale serial is rejected, and a push with the wrong lineage is refused as not a descendant of the stored state.
Backing Up Beyond Versioning
Versioning protects against bad writes and accidental object deletes, but not against a deleted bucket or a compromised project. If hatch-tfstate itself is gone, every generation goes with it. The second line of defense is a copy that lives somewhere else: a periodic terraform state pull to a separate location, or bucket replication to another project. For the org's most important file, one storage location is not a backup strategy.
# Versioning protects the object; this protects against losing the whole bucket terraform state pull > "backups/hatch-$(date +%F).tfstate"
Rebuilding Lost State with import
When state is truly gone — bucket deleted, no off-site copy — you reconstruct it with import. You write an import block (or run terraform import) for each live resource, matched to its HCL address, until plan shows no changes. It is tedious and deterministic, and it is the reason your HCL must accurately describe what is actually deployed: the rebuild only converges if the configuration matches reality. An import-driven rebuild of a full org is days of work, which is the whole argument for the versioning and backups above.
Why the State Bucket Needs Versioning, IAM, and CMEK Together
The three protections solve different problems and only work together. Versioning gives recoverability. Tight IAM — only the CI service account and break-glass admins can read, write, or delete — keeps the org's full blueprint out of reach, because the state file is a complete map of every resource and any sensitive values stored in it. CMEK lets you control and audit the encryption key, and revoke it if the project is compromised. The state file is a credential-grade asset and gets protected like one.
GCS object versioning — automatic, per-write, and recovers from corruption or accidental object deletion in place. It is the first line of defense and is mandatory on any state bucket, but it lives inside the same bucket it protects.
A state pull backup — a deliberate copy to a separate location that also survives bucket or project loss. Run both: versioning protects the object, and an external copy protects against losing the bucket itself.
- Hosting
hatch-tfstatewithout object versioning, so a corrupted write or an accidental delete leaves no prior generation to restore and the only path left is a fullimportrebuild. - Restoring an older state generation without checking the
serial/lineage, then applying against a state that predates real resources and proposing to recreate them. - Granting broad read access to the state bucket, exposing the org's complete resource map and any sensitive values in state to anyone with project viewer.
- Treating versioning as a complete backup and never copying state off the bucket, so a deleted bucket or compromised
hatch-adminproject takes the only copy with it. - Editing or pushing state with a mismatched
lineageand getting a backend rejection, because the pushed state is not a descendant of the stored one.
- Enable object versioning on the state bucket and treat it as mandatory, not optional, for any backend.
- Lock down state-bucket IAM to the CI service account plus break-glass admins, and encrypt with CMEK so the key is auditable and revocable.
- Keep a periodic off-bucket copy of state — a scheduled
state pullor bucket replication — to survive bucket-level loss. - Verify
serialandlineageafter any restore and confirm a cleanplanbefore applying against recovered state. - Keep your HCL describing exactly what is deployed, so an
importrebuild can actually converge to a no-op plan.
Knowledge Check
Why is losing the state file worse than losing a single resource?
- State is the only map from HCL addresses to real resource ids; lose it and apply sees an empty state against a full org and proposes to recreate everything
- A lost resource is permanently unrecoverable, but the state file always rebuilds itself automatically on the next plan by re-reading every live resource in the org through the provider and re-deriving the address mapping
- State loss only affects the computed output values, never the real underlying resources
- Losing the state file immediately destroys every managed resource across the GCP org
How does GCS object versioning recover a corrupted state file, and why is it mandatory?
- Every write keeps the previous generation, so recovery is restoring the last-good one — a minutes-long restore instead of an
importrebuild - It encrypts each state write with a customer-managed CMEK key and verifies the checksum on read, so a corrupted or partially written object can never occur or be served back to the client at all
- It holds a write lock on the object during every apply, which is what prevents corruption
- It replicates the entire state bucket to a second region automatically on each write
What do serial and lineage each tell you?
serialis the monotonic write counter that detects out-of-order writes;lineageis the UUID confirming two files are the same state's history- Both are timestamps of the last write written in RFC 3339 form;
serialrecords the calendar date of that write andlineagerecords the wall-clock time of day down to the second serialis the count of managed resources andlineageis the pinned provider version- They are decorative metadata fields that Terraform records but ignores entirely on read
Why is object versioning not a complete backup on its own?
- It lives inside the same bucket it protects, so a deleted bucket or compromised project takes every generation with it
- It only ever retains the five most recent generations on a fixed hard-coded limit and silently discards everything older the moment a sixth write lands in the bucket
- It cannot recover from an accidental delete of the live state object at all
- It rewrites and corrupts the
lineagefield on every single restore
You got correct