Terraform on GCP
by Sergey Okinchuk
Ch 12State Surgery

Chapter Thirteen

Production Operations & Advanced GCP

The day-two work that keeps a mature org running: editing state without breaking it, walking the google provider across major versions, replacing live resources without dropping traffic, recovering after state loss, debugging GCP's asynchronous propagation, and standing the whole org up on Cloud Foundation Fabric / FAST.

6 topics

Everything before this chapter built infrastructure; this chapter keeps it alive. The Hatch org is past greenfield — a real hierarchy of folders, a fleet of state files in hatch-tfstate, and live resources that bill money and serve traffic. The operations that matter now are the ones you run while the system is in production, where a wrong command is not a failed plan but an outage or a wiped database.

Six topics cover that surface. State surgery and its safer declarative replacements. Upgrading the google provider across dozens of stacks without a week of firefighting. Replacing resources with zero downtime. Disaster recovery when state is corrupted or gone. Debugging the propagation and eventual-consistency failures that define Terraform-on-GCP. And, last, building the org's whole foundation with Cloud Foundation Fabric and FAST instead of hand-assembling it from primitives.

Topics in This Chapter

Topic 80
State Surgery
Editing what Terraform thinks exists without touching what does — state rm, state mv, and -replace=. Why each is a loaded gun, and why moved blocks and import are the safer declarative path you reach for first.
StateOperations
Topic 81
Upgrading Providers and Terraform Versions
The two versions you pin, reading the google provider's upgrade guide, and staging a major bump one stack at a time across the Hatch fleet instead of breaking every stack in one PR.
VersioningOperations
Topic 82
Zero-Downtime Resource Replacement
Catching the destroy-then-create gap in a plan, inverting it with create_before_destroy, and driving fleet rollovers through MIG rolling updates with health checks and draining.
LifecycleOperations
Topic 83
Disaster Recovery for State
Why losing state is the worst failure, how GCS object versioning makes it a restore not a rebuild, what serial and lineage guarantee, and rebuilding from nothing with import.
StateRecovery
Topic 84
Debugging and GCP Propagation Gotchas
TF_LOG and reading a raw Google API error, the propagation class of failure that passes on re-run, and the depends_on / time_sleep fixes for GCP's asynchronous control plane.
DebuggingGCP
Topic 85
Landing Zones with Cloud Foundation Fabric / FAST
What a landing zone provides, the staged FAST bootstrap, greenfield versus the harder import-driven brownfield adoption, and where Config Connector fits as the KRM alternative.
Landing ZoneGCP