Chapter 13: Production Operations & Advanced GCP
Topic 82

Zero-Downtime Resource Replacement

LifecycleOperations

Some changes force Terraform to replace a resource rather than update it in place, and the default order is brutal: destroy the old, then create the new. That ordering means a window where the resource does not exist — for the seconds or minutes between the destroy and the create, traffic to it fails. On a stateless side resource nobody notices; on anything serving production it is an outage.

create_before_destroy inverts the order so the replacement is live before the original goes away. For the resources that genuinely cannot replace without a gap — a regional Cloud SQL primary, a resource pinned to a globally-unique name — the answer is sequencing and orchestration, not a single lifecycle flag. Knowing which case you are in starts with reading the plan.

Why Replacement Happens

Changing a force-new attribute makes Terraform replace rather than update. A Compute Engine instance's machine_type on some configurations, a bucket's location, a resource's name — these cannot be changed in place, so the plan shows -/+, meaning destroy then create. Recognizing the -/+ marker in a plan is how you catch downtime before it ships, because the change that triggers it often looks like an innocent edit.

The plan marker that means destroy-then-create
# Changing location forces replacement — the -/+ is the warning
  # google_storage_bucket.raw must be replaced
-/+ resource "google_storage_bucket" "raw" {
      ~ location = "US" -> "EU" # forces replacement
    }

create_before_destroy

The lifecycle block's create_before_destroy = true flips the order: Terraform stands up the new resource first and only then tears down the old. The replacement is live before the original disappears, so there is no gap. The catch is that both exist briefly, which means anything with a globally-unique name collides with itself — a GCS bucket, an instance with a fixed name — unless the name can vary, typically through name_prefix or a random suffix.

Invert the order — but only with a varying name
resource "google_compute_instance_template" "app" {
  name_prefix = "hatch-app-"   # varying name so old and new don't collide
  # ...
  lifecycle {
    create_before_destroy = true
  }
}

MIG Rolling Updates

A managed instance group does zero-downtime replacement natively, and this — not create_before_destroy on individual VMs — is how you reroll a Hatch compute tier. You point the MIG at a new instance template and configure update_policy: surge instances, max unavailable, a rolling strategy. The group then rolls VMs over gradually, a few at a time, with health checks gating each batch, instead of replacing the whole fleet at once.

A MIG rolls the fleet instead of swapping it
resource "google_compute_region_instance_group_manager" "app" {
  version {
    instance_template = google_compute_instance_template.app.id
  }
  update_policy {
    type                  = "PROACTIVE"
    minimal_action        = "REPLACE"
    max_surge_fixed       = 3   # add new VMs before removing old
    max_unavailable_fixed = 0   # never drop below capacity
  }
}
Two zero-downtime paths for replacement
create_before_destroy
Single resource: stand up the new before tearing down the old. Needs a varying name.
MIG rolling update
New instance template plus update_policy and health checks roll VMs a few at a time.

Draining and Health Gating

A zero-downtime rollover is only real if the new instances are healthy before they take traffic and the old ones drain in-flight connections before they die. Load-balancer health checks gate the first half: the LB does not route to an instance until its check passes. Connection draining gates the second: an instance being torn down finishes its in-flight requests instead of dropping them. Skip either and the surge-and-replace stops being safe — the LB sends a request to a not-yet-ready instance, or kills a connection mid-response.

Resources That Cannot Replace Cleanly

Some resources have no in-place zero-downtime path at all. A regional Cloud SQL primary, a resource pinned to a fixed global name, anything with external dependents that cannot be repointed atomically — no lifecycle flag covers these. The answer is a sequenced runbook expressed as staged applies: stand up a replica, promote it, repoint the clients, retire the old primary. That is operations work you script and stage deliberately, not a hope that one flag makes the gap disappear.

create_before_destroy vs a MIG rolling update

create_before_destroy — a Terraform lifecycle ordering flag for a single replaceable resource. It needs a varying name (name_prefix or a random suffix) so the new and old do not collide while both briefly exist. Use it for standalone resources.

A MIG rolling update — GCP-native fleet orchestration driven by a new instance template and an update_policy, with health checks and surge built in. Use it for any group of instances behind a load balancer, where rolling a few VMs at a time is the safe path.

Common Mistakes
  • Adding create_before_destroy to a google_storage_bucket or an instance with a fixed name and hitting a name-collision error, because both the old and new exist at once and bucket names are globally unique.
  • Changing a MIG's instance template without an update_policy, so the group replaces every VM at once instead of rolling, and the service drops traffic during the swap.
  • Missing the -/+ in a plan and shipping a force-new change to a stateful resource, taking an unplanned outage on what looked like an edit.
  • Rolling a MIG without health checks or connection draining, so the load balancer routes to instances that have not finished starting and to instances being torn down mid-request.
  • Assuming a regional Cloud SQL primary can be replaced with zero downtime by a lifecycle flag — it cannot; the change needs a replica-promote-repoint runbook.
Best Practices
  • Read every plan for -/+ replacement and decide deliberately whether the resource can tolerate the gap before applying.
  • Use create_before_destroy only on resources whose name can vary — use name_prefix or a random suffix — so the new and old do not collide.
  • Drive instance-fleet changes through MIG instance-template swaps with a surge-based update_policy, health checks, and connection draining.
  • Write an explicit staged runbook for resources with no clean replacement path — stateful primaries, fixed-name globals — rather than relying on a lifecycle flag.
  • Keep max_unavailable at zero on any production MIG with an SLO so the fleet never drops below capacity during a roll.
Comparable tools AWS provider create_before_destroy plus the ASG rolling-update analog Kubernetes · Config Connector rolling pod replacement via Deployments Cloud Deploy · Spinnaker higher-level deploy orchestration

Knowledge Check

Why does create_before_destroy collide on a resource with a globally-unique name like a GCS bucket?

  • It creates the new resource before destroying the old, so both exist briefly and a fixed unique name conflicts with itself
  • It issues two back-to-back destroy calls on the same bucket, which the GCS API rejects
  • A google_storage_bucket cannot legally carry a lifecycle block of any kind
  • It forces the bucket to move into a different region as part of the create step, and GCS disallows changing the location of an existing bucket in the middle of a replacement like that

How does a MIG achieve zero-downtime replacement that a single-resource lifecycle flag does not?

  • You point it at a new instance template and an update_policy rolls VMs over gradually with surge and health checks
  • It stamps create_before_destroy onto every member VM in the group automatically
  • It pauses all inbound traffic at the load balancer, tears down and swaps the entire fleet of VMs at once in a single batch, then lifts the pause and resumes serving once the new ones are up
  • It keeps two complete fleets running permanently at double the cost

Why are health checks and connection draining required for a safe rollover?

  • Without them the LB routes to instances that have not finished starting and kills connections on instances being torn down mid-request
  • They are only ever needed for stateful workloads and are entirely optional for stateless services
  • They make the roll finish faster but have no effect at all on request correctness during the swap
  • They are mandated purely by Terraform's plan-time schema validation, which rejects a MIG block that omits them, and have no bearing on anything the runtime rollover actually does once the apply succeeds

What is the right approach for a regional Cloud SQL primary that has no clean zero-downtime replacement path?

  • A sequenced runbook — stand up a replica, promote it, repoint clients, retire the old primary — staged as deliberate applies
  • Add create_before_destroy to the instance and let the lifecycle flag migrate the data for you
  • Run a MIG rolling update with surge against the Cloud SQL database instance itself
  • Accept the default destroy-then-create gap and let the apply replace the instance in place, since a managed database snapshots and restores its data fast enough to tolerate the brief outage fine

You got correct