Provisioners and When to Avoid Them

Provisioners — local-exec and remote-exec — run a shell command as part of resource creation, and on GCP they are almost always the wrong tool. They run once at create time, are not reconciled on later applies, and a failure can taint a resource and force a recreate. For anything you would reach for them on a VM, GCE's startup scripts, instance metadata, and cloud-init do the job declaratively and survive a reboot or a replacement.

The reason this matters more on GCP than you might expect is the MIG model from Topic 60. When the group replaces an instance, a provisioner that ran on the original does not run on the replacement — the new VM boots unconfigured. The declarative alternatives travel with the instance template, so every replacement is configured identically. That single difference is why remote-exec essentially never wins for VM bootstrap here.

What Provisioners Are

A local-exec provisioner runs a command on the machine running Terraform — your laptop or the CI runner. A remote-exec provisioner connects to the newly created resource over SSH and runs commands on it. Both are escape hatches for when no resource or API covers what you need, and the word "escape" is doing real work: they are the exit you take when the declarative model has genuinely run out.

They are part of the language, supported, and occasionally correct. But "supported" is not "recommended," and most uses you will see in the wild are reaching for an escape hatch when a real resource or a metadata script was right there.

Run Once, Never Reconciled

A provisioner fires only when the resource is created — or destroyed, for a destroy-time provisioner. Terraform never checks it again and never re-runs it on a later apply. That means it cannot keep anything in a desired state the way a real resource does: if what the provisioner set up drifts, no future apply corrects it, because Terraform has no model of what the provisioner was supposed to achieve.

This is the core disqualifier. Terraform's whole value is continuous reconciliation toward a declared state, and a provisioner opts out of exactly that. Relying on one to keep something configured is relying on a one-shot side effect that the engine forgets the instant it finishes.

Why Startup Scripts Beat remote-exec on GCE

A metadata_startup_script — or cloud-init delivered through metadata — runs on every boot of the VM, needs no SSH path open from the Terraform runner, and travels with the instance template. A MIG-replaced VM is therefore configured identically to the one it replaced. A remote-exec provisioner does none of this: it runs once, needs network reachability and credentials from the runner, and is absent from every replacement instance.

Stack the two up and there is no contest for GCE bootstrap. The startup script is declarative, reproducible, and part of the resource; the provisioner is imperative, one-shot, and external to it.

resource "google_compute_instance_template" "app" {
  project      = "hatch-app-prod"
  name_prefix  = "app-"
  machine_type = "e2-standard-2"

  metadata = {
    # runs on every boot, travels with the template, no inbound SSH needed
    startup-script = "#!/bin/bash\nsystemctl start hatch-agent"
  }

  # NOT this:
  # provisioner "remote-exec" { ... }   # runs once, never on a MIG replacement
}

Metadata and cloud-init as the Declarative Path

Instance metadata is the GCE-native channel for bootstrap configuration, read by the guest agent and by cloud-init. It is part of the resource, versioned alongside the instance template, and applied by the platform rather than by Terraform reaching into the box over the network. Configuration delivered this way is reproduced on every instance the template stamps, automatically.

For Hatch, this is the only sane way to configure a VM in a regional MIG. Bake the heavier setup into the image with Packer, keep the boot-time script minimal, and let metadata carry the small amount of per-boot config — so instances start fast and identically without anyone SSHing in.

The Legitimate Last Resorts

There is a narrow band where a provisioner is defensible. A local-exec that triggers a one-off external action — kicking a webhook, running a CLI for a service with no Terraform provider — is a reasonable use, because there is genuinely no resource for it. Reaching into a VM with remote-exec to configure it is the use to design away every time.

The discipline is to treat provisioners as a last resort and to look hard for a real provider resource or a managed-service feature first, since one almost always exists. When you do use local-exec, keep it to a genuinely external action, not in-box configuration hidden in a shell script that no plan can show and no apply re-runs.