Debugging and GCP Propagation Gotchas

Most Terraform-on-GCP failures are not bugs in your HCL — they are timing. GCP grants IAM, enables APIs, and creates resources asynchronously, so a dependent resource in the very same apply hits a control plane that has not caught up yet and fails with a permission or "not enabled" error that vanishes when you re-run. The configuration is correct; the clock is the problem.

TF_LOG shows you the real API request and response behind Terraform's tidy error message, and time_sleep, depends_on, and provider-level retry are the fixes for the propagation class of failure that defines debugging on GCP. Learn to recognize the signature — fails on a fresh apply, passes on the re-run — and most of these stop being mysterious.

TF_LOG and TF_LOG_PATH

Setting TF_LOG=DEBUG (or TRACE for more) exposes every HTTP request and response between the provider and the Google APIs, and TF_LOG_PATH writes it to a file so you can read it without it scrolling past. This is how you turn "Error 403" into the exact API, method, and message that actually failed — the difference between guessing at a cause and reading it.

# Exposes every request/response between the provider and the Google APIs
export TF_LOG=DEBUG
export TF_LOG_PATH=./tf-debug.log
terraform apply

Reading a Google API Error

The provider surfaces the raw API error, and learning to read it is faster than guessing. The HTTP code names the class — 403 permission, 404 not-found, 409 conflict — and the body carries the reason and often a precise detail like SERVICE_DISABLED or userProjectMissing. The message usually names the missing grant or the service that was not enabled; once you read it as the API's own words rather than Terraform's, the fix is obvious.

The GCP Propagation Class of Failure

An IAM binding granted in one resource is not instantly visible to a resource that depends on it. An API enabled by google_project_service is not instantly usable by the next resource. So an apply that creates the grant and the dependent in one run fails intermittently — it passes on a re-run because by then the grant has propagated, and fails on a fresh apply because it had not. That pattern, "fails fresh, passes on re-run," is the signature of asynchronous propagation, not a logic error in your HCL.

The Fixes

There are three fixes, and you reach for the lightest that makes the apply deterministic. depends_on orders the dependent strictly after the grant or enablement, so Terraform at least waits for that API call to return. time_sleep, from the hashicorp/time provider, inserts a deliberate wait for the propagation that the API return does not guarantee. And provider-level retry or google_project_service timeouts absorb the transient error. Use depends_on for ordering and add time_sleep only when ordering is correct but the grant still has not propagated.

resource "time_sleep" "wait_for_iam" {
  depends_on      = [google_project_iam_member.processor]
  create_duration = "30s"   # cover propagation, not the API call
}

resource "google_cloud_run_v2_service" "api" {
  depends_on = [time_sleep.wait_for_iam]
  # ...
}

Crash Logs and When It Is Actually a Bug

A true provider crash writes a crash.log with a Go stack trace. That, not a 403, is what you attach to a provider issue — a 403 is the API telling you something is missing, while a crash is the provider itself falling over. Distinguishing a propagation flake (the re-run passes) from a real crash (the re-run still crashes) tells you whether to add a time_sleep or file a bug, and saves everyone the noise of "Terraform is broken" reports that were really IAM timing.