Drift Detection and Reconciliation

Drift is what happens when someone changes a resource in the Console or with gcloud and the Terraform code no longer matches reality — a firewall rule widened during an incident, a label added by hand, a service-account role granted out-of-band. The code still says one thing; the cloud now says another. Drift is not a bug in Terraform; it is the gap that opens whenever a change bypasses the pipeline.

A scheduled plan in CI is how you catch it. A non-empty plan on a config nobody touched means reality moved, and then you decide whether to reconcile the code to reality or revert reality to the code. The danger is not the drift itself but discovering it at the worst time — when an unrelated apply tries to revert someone's emergency fix mid-deploy.

Drift on GCP Specifically

GCP makes drift easy to create. The Console and gcloud make out-of-band changes trivial — a few clicks or one command and a managed resource has changed without going through Terraform. And GCP's eventually-consistent IAM means a hand-granted role shows up as drift on the next plan even when the grant was reasonable. The cloud does not stop anyone from editing what Terraform owns; it just records the change for the next plan to find.

The sharpest GCP drift source is the IAM _member / _binding / _policy trap from Chapter 8. Each of those resources manages a different scope — one member, one role's bindings, or the whole policy — so a role granted by hand may or may not surface as drift depending on which resource type your config uses. Manage members additively and an out-of-band grant can hide entirely, because Terraform is only watching the members it knows about.

Scheduled plan in CI

A Cloud Build trigger or a GitHub Actions cron runs terraform plan on a schedule against an unchanged config. A non-empty plan is the signal that something changed outside Terraform. The job's job is to surface that signal, not to apply — it reports drift and stops, leaving the decision to a human. Run it nightly and drift is caught in hours instead of discovered weeks later during an unrelated change.

The cron below runs a plan with a detailed exit code, so the pipeline can tell "no drift" from "drift found" from "error" and alert only on the middle one.

# exit 0 = no changes, 2 = drift detected, 1 = error
terraform plan -detailed-exitcode -refresh-only -input=false
if [ "$?" -eq 2 ]; then
  notify "drift detected in apps/prod"   # alert a human; do not apply
fi

Wiring the exit code to an alert is the difference between a job that runs silently and one that tells you something moved. The plan finds the drift; the notification makes someone look.

Reading Drift with -refresh-only

terraform plan -refresh-only updates state from reality without proposing config changes, showing you exactly what drifted at the state level. It is the read-only way to see drift, separate from a plan that wants to "fix" reality back to the config. A plain plan mixes two things — pending config changes you have not applied, and drift that happened in the cloud — and -refresh-only isolates the second so you see purely what reality did.

Use it when you want to understand a drift before deciding what to do about it. A normal plan answers "what would apply change"; -refresh-only answers "what changed underneath me," which is the question drift detection is actually asking.

Reconciling vs Reverting

There are two valid responses to drift, and the choice is a judgment call, not an automatic one. You reconcile the code to match a deliberate manual change — someone fixed prod correctly during an incident and you codify what they did so the code tells the truth again. Or you revert reality to the code by running apply — an unauthorized or accidental change you undo because it should never have happened.

Treating every drift as something to revert is a mistake as common as ignoring drift entirely. Sometimes the manual change was the right call and the code is what is wrong; reconciling the code, not reverting the fix, is then the correct move. The detection job surfaces the drift; a human decides which direction to close the gap.

Alerting and Why Auto-Apply Is Dangerous

Wire the scheduled job to alert when the plan is non-empty — a Cloud Monitoring signal, a Slack message, a failed build. The point is that drift gets noticed in hours, not discovered during the next unrelated change when an apply silently reverts it. An unmonitored drift is a landmine the next deploy steps on.

Do not auto-apply on a non-empty scheduled plan. Automatically reverting any drift can undo a legitimate emergency fix someone made in the Console during an incident, turning your drift-detection job into the cause of the next outage. Detection should alert a human; reversion should be a decision. That separation is the whole safety property — the job that finds drift must never be the job that blindly fixes it.