Drift Detection and Reconciliation

Drift is the gap between what Terraform's state says exists and what is actually in AWS. It opens when someone changes a resource in the console, an autoscaler adjusts a desired count, or another tool edits a tag — anything that touches a Terraform-managed resource outside Terraform. Detecting drift early and reconciling it deliberately is core to operating Terraform-managed infrastructure; ignore it and the next unrelated apply produces a surprising change nobody planned.

The trap is that the same apply that "fixes" drift can also undo a legitimate change. Reconciliation is a decision, not a reflex: sometimes the right move is to revert reality back to the code, and sometimes it is to accept reality into state. Getting that call wrong either re-breaks something someone fixed or bakes a console mistake into your configuration.

What Causes Drift

Most drift comes from people: a console change made during an incident, a quick fix applied by hand and never codified. The rest comes from systems — an autoscaler that owns the desired_capacity of an ASG and moves it independently, another tool that writes tags, or AWS itself applying a default to an attribute after creation. All of it is reality diverging from what Terraform last recorded, and Terraform has no idea until it next refreshes.

Detecting Drift

You detect drift by refreshing state against reality and seeing whether they disagree. The clean way to do this without touching anything is a plan -refresh-only run on a schedule in CI — nightly, say — that alerts when the refresh shows a difference. A plain scheduled plan works too, but -refresh-only isolates "reality changed" from "the config changed," which is exactly the signal you want for drift.

# scheduled nightly in CI
terraform plan -refresh-only -detailed-exitcode

# exit 0 = no drift, 2 = drift detected, 1 = error
# a non-zero "changes" exit fires the alert

-refresh-only updates Terraform's view of reality and reports what differs from state, changing nothing. With -detailed-exitcode, an exit code of 2 means a difference was found — the pipeline turns that into an alert. This is the difference between learning about drift on your schedule and learning about it mid-incident when an unrelated apply suddenly wants to revert a console hotfix.

Reconciliation Options

Once you have found drift, there are three ways to reconcile. Let Terraform revert the change with a normal apply, putting reality back to the declared config. Accept reality into state with a -refresh-only apply, or stop fighting the attribute with ignore_changes if it will keep changing. Or, if the resource was created out of band entirely, import it so Terraform starts managing it. Which one is right depends on whether the out-of-band change should have happened.

The refresh-only Plan

A -refresh-only apply is the safe reconciliation primitive: it updates state to match reality without changing any infrastructure. It is how you accept a legitimate external change — the autoscaler moved the count, and you want state to record the real value rather than have the next apply drag it back. It touches nothing in AWS; it only rewrites Terraform's bookkeeping to agree with what is already there.

Preventing Drift

The cheapest drift to handle is the drift that never happens. Reduce console write access on Terraform-managed resources so the path of least resistance is a code change, not a click. When you find people repeatedly changing the same thing by hand, that is a signal to codify it — make the console the harder option. Drift is rarely malice; it is usually convenience, and you fix it by making the convenient path go through Terraform.