Team Backends on GCS

A solo backend is one bucket and one engineer running apply. A team backend is the same hatch-tfstate bucket carved into many prefix-separated stacks, with native GCS locking so two pipelines never write the same state at once, and bucket IAM tight enough that only CI and admins can touch any of it. The bucket from Chapter 2 already has object versioning and a customer-managed encryption key; this topic is about making it safe for many hands.

Nothing here is a new bucket. The whole move from solo to team is policy on the one bucket you already have: how stacks share it, how locking prevents corruption, who has a write path to state, and how you recover when an apply goes wrong. Get those four right and a dozen engineers can run Terraform against the same backend without ever stepping on each other.

One Bucket, Many Prefixes

Every stack in the org uses the same hatch-tfstate bucket and is separated by a distinct prefix — prod/network, apps/prod, data/prod. The prefix is the only thing that changes between one stack's backend block and another's, so the configuration is nearly identical everywhere and the bucket's IAM, versioning, and CMEK policy apply uniformly to all of it. One bucket to secure, one policy to reason about, many stacks behind it.

The backend block below is what every stack carries, varying by a single line. That single-line difference is deliberate: it means a new stack inherits the bucket's entire security posture for free, rather than recreating it.

terraform {
  backend "gcs" {
    bucket = "hatch-tfstate"      # the one shared bucket
    prefix = "apps/prod"          # the only line that differs per stack
  }
}

Point the network stack at prod/network and the data stack at data/prod and they share storage but never share a state object. The bucket is the unit of security; the prefix is the unit of isolation.

Native GCS Locking

The GCS backend acquires a lock object alongside the state object before it writes. A second apply against the same prefix sees that lock and waits or fails fast, rather than racing the first one and corrupting the state. This is built into the backend — there is no separate lock table to provision. If you came from AWS, this is the sharp contrast: the S3 backend needs a DynamoDB table standing beside it to hold locks, and GCS folds that responsibility into the bucket itself.

The practical payoff is one less thing to bootstrap and one less thing to keep in sync. You do not create, secure, or pay for a lock table; the moment the bucket exists, locking works for every prefix in it.

Bucket IAM, Not Object ACLs

Turn on uniform bucket-level access and govern the whole bucket with one IAM policy: roles/storage.objectAdmin granted only to the CI service account and the org-admin group, and to no one else. Developers get no write path to state at all. The only way prod state changes is through the pipeline, which is exactly the property you want — a developer cannot run a local apply against prod even by accident, because they cannot write the resulting state.

Uniform bucket-level access matters because the alternative, legacy per-object ACLs, drifts. Every state object would carry its own access list, those lists diverge over time, and you lose the single policy that should govern all state. One bucket, one policy, no per-object exceptions.

Versioning as the Recovery Mechanism

Object versioning keeps every prior state object, which makes it the team's undo button. A bad apply that mangles state, or an accidental terraform state rm that drops a resource, is recoverable by promoting the previous version of the state object back into place. With versioning off, a botched state rm is permanent: the previous state is gone and you rebuild it by hand against live resources, mapping each real object back into a fresh state file.

Keep versioning on permanently, and pair it with a lifecycle rule that retains noncurrent versions for a bounded window — 90 days is a reasonable recovery horizon — so the bucket does not grow without limit while still giving you a real window to recover. Versioning is why the state bucket is never lifecycle-deleted aggressively.

Locking Down the Bucket Itself

The bucket holds every team's state, so destroying it is catastrophic. Set force_destroy = false so a terraform destroy cannot wipe a bucket full of state objects, adopt a deletion-protection posture, and let the noncurrent-version lifecycle rule keep recovery possible without unbounded growth. These settings make the bucket boring and durable, which is the goal.

There is a bootstrap wrinkle: the hatch-tfstate bucket cannot store the state that creates it. So it lives in its own hatch-admin project and is managed by a small bootstrap config that uses local state — a chicken-and-egg you resolve by letting one tiny config keep its state on disk while every other stack stores state in the bucket that bootstrap config creates.