for_each

for_each stamps out one resource instance per entry in a map or a set of strings, addressing each by its key rather than its position: google_storage_bucket.raw["billing"], not [1]. Inside the block you read each.key and each.value, and because the key is stable, adding or removing one entry touches only that entry. That single property is why for_each is the default choice for almost every collection of resources.

This is the construct that lets the Hatch pipeline fan out cleanly. One raw bucket per source system, one Pub/Sub topic per event type, one service account per consumer — each is a for_each over a map, keyed on a human-meaningful name. The keying is not cosmetic: it is what makes a later edit to the collection a surgical one-line diff instead of the destructive renumber that count produces.

Map and Set Inputs

for_each accepts two input types: a map(...) mapping each key to a config object, or a set(string) of bare names. It deliberately does not accept a list, because a list has an order and for_each is order-free by design — feed it a list and Terraform errors. To iterate a list you wrap it in toset() first, which discards the ordering on purpose and gives you a set of unique members.

resource "google_pubsub_topic" "events" {
  for_each = toset(["orders", "clicks", "billing"])
  name     = "hatch-events-${each.key}"   # each.value == each.key for a set
}

each.key and each.value

Inside the block, each.key is the map key (or the set member), and each.value is the map value (or, for a set, identical to the key). You build each instance's name, region, and labels from these two. For a set, the only thing available is the string itself, exposed as both each.key and each.value; for a map of objects, each.value is the whole object and you reach into its fields.

variable "sources" {
  type = map(object({
    location = string
    labels   = map(string)
  }))
}

resource "google_storage_bucket" "raw" {
  for_each = var.sources
  name     = "hatch-events-raw-${each.key}"   # key is the source name
  location = each.value.location           # value carries the settings
  labels   = each.value.labels
}

Stable Keys Survive Edits

Instances are keyed by the map key, so the address Terraform tracks in state is google_storage_bucket.raw["billing"] — tied to the name, not a position. Insert "events-billing" into the map and Terraform creates exactly that one bucket; remove "events-legacy" and it destroys exactly that one. Every other bucket, topic, and dataset is untouched, because none of their keys moved. This is the precise opposite of count, where removing a middle element renumbers and churns the tail.

The payoff is that the Hatch source list can grow and shrink over the pipeline's life with no collateral damage. Onboarding a new source system is adding one map entry; retiring one is deleting an entry. Neither operation ever proposes destroying a bucket you did not name.

The Canonical Pattern

A for_each over a map of objects is the workhorse of real configurations. It is how Hatch creates one Pub/Sub topic per event type, one BigQuery dataset per tenant, or a map of service accounts in a single block — each entry's per-instance settings carried in the value object. One block, one collection variable, and the entire fan-out lives in the map rather than in copy-pasted resource blocks.

resource "google_service_account" "consumer" {
  for_each     = var.consumers
  account_id   = "hatch-${each.key}"
  display_name = each.value.display_name
}

Constraints

The keys — and set members — must be known at plan time, not derived from a computed attribute. for_each over a value that is only knowable after apply (a generated id, another resource's output) errors, because Terraform must expand the instances during plan to show you what it will do. This is the same plan-time rule that governs count: the structure of the collection is fixed before anything is created. The values inside each entry can be computed; the keys cannot.