Alertmanager
Prometheus decides that something is wrong; Alertmanager — on obs-01:9093 — decides who hears about it, how often, and bundled with what. Every firing alert is pushed to it over HTTP, and between a firing rule and Mara's phone stand four mechanisms: the routing tree picks a receiver from the alert's labels, grouping batches related alerts into one notification, silences mute what is already known, and inhibition suppresses the downstream noise of an upstream failure.
Each mechanism kills a specific species of noise, and the recurring failure in real configs is aiming the wrong mechanism at the wrong species — stretching a grouping timer to fix a flapping rule, or hand-editing a route to survive a maintenance window. This topic walks the four in the order an alert meets them.
The Division of Labor
Rules, expressions, and thresholds live in Prometheus; deduplication, grouping, routing, and delivery live in Alertmanager. Several Prometheus servers can feed one Alertmanager, which dedupes identical alerts — the design that later lets Harborline run a second Prometheus without doubling its pages. The split is also organizational: "the error threshold is too tight" is an edit to a rule file, "the wrong team got paged" is an edit to alertmanager.yml, and the two changes never collide in review.
The Routing Tree
The config declares one root route with nested child routes, each carrying label matchers like severity="page" or team="bookings". An arriving alert descends the tree depth-first and stops at the first route whose matchers all hold, unless that route sets continue: true — in which case matching keeps going and the alert can land on several receivers. Anything that matches no child is handled by the root's own receiver, which makes the root the catch-all of last resort.
# alertmanager.yml — the routing tree
route:
receiver: slack-catchall
group_by: [alertname, job]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: [severity="page", team="bookings"]
receiver: page-bookings
- {matchers: [severity="ticket"], receiver: ticket-webhook}
Harborline's tree is three lines of intent: pages for the bookings team go to the page-bookings receiver, anything marked severity: ticket goes to the webhook that files a ticket, and everything else falls through to the slack-catchall receiver on the root. Order matters — child routes are checked top to bottom and the first match wins, so the specific route must sit above any broader one that would swallow its traffic.
Grouping
group_by collapses alerts that share the listed labels into one notification. With group_by: [alertname, job], nine instances of the same alert against the same service arrive as one message listing nine firing entries — one conversation, not nine parallel ones. This is the mechanism that turns "a dead database fired 9 alerts" from a pager barrage into a single readable incident summary.
Three timers govern the batch. group_wait (default 30s) holds the first notification of a new group so late-arriving siblings can join it. group_interval (default 5m) is the minimum wait before a group whose membership changed notifies again. repeat_interval (default 4h) re-sends an unchanged, still-firing group, so an alert that fired during a shift handover cannot simply be forgotten. All three trade freshness against volume, and the defaults are sane enough that Harborline ships them untouched.
Silences
A silence is a set of label matchers plus an expiry, created in the web UI or with amtool silence add. Matching alerts still evaluate and still show as firing — they just stop notifying. The built-in expiry is the point: a silence covering Tuesday night's 2-hour PostgreSQL maintenance on db-01 removes itself when the window closes, where a config edit would need a human to remember the revert. Silences are also visible in the UI with an author and a comment, so the next responder can see what is muted and why.
Inhibition
inhibit_rules suppress target alerts while a source alert fires. When HarborlineBookingsDown fires at severity: page, the checkout latency and error-ratio alerts for the same job add nothing — the service being down already explains them — so an inhibit rule mutes them while the source fires. The equal: [job] clause is the scope guard: suppression applies only to alerts sharing those label values, so a dead bookings service cannot mute an unrelated payments page.
- Placing a broad route above a specific one — a
severity="page"route listed before theteam="bookings"route swallows every page, the bookings receiver never sees traffic, and matchers stop working "mysteriously". Child routes are checked in order; first match wins. - Stretching
group_waitto 10 minutes to "stop flapping" — flap suppression isfor:in the Prometheus rule; here the only effect is that every page, including the real outage, arrives 10 minutes late. - Writing an inhibit rule without
equal:— the moment any source alert fires anywhere, every matching target in the system is suppressed, and aweboutage silently mutes thepaymentspages. - Silencing on a broad matcher like
severity="page"during an incident — every page in the company is now muted, and a second, unrelated outage during the window notifies no one. - Setting
repeat_interval: 5m"so nobody misses it" — an incident that takes 40 minutes to fix delivers 8 copies of the same page, which is spam wearing an alert's clothes.
- Test the tree with
amtool config routes test severity=page team=bookingsbefore deploying a routing change, so you see which receiver wins instead of discovering it at night. - Keep the root receiver pointed at a low-noise catch-all — Harborline's
#harborline-alertsSlack channel — so an alert with unrouted labels degrades to visible, never to lost. - Group by
alertnameandjobas the baseline, so one incident produces one conversation-sized notification rather than a per-instance barrage. - Use expiring silences for every planned maintenance window and never hand-edit routes as a temporary mute — silences are visible in the UI, auditable, and self-removing.
Knowledge Check
An alert fires and resolves every few minutes, sending a notification each round. Which knob fixes it?
- Stretch group_wait so the flaps get absorbed into one batch and stop generating separate notifications
- The for: duration on the Prometheus rule — flap suppression lives in the rule, not in Alertmanager
- Raise repeat_interval so the copies come less often
- Add a silence matching the alert until it settles down
What does continue: true change about an alert's descent through the routing tree?
- It sends a copy of the alert to the root receiver as a backup
- It switches traversal from depth-first to breadth-first
- Matching does not stop at that route — the alert can also match later siblings and land on multiple receivers
- It makes the route re-notify continuously until the alert resolves
Which timer holds the first notification of a brand-new group, and what is its default?
- group_interval, default 5m
- repeat_interval, default 4h
- group_wait, default 30s
- evaluation_interval, default 1m
Why does an inhibit rule need an equal: clause?
- It scopes suppression to alerts sharing the listed label values, instead of muting matching targets everywhere
- Without it the configuration fails validation and Alertmanager refuses to start until the equal: block is added back
- It is what deduplicates identical alerts from two Prometheus servers
- It sets how long the suppression lasts before expiring
You got correct