Notification Channels

A receiver is where a routed alert becomes a message a human sees: a named entry in Alertmanager's config holding one or more integrations — Slack, email, a generic webhook, or an on-call platform like PagerDuty or Opsgenie. The routing tree from the last topic ends at a receiver name; this topic is what happens after that name resolves.

The split of responsibilities matters more than any single integration. Alertmanager decides which receiver gets the alert; the on-call platform decides which human is on duty and what happens when that human does not answer. Blur the two and you end up rebuilding escalation with grouping timers, which is the one thing they cannot do.

Receivers and the Built-in Integrations

A receiver bundles integration blocks: slack_configs, email_configs, webhook_configs, pagerduty_configs, opsgenie_configs, and friends. One receiver can fan out to several at once. Harborline's page-bookings receiver hits PagerDuty and posts to Slack, so the phone makes noise and the incident has a visible thread from minute one — the responder wakes to a channel where the summary and links are already waiting.

# alertmanager.yml — one receiver, two integrations
receivers:
  - name: page-bookings
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/pd_routing_key
        send_resolved: true
    slack_configs:
      - api_url_file: /etc/alertmanager/slack_webhook
        channel: "#harborline-alerts"

Two details in that receiver carry weight. The credentials arrive through the _file variants — routing_key_file, api_url_file — so the secrets live on obs-01's filesystem, outside version control. And send_resolved: true means the same pipe that carried the page carries the all-clear, so PagerDuty closes the incident by itself when the alert stops firing.

The Universal Escape Hatch

webhook_configs POSTs the alert group as JSON to any URL you name. That is the whole feature, and it is how Alertmanager reaches everything it has no native integration for: the ticket system that receives Harborline's severity: ticket alerts, chat platforms outside the built-in list, home-grown automation that restarts a stuck worker. If a system can accept an HTTP POST, it can be a receiver.

The On-Call Platforms

PagerDuty and Opsgenie — with Grafana Cloud IRM and Splunk On-Call in the same family — own the on-call schedule, acknowledgment, and escalation. Alertmanager delivers an event with a routing key; the platform turns it into an incident and pushes to whoever the schedule says carries the pager this week. At Harborline that is Mara, every week, until the rotation grows — but the schedule abstraction is already earning its keep, because the alert config never has to name a person.

Escalation in the On-Call Tool

Alertmanager has no concept of "nobody acknowledged in 5 minutes". It delivers notifications; it does not track whether a human responded. The escalation policy — page Mara, then the secondary after 5 minutes, then the team lead — is configured in PagerDuty, which knows about acknowledgment because acknowledgment happens in its app. Imitating escalation with a short repeat_interval just re-sends the same page to the same sleeping person, and no second responder is ever engaged.

ReceiverAlertmanager

→

On-call platformschedule + ack

→

Escalationprimary → secondary → lead

→

Phoneon-call engineer

Severity to Channel

The severity-to-channel mapping is what makes Topic 44's sorting mechanical. severity: page routes to the PagerDuty receiver — a phone makes noise at any hour. severity: ticket routes to the webhook receiver that files a ticket reviewed each working morning. Everything informational lands in #harborline-alerts, a channel nobody is required to read. The destinations differ in exactly one dimension: how hard they interrupt.

With send_resolved: true on every integration, resolution flows down the same mapping — the ticket closes, the Slack thread gets its green check, the PagerDuty incident auto-resolves. An open incident then always means an open problem, which is a property worth more than it sounds the first time the incident list is consulted during an outage.