What to Alert On

Network monitoring fails in two directions, and both end the same way: nobody trusts the pager. Alert on everything and the noise trains people to ignore alerts, so the one that matters scrolls past unread. Alert on nothing useful and an outage runs silent until a customer reports it. The fix is not more alerts or fewer — it is choosing the few signals that track what users actually feel, and routing only those to a human at 3 a.m.

This topic applies SLI thinking to networks. Instead of paging on raw counters — CPU at 80%, an interface at 7 Gbps — you page on symptoms: latency users experience, packets lost, requests erroring, links running out of headroom. The counters still matter, but as the things you consult to explain a symptom, not as the things that wake you. Get that split right and the alert volume drops by an order of magnitude while the coverage of real incidents goes up.

The Four Golden Signals for Networks

The SRE golden signals translate cleanly to networking. Latency is round-trip time and how it changes — the first thing users feel. Traffic is the demand on the system: requests per second, bits per second, new connections per second. Errors are the rate of things going wrong: dropped packets, interface CRC errors, failed connections, TCP resets. Saturation is how full the system is relative to capacity — the signal that predicts the other three are about to get worse.

These four cover most of what can go wrong on a network with a small, stable set of metrics. The trick is that they are rates and ratios, not absolute counters: error rate per second, not lifetime error count; utilization percentage, not raw byte total. A counter that only ever climbs tells you nothing about right now; the derived rate is what a threshold can sensibly fire on.

Symptom versus Cause Alerting

A symptom is something a user can feel: requests are slow, connections fail, a page errors. A cause is a mechanism that might produce a symptom: CPU is high, a queue is deep, a link is busy. The rule is to page on symptoms and dashboard the causes. CPU at 80% is not an incident if latency and error rate are fine — it is headroom being used as intended. Paging on it generates a 3 a.m. wake-up for a system that is working.

The reason symptom alerting wins is coverage. There are a hundred ways to make a service slow, and you cannot enumerate and threshold all of them; but they all converge on the same handful of symptoms — latency, errors, loss. Alert on the symptom and you catch causes you never anticipated. Then, when the symptom fires, the cause dashboards are right there to tell you which of the hundred mechanisms it was this time.

Saturation and Headroom

Saturation is the signal that buys you time, because it leads the others — a link at 95% is still passing traffic, but the queue is filling and latency is about to spike. The catch is that the dangerous saturations are rarely raw bandwidth. The conntrack table filling on a firewall, the ephemeral-port pool exhausting on a NAT gateway, or a switch's output queue going deep will drop traffic while the interface graph still reads a comfortable 40%.

These table- and pool-fill metrics are the ones teams forget to instrument until an incident teaches them to — connection-tracking exhaustion drops new flows with the link half-idle, and the throughput graph offers no clue. Alert on percentage-of-capacity for every bounded resource, not just the obvious one: link utilization, conntrack entries against the table limit, ephemeral ports in use, queue depth. Headroom is the early warning the bandwidth graph cannot give you.

Thresholds, Percentiles, and Error Budgets

Averages lie about latency. If the mean response time is 40 ms but the p99 is 900 ms, one request in a hundred is painfully slow — and at a thousand requests per second that is ten suffering users every second, completely invisible in the average. Alert on the high percentiles (p95, p99) because they are where users actually live; the average is dragged down by the fast majority and hides the tail that generates complaints.

Thresholds need hysteresis or they flap. A bare "fire if latency > 200 ms" alert will trigger and clear repeatedly on every transient spike, and a pager that cries wolf gets muted. Require the condition to hold for a duration (latency above 200 ms for 5 minutes) and frame budgets over a window — an error budget of, say, 0.1% of requests per month — so a brief blip spends budget instead of paging, and only sustained burn wakes someone. The goal is one page per real problem, not one per spike.