Topic 09

USE and RED

Method

node_exporter alone exposes close to a thousand series per host. Five services and six hosts put thousands of numbers on the table before Mara has instrumented a single line of code, and every one of them looks plausible on a graph. USE and RED are the two checklists that decide which of them matter: USE interrogates every resource, RED interrogates every request-driven service.

Between them, they cover the Harborline fleet with about a dozen questions instead of a thousand series. That compression is the whole point — a method turns "what should I look at" from a matter of taste into a fill-in table.

The Selection Problem

"Collect everything, look at nothing" is the default failure mode of a fresh metrics stack. Storage is cheap enough that nothing forces a choice, so nobody makes one, and six months later the dashboard folder holds 40 boards of plausible-looking noise that no one opens during an incident. A checklist with a finite number of cells is the difference between a dashboard and that folder.

USE: Utilization, Saturation, Errors

Brendan Gregg's checklist, applied per resource: for every resource, record how busy it is (utilization), how much work is queued because it cannot keep up (saturation), and its error count. CPU gets busy-percentage and run-queue length. Disk gets busy-percentage and I/O queue depth. Network gets bandwidth-percentage and drops.

The discipline is in the words "every resource." The checklist is only as good as the inventory it walks, and the inventory that matters includes things no host graph will ever show — the next two sections come back to exactly that.

RED: Rate, Errors, Duration

Tom Wilkie's counterpart for request-driven services: requests per second (rate), failed requests per second as a ratio of rate (errors), and the latency distribution (duration). RED measures what a customer feels — nobody has ever emailed support about run-queue length. The checkout SLO — 99.9% availability, 95% of requests under 500 ms — is nothing but RED's E and D with targets attached.

USE vs RED — resources versus request paths

USEevery resource

Utilization — how busy it is. Saturation — work queued because it cannot keep up. Errors — its error count. Walks every resource, including software ones like the Redis connection pool.

REDevery request-driven service

Rate — requests per second. Errors — failed requests as a ratio of rate. Duration — the latency distribution. Measures what a customer actually feels.

Saturation as the Early Warning

100% utilization is not inherently wrong; saturation is. A disk at 90% busy with an empty queue is earning its keep. A disk at 60% busy with a deep I/O queue is already delaying requests. Queue depth moves before latency does, which makes saturation the leading indicator that utilization only pretends to be — and the reason USE gives it a column of its own instead of folding it into "busy."

The Harborline Fill-In

USE, per resource: CPU, memory, disk, and network on app-01 through mq-01 — and, crucially, the software resources. The Redis connection pool inside bookings is a resource: utilization is connections in use divided by pool maximum, saturation is requests waiting for a connection. The gunicorn worker pool is a resource by the same arithmetic. These rows sit in the table next to CPU and disk, or the table is lying about coverage.

RED, per service: bookings, search, and payments each get a request rate, an error ratio, and a duration histogram. notifier serves no HTTP requests, so its RED is translated, not skipped: rate is jobs consumed per second from RabbitMQ, errors are failed jobs, duration is job processing time.

Which Method, and How They Meet

Resource, USE; request path, RED. Most incidents present as a RED symptom — checkout p95 rising — caused by a USE condition: a saturated pool, a full disk. So the operating rule is alert on RED, diagnose with USE. That sentence is the first appearance of the symptom-versus-cause philosophy this book builds its whole alerting chapter on, and it is worth flagging that the Saturday complaint, whatever is behind it, will present exactly this way.

Common Mistakes

Dashboarding utilization without saturation — CPU at 80% looks comfortable while the run queue holds four times the core count; the slowness lives in the queue, and the utilization panel will never show it.
Applying USE only to hardware — connection pools, worker pools, and queues are resources too, and a pool of 10 connections serving 40 concurrent checkouts is a saturated resource that no host-level graph will ever surface — a blind spot that swallows real incidents.
Forcing HTTP RED semantics onto notifier — it serves no requests, so the naive checklist leaves it unmeasured entirely; its rate is queue consumption and its duration is job processing time, and skipping that translation means the one async service is the one still flying blind.
Alerting on absolute error counts without the rate — 50 errors a minute in search at 3,000 requests a minute is a 1.7% blip; 50 errors a minute in payments at 60 requests a minute is a full outage. The count alone cannot tell those two apart.
Confusing collection with coverage — filling every USE and RED cell into Prometheus while alerting on none of it means the method chose what to store, and still nobody finds out when checkout degrades.

Best Practices

Walk the USE checklist across every host and every resource once, in an actual table, explicitly including software resources — the Redis pool and the gunicorn workers earn their rows next to CPU and disk.
Give every request-serving service its three RED series as the first instrumentation it receives, before any bespoke metric.
Alert on RED symptoms and keep USE on diagnosis dashboards — the customer feels duration and errors, never run-queue length.
Express Errors as a ratio of Rate (errors/s ÷ requests/s), so the same alert threshold means the same severity at any traffic level.

Comparable toolsGoogle SRE the Four Golden Signals — latency, traffic, errors, saturation — are RED plus a saturation borrow from USEBrendan Gregg the original USE material ships as per-OS checklists for Linux and SolarisNetdata auto-builds per-resource dashboards that amount to USE out of the boxDatadog / New Relic encode the split as product structure — infrastructure view ≈ USE, APM view ≈ RED

Knowledge Check

Redis connections in use divided by pool maximum, and requests waiting for a free connection — which USE columns are these?

Utilization and errors
Utilization and saturation
Saturation and errors
Rate and duration

Why can saturation warn before latency while utilization sits at 100% harmlessly?

Saturation metrics are scraped more frequently than utilization metrics
Utilization pinned at 100% always indicates a real outage that latency confirms later
A busy resource with no queue delays nothing; a growing queue is already delay
Saturation is a statistical forecast computed from utilization trends

How does RED translate onto notifier, which serves no HTTP requests?

It does not apply — queue workers are covered by USE alone
Rate becomes jobs consumed per second, errors failed jobs, duration processing time
Measure mq-01's CPU and memory as a proxy for notifier's health
Add a synthetic HTTP endpoint to notifier so the standard RED method applies unchanged

Why alert on RED and diagnose with USE, rather than alerting on both?

USE metrics are simply too expensive to evaluate continuously inside alert rules
Resource conditions never matter to reliability
RED metrics are statistically cleaner than USE metrics
Customers feel RED symptoms; a USE condition that hurts nobody should not page

You got correct