Client Libraries

Everything Prometheus has scraped so far watched Harborline from the outside: node_exporter reading /proc on the hosts, an exporter logging into a database and republishing what it saw. A client library moves the measurement inside the process. prometheus_client keeps metric state in the application's own memory, an increment costs well under a microsecond, and the /metrics endpoint renders the current state only when Prometheus asks. The application never pushes anything — it keeps score and answers scrapes.

This is the week Mara pairs with the bookings team. Chapters 3 and 4 could say what app-01 was doing on a Saturday morning; nothing could say what checkout was doing. Three instrument types, one mounted endpoint, and one deployment trap stand between the bookings service and its first service-level metrics.

Three Instruments, Three Questions

A Counter only goes up and answers "how many since the process started." The monotonic-only rule is load-bearing: it is what lets Chapter 4's rate() treat any decrease as a process restart instead of data loss. A Gauge moves both ways and answers "how many right now" — in-flight requests, Redis pool connections checked out. A Histogram answers "how long, in what distribution" by sorting every observation into cumulative buckets.

from prometheus_client import Counter, Gauge

BOOKINGS = Counter("harborline_bookings_total",
                   "Confirmed bookings")
REDIS_IN_USE = Gauge("harborline_redis_pool_in_use",
                     "Redis connections checked out")

The snippet declares a Counter and a Gauge at module scope. harborline_bookings_total gets BOOKINGS.inc() once per confirmed booking and nowhere else — it counts a business event, not HTTP traffic. harborline_redis_pool_in_use is a level: inc() when a connection is checked out, dec() when it returns. A Counter would be the wrong instrument there, because the value falls as often as it rises.

The Canonical Histogram

The metric this book has been building toward is checkout latency. The library ships 15 default buckets spread from 5 ms to 10 s for a generic RPC; checkout needs precision between 50 ms and 2.5 s, because that is where the threshold Chapter 10 turns into an SLO — 95% of checkouts under 500 ms — will live. Bucket edges are where all of Chapter 4's histogram_quantile() accuracy comes from: the function interpolates inside a bucket, so a quantile is only as sharp as the edges around it.

from prometheus_client import Histogram

CHECKOUT_LATENCY = Histogram(
    "harborline_checkout_duration_seconds",
    "Checkout request duration in seconds",
    buckets=[.05, .1, .25, .5, .75, 1, 1.5, 2, 2.5])

# in the request path:
CHECKOUT_LATENCY.observe(elapsed_seconds)

Declared once at module scope, observed on every request. Nine explicit edges plus the implicit +Inf give ten buckets clustered around the region that matters, with 0.25, 0.5, and 0.75 bracketing the future SLO threshold. harborline_checkout_duration_seconds is born here — Chapters 6, 8, and 10 all build on it.

The /metrics Endpoint

Exposition is one line: app.mount("/metrics", make_asgi_app()) on the FastAPI app. Prometheus scrapes it with job="bookings" every 15 s, exactly like any exporter, and rendering the page is a read of in-memory state — cheap enough that the scrape never registers in the service's own latency. Resist hand-writing the endpoint: make_asgi_app() handles content negotiation, which topic 25 depends on when the scraper starts asking for OpenMetrics.

The gunicorn Problem

Deploy that code behind gunicorn and the numbers go quietly wrong. gunicorn preforks N workers; each worker is a separate process with its own private registry, and every scrape lands on whichever worker accepted that particular connection. With four workers, one scrape sees roughly a quarter of the bookings, and the next scrape sees a different quarter.

The symptom is counters that appear to undercount by roughly 1/N and jump backward between scrapes. rate() reads each backward jump as a counter reset, so the graphs render plausible-looking garbage instead of an error. This is the first live demo Mara runs for the bookings team, because everyone deploys it wrong once — better in a demo than in an incident review.

Multiprocess Mode

The fix ships with the library. Set the PROMETHEUS_MULTIPROC_DIR environment variable and each worker writes its metric state to mmap'd files in that directory; at scrape time, a MultiProcessCollector on a fresh registry merges them into one coherent view. Two pieces of wiring are mandatory: the directory must be wiped at every application start, and gunicorn's child_exit hook must call mark_process_dead() so recycled workers stop haunting the numbers.

# gunicorn.conf.py — required for multiprocess mode
from prometheus_client import multiprocess

def child_exit(server, worker):
    multiprocess.mark_process_dead(worker.pid)

# container entrypoint, before gunicorn starts:
#   rm -rf "$PROMETHEUS_MULTIPROC_DIR"/*

The price list is real. Gauges need an explicit multiprocess_mode (sum, max, liveall, among others) because "current value" is ambiguous across four processes. Info and Enum metrics, custom collectors, and the default process_* metrics are unavailable, and exemplars do not work — a constraint topic 25 has to design around. Multiprocess mode is not a flag but a mode, and half-wiring it reproduces the flapping it was meant to fix.