Instrumenting an Existing System

Almost nobody instruments a greenfield system. You inherit six services, two databases, and zero telemetry, and the question is where the first hour of work buys the most visibility. The brownfield playbook is topic 71's maturity path compressed into a deployment order: start at the edge, where one config change covers every request, work inward through exporters and logs, and only touch application code where the money is.

This is Harborline's own Chapter 3 to Chapter 10 arc, retold as the general recipe. Nothing in it requires permission from six teams or a quarter of sprint capacity — the first four stages are config changes and sidecar containers, and by the time code changes come up, the stack has already shortened an incident or two and argues for itself.

Edge + probeszero code

→

Store exportersone container each

→

Structured logsconfig switch

→

Code metricsone service first

→

Hottest-path tracingone path

The Edge First

The load balancer or ingress already sees every request in the system. nginx's stub_status gives instant global request rate, and its JSON access logs — or HAProxy's built-in Prometheus endpoint, already broken out per backend — add per-route errors and duration for everything behind them, with zero application changes. Harborline's web nginx was emitting the first request-rate data on day one of Chapter 5's edge pass, before anyone had opened a single service's source. One config change, 100% request coverage: no later stage matches that ratio.

Blackbox Probes for Instant Coverage

blackbox_exporter probing each service's health endpoint every 15 s answers "is it up, and how slow is the front door" for the four HTTP services from one config file. Probes are the outside-in view, and they hold a property no in-process metric has: they keep reporting when the service is down hard enough to stop reporting about itself. Roughly 10 lines of config, and whole-service failure is visible before any code is instrumented.

Exporters for the Stores

postgres_exporter on db-01, redis_exporter on cache-01, RabbitMQ's Prometheus plugin on mq-01. Brownfield incidents concentrate in the backing stores — connection limits, replication lag, queue depth — and each exporter is one container plus one scrape job, no application changes. The exporter catalog encodes years of which-metrics-matter decisions; writing a custom exporter for a store that has an official one is pure waste.

Structured Logs Next

Switch the application loggers to JSON and point Grafana Alloy at the container logs — Chapter 7's pipeline. For most frameworks this is a logging-config change, not a code rewrite, and it converts the log volume the system already produces from grep-fodder into queryable events with service and level labels. The bespoke log lines that resist the config switch can migrate opportunistically; the config-level change captures 80% of the value now.

Code-Level Metrics Where the Money Is

RED instrumentation from Chapter 5 goes into one service first: the one whose failure costs revenue or wakes someone. For Harborline that was bookings, not search — checkout is money, and search has a cache in front of it. Resist instrumenting alphabetically, and resist instrumenting uniformly: notifier needs a queue-depth gauge and a failure counter, not the full histogram treatment bookings gets. Depth follows risk, not fairness.

Tracing the Hottest Path Only

OpenTelemetry goes along the single most valuable request path — for Harborline, web → bookings → payments in Chapter 8 — not across all services at once. Context propagation through three services is a bounded project with a demo at the end: one complete waterfall in Tempo that answers a real question. A half-propagated rollout produces orphaned spans that look like broken instrumentation, and the org's first impression of tracing becomes "it doesn't work." The remaining paths join after the first one proves the pattern.