Instrumenting What You Can't Change

Half of Harborline will never call prometheus_client. Postgres, Redis, and nginx are other people's code — the bookings team can configure them but not edit them, and a client library is useless without somewhere to put it. The ecosystem's answer is the exporter: a sidecar process that speaks the target's native protocol, asks it what it knows, and republishes the answers as a /metrics endpoint.

Exporters cover most of the gap, not all of it. This topic adds the two fallbacks for when even an exporter falls short — probing from outside, and mining logs — and closes with the decision ladder that keeps a team from picking the wrong rung. By the end, every Harborline component has a rung and a port.

The Exporter Pattern

postgres_exporter on :9187 logs into db-01 and turns the pg_stat_* views into metrics. redis_exporter on :9121 runs INFO against cache-01. nginx-prometheus-exporter on :9113 reads web's stub_status page. Each becomes one more scrape job in obs-01's Prometheus, indistinguishable from a natively instrumented service once the samples land.

The nginx case shows the fidelity ceiling. stub_status is seven numbers — active connections, accepts, handled, total requests, and three connection states — so per-route latency from open-source nginx is not on this rung at any price. An exporter can only republish what its target chooses to expose; it translates, it does not add.

The Native Exception

RabbitMQ needs no sidecar at all: it has shipped its own rabbitmq_prometheus plugin since 3.8, serving :15692, so mq-01 is covered by a single rabbitmq-plugins enable rabbitmq_prometheus. The standing rule is to check for built-in Prometheus support before shopping for an exporter — more infrastructure grows it every year, and a first-party endpoint beats a third-party translation of the same numbers.

Blackbox Probing

blackbox_exporter on :9115 tests from outside: an HTTP probe against https://harborline.example/ asserting a 200 and a TLS certificate that is not near expiry, a TCP probe against db-01:5432. It sees zero internal state and exactly what a customer sees — which makes it the one rung that catches "every service reports healthy, site unreachable."

# prometheus.yml — probe the public site via blackbox_exporter
- job_name: "blackbox-http"
  metrics_path: /probe
  params: {module: [http_2xx]}
  static_configs: [{targets: ["https://harborline.example/"]}]
  relabel_configs:
    - {source_labels: [__address__], target_label: __param_target}
    - {source_labels: [__param_target], target_label: instance}
    - {target_label: __address__, replacement: "obs-01:9115"}

The relabeling dance is mandatory. The listed target must not be scraped directly — the exporter must be scraped about it — so the config moves the URL into the target parameter, copies it into the instance label so graphs name the site rather than the prober, and points the actual scrape address at obs-01:9115. Skip any step and the job quietly probes nothing while reporting success.

Logs into Metrics, the Last Rung

When a system exposes nothing and accepts no probe worth having, its log lines can be counted: Loki metric queries promoted into recorded metrics through the ruler, or mtail and grok_exporter tailing files directly. It works, and it is coupled to log format — one upstream release that rewords a line silently zeroes the metric while everything else reports healthy. This rung ships with a ticket to climb off it.

The Decision Ladder

Native instrumentation, then official exporter, then blackbox probe, then log parsing — in that order. Each step down trades interior visibility for exterior observation, and a stable contract for fragile scraping: a client library reads the program's own state, an exporter reads what the target admits, a probe reads only the surface, and a log parser reads whatever a formatting decision left behind.

Placing Harborline on the ladder takes one paragraph. The Python services — bookings, search, payments, notifier — sit on rung one with prometheus_client. Postgres, Redis, and nginx sit on rung two behind their exporters, while mq-01 joins rung one courtesy of its built-in plugin. The public checkout URL gets a rung-three blackbox probe, and nothing, yet, lives on rung four — a state worth defending.

Rung 1 · It is your code and can call a client library — bookings, search, mq-01's plugin→Native instrumentation

Rung 2 · You cannot edit it, but it exposes native state — pg_stat_*, INFO, stub_status→Official exporter

Rung 3 · You need only what a customer sees — reachability, TLS expiry→Blackbox probe

Rung 4 · It exposes nothing and accepts no probe worth having — only log lines remain→Log parsing