Topic 41

The OTel Collector

Collector

The OpenTelemetry Collector is a standalone process that receives telemetry, works on it, and forwards it — a programmable junction box between instrumented services and backends. Its configuration is three vocabularies wired into pipelines: receivers take data in, processors transform it in order, exporters send it out.

One Collector config carries Harborline's traces, metrics, and logs through the same binary. That makes the Collector, not any backend, the narrow waist of the telemetry pipeline: Topic 40 made instrumentation portable, and this process makes the operational half portable too — buffering, enrichment, sampling policy, backend credentials, all in one place that isn't application code.

Receivers, Processors, Exporters

The config groups components into named blocks — this chapter's pipeline uses three: receivers:, processors:, and exporters: declare components — and declaring one does nothing by itself. service.pipelines is where composition happens: each pipeline names its signal (traces, metrics, or logs) and lists which declared components it uses, in processing order. Harborline's gateway declares an OTLP receiver on both standard ports, a memory limiter capped at 80% of the process ceiling, and a batcher.

receivers:
  otlp:
    protocols:
      grpc:   # listens on 4317
      http:   # listens on 4318
processors:
  memory_limiter:
    limit_percentage: 80
  batch:

Declaration and wiring being separate is what makes components reusable: the same otlp receiver and the same batch processor appear in the traces pipeline, the metrics pipeline, and the logs pipeline without being defined three times. It is also the config's classic trap — a component declared but never listed in a pipeline is silently inert, which the Mistakes box below returns to.

One pipeline — receivers in, processors in order, exporters out

Receiversotlp 4317 / 4318

→

memory_limiterprotection first

→

batchpacking last

→

Exportersotlp → Tempo

The Ingest and Egress Edges

On the way in, the otlp receiver — gRPC on 4317, HTTP/protobuf on 4318 — is often the only receiver a fleet needs, because everything instrumented in Topic 40 already speaks OTLP. The longer tail exists for swallowing legacy sources: a Prometheus-scrape receiver, a filelog receiver, Jaeger and Zipkin receivers for pre-OTel instrumentation. On the way out, an otlp exporter carries traces to Tempo, prometheusremotewrite pushes metrics into Prometheus, and an OTLP/HTTP push covers Loki — three signals, three egress components, one process.

The Two Processors That Are Never Optional

memory_limiter goes first in every pipeline. When the process approaches its memory ceiling, it starts refusing incoming data, and the refusal lands on the sender's retry logic — the agent or SDK buffers and tries again. Without it, a traffic spike or a slow backend queues telemetry in the Collector's memory until the kernel OOM-kills the process, and all three signals go dark at once. Refusing a batch is recoverable; dying is not.

batch goes last before export, packing telemetry into fewer, larger requests. At Harborline's span volumes the difference is thousands of network calls per second versus dozens — export bandwidth, gateway CPU, and the backend's rate limiter all feel it. Every real pipeline runs both: protection first, packing last, any transforms in between.

Agent vs Gateway

The Collector deploys in two shapes, and they compose rather than compete. An agent runs on every host: it receives from local processes over loopback, enriches telemetry with host metadata, and buffers through backend restarts so a 30-second Tempo outage costs nothing. A gateway is a central fan-in tier that owns the fleet-wide concerns — sampling policy in Topic 42, backend endpoints and credentials — so those live in exactly one config instead of one per host.

Harborline already runs Grafana Alloy on every host as its agent. Alloy is a distribution built on Collector components — the same receiver-processor-exporter concepts, a different configuration syntax — and this chapter teaches the vanilla Collector precisely because the concepts transfer. Each Alloy forwards OTLP to one gateway Collector on obs-01, and that gateway is where the rest of this chapter happens.

One Config, Three Signals

The gateway's service.pipelines block draws the payoff concretely: the traces pipeline and the metrics pipeline reuse the same otlp receiver and the same two mandatory processors, then diverge at the exporter — traces to Tempo, metrics to Prometheus remote-write. Each list reads left to right in processing order, protection first and batching last.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

This junction is where fleet-wide mutation belongs. Scrubbing a card-number attribute before it reaches storage, renaming an attribute, stamping env="prod" onto everything — done here, each is a config change and a reload. Done in application code, each is five edits and five service deploys, and the sixth service added next year ships without them.

Core vs Contrib

Two official distributions exist. Core is the minimal component set; contrib adds the long tail of receivers, processors, and exporters accumulated by the community. The dividing line matters immediately: tail_sampling, which Topic 42 is built around, ships only in contrib — so Harborline runs the contrib image on the gateway and never thinks about it again.

Common Mistakes

Running a pipeline without memory_limiter — a traffic spike or a slow backend queues data in memory until the kernel OOM-kills the Collector, taking all three signals down at once instead of shedding load gracefully.
Omitting the batch processor — every span becomes its own export request, the gateway burns CPU on protocol overhead, and the backend rate-limits the flood.
Pointing every service directly at the gateway with no per-host agent — one gateway restart drops in-flight telemetry from all five services, and host-metadata enrichment has nowhere local to happen.
Declaring a component but forgetting to list it in service.pipelines — the config validates, the Collector starts, and the exporter silently never runs; the fix is checking the pipeline wiring, not the component block.
Pulling the core image and configuring tail_sampling — the component doesn't exist there; the error names an unknown processor, and the fix is the contrib distribution.

Best Practices

Order every pipeline memory_limiter first, batch last, with transforms between — the list is the processing order, and protection must precede work.
Deploy agent-per-host plus a central gateway once you pass a handful of hosts — local buffering and enrichment at the edge, sampling policy and backend credentials in exactly one place.
Do all fleet-wide telemetry mutation in the Collector, not in application code — redaction, attribute renames, and environment stamping change with a config reload instead of five service deploys.
Watch the Collector's own telemetry — it exposes its own metrics, and otelcol_processor_refused_spans (the memory limiter pushing back) plus the exporter queue and failure counters are the early warning that the pipeline, not the app, is dropping data.

Comparable toolsGrafana Alloy — a distribution built on Collector components, its own config languageLogs lineage Fluent Bit — began as a log shipper, now speaks OTLP for all three signalsDatadog Vector — a comparable high-performance pipelineProprietary Datadog Agent / Elastic Agent — the platform-coupled equivalents

Knowledge Check

A pipeline is configured processors: [batch, memory_limiter]. Why is this wrong?

The Collector fails validation, since that order is illegal
The list is the processing order, so protection runs after the work it should gate
batch can only pack data that memory_limiter has already compressed and deduplicated
Processor lists must be alphabetical

When memory_limiter refuses incoming spans, what actually prevents data loss?

The backend pulls the refused spans later, once the Collector recovers
The batch processor holds refused spans until memory frees up
The sender's retry logic, which buffers and re-sends the refused batch
The kernel pages the Collector's memory to swap

Where should scrubbing a card-number attribute out of Harborline's telemetry happen, and why?

In each service's SDK configuration, closest to the source before export
In Tempo, filtered out at query time
In the gateway Collector's pipeline, once for the whole fleet
In a Grafana panel transformation

The Collector starts cleanly, but spans never reach Tempo and no errors appear. Which config mistake fits these symptoms?

tail_sampling configured on the core image
The Tempo exporter is declared but not listed in service.pipelines
memory_limiter is missing from the traces pipeline
The OTLP receiver is on 4318 instead of 4317

You got correct