Continuous Profiling

Metrics told Mara that search p99 latency doubled; a trace told her which span got slow — a 180 ms leaf span inside search with no children and nothing further to blame. Neither signal can say what the CPU was doing inside that span. The stack has run out of resolution exactly one level above the code.

Continuous profiling is the fourth signal: always-on, sampled stack traces collected from production at 1–5% overhead and rendered as flamegraphs. It turns "why is this function slow" from a guess into a query, and because it runs around the clock, every answer comes with a baseline attached.

The Fourth Signal

A profile is a statistical sample of call stacks over time. The profiler interrupts each core about 100 times per second, records whatever stack it finds running, and aggregates thousands of those observations into statements like "this function held the CPU for 34% of the window." No single sample means anything; the aggregate is the measurement.

That gives profiles a question the other three signals structurally cannot answer. Metrics say whether something is wrong, traces say where in the request path, logs say what happened at that moment. A profile says which lines of code — the only signal whose unit of attribution is a function rather than a request or a service.

Continuous vs On-Demand

Attaching py-spy to a process mid-incident yields one profile of one bad moment, with nothing to compare it against. A continuous profiler stores profiles around the clock, so Mara can diff search across Tuesday's deploy and see exactly which function grew from 8% to 31% of CPU. The baseline is the actual product; the flamegraph is just the rendering.

Reading a Flamegraph

Width is the share of samples — cost — and depth is the call stack. Ignore depth on the first read: the wide, flat plateaus at the top are where the time goes, because those frames were on-CPU when the sampler fired. A narrow, deep tower is a long call chain that costs almost nothing.

A CPU flamegraph shows on-CPU work only. A span that spends 150 ms waiting on cache-01 barely samples, because a waiting thread is not running when the profiler interrupts. This is why profiles complement traces instead of replacing them: the trace already proved the span was waiting, and the fix for a wait lives in the connection pool or the query, not in the profiler.

Collection Models: SDK and eBPF

Grafana Pyroscope ingests from in-process SDKs — Python, Go, Java, .NET — that push labeled profiles from inside the application, while Parca and Grafana Alloy's eBPF profiling collector sample every process on the host from the kernel with zero code changes. The trade is depth against breadth: SDKs deliver richer language-level detail and application labels; eBPF covers the whole fleet, including binaries you cannot modify. Harborline runs the SDK on the services it owns, the same division the next topic formalizes for metrics and traces.

Profile–Trace Correlation

Span profiles close the loop between the third signal and the fourth. The SDK tags profile samples with the active span_id using pprof labels, so in Grafana, Mara clicks the slow 180 ms span in a Tempo trace and gets the flamegraph for that span's code specifically. The trace said where; the profile says why.

The Search Experiment

Harborline pilots Pyroscope on search alone — the highest-QPS, most CPU-bound service, the one place a CPU profiler has the most to say. The flamegraph shows 41% of CPU in JSON serialization of the route objects the cache returns, and the fix is one line. No amount of dashboard-staring would have found it, because no dashboard attributes cost to a function.

Tracewhich span is slow

→

Profilewhich function

→

One-line fix41% in JSON