Cloud Trace
Cloud Trace is the distributed tracing service in Google Cloud Observability. Every request gets a trace ID; every operation in its handling — an HTTP call to a downstream service, a database query, a cache lookup — becomes a span. Spans are linked in parent-child relationships and reconstruct the full causal tree of what handled the request, in what order, and how long each piece took. Where logs answer "what happened" and metrics answer "how often", traces answer "where did the time go".
The 2026 standard for distributed tracing is OpenTelemetry (CNCF). Google's tracing stack is OTel-native — Cloud Trace accepts OTLP, GCP managed services emit OTel-compatible traces, and the SDKs for application instrumentation are the same ones that work with any other tracing backend. The platform lock-in concern from the older OpenCensus era is gone; trace data is portable.
Distributed Tracing Basics
0–800ms
50–120ms
130–600ms
65–80ms
A trace is the full end-to-end record of one request through the system. A span is one operation inside that trace — an HTTP call, a function execution, a database query, a queue publish. Spans have a name, a start time, a duration, and a parent (the span that caused them). The tree of spans is the trace.
Context propagation is what makes distributed tracing work across service boundaries. When service A calls service B, A injects the trace ID and the current span's ID into the request headers; B reads them and continues the trace. Without context propagation, A and B record independent traces with no link between them, which defeats the purpose. The standard headers are defined by the W3C Trace Context specification; OpenTelemetry SDKs handle them by default.
OpenTelemetry on GCP
OpenTelemetry SDKs are available for every major language — Python, Java, Go, Node.js, Ruby, .NET, C++. Instrumentation comes in two flavors: auto-instrumentation uses language-specific hooks to capture spans for common frameworks (HTTP clients, ORMs, RPC libraries) without code changes, and manual instrumentation adds custom spans for business-level operations the auto layer cannot see.
GCP managed services that handle requests (Cloud Run, Cloud Functions, App Engine, GKE with the Workload Identity-aware setup) emit OTel spans for the platform's part of request handling. Application code adds its own spans on top. The result is a trace that reaches from "request entered the load balancer" through "application logic ran" down to "database returned in 27 ms".
Sampling Strategies
At any meaningful scale, capturing every span of every request is too expensive in cost and storage. Tracing must sample. Two strategies:
- Head-based sampling — decide at the start of the request whether to capture it. Random N% of requests get full traces; the rest produce no spans. Simple, predictable cost, but tail latency outliers (the slow requests that are exactly the ones you want to investigate) are sampled at the same N% as fast requests, which means most slow requests are dropped.
- Tail-based sampling — capture everything in memory at the trace level and decide after the trace finishes whether to keep it. The decision can use real signals: "keep if any span errored", "keep if total duration was above the p99". Tail-based catches the slow requests; the cost is more memory in the collector, and the complexity of a collector layer (OpenTelemetry Collector with tail-sampling processor).
The right pattern for production: head-based sampling at a low baseline rate (1-5%) for representative coverage, plus tail-based for errors and slow requests so the outliers always make it through.
Trace-Log Correlation
Traces and logs answer different questions but get much more powerful together. The pattern: every log entry that an application emits during a request handler should include the current trace ID as a structured field. Cloud Logging recognizes the field; the Log Explorer offers a "jump to trace" link; the Trace Explorer offers a "jump to logs" link. Investigating a slow request becomes a single click between the slow span and the log lines emitted during it.
In practice this requires the application to pull the current trace ID from the OpenTelemetry context and include it in every log call. Most language ecosystems have a structured-logging library that does this automatically once OTel is installed; pick one and the correlation works without per-call code.
Cloud Trace vs Third-Party APM
Cloud Trace — native GCP integration, free at modest volumes, OpenTelemetry-native, tight links to Cloud Logging and Cloud Monitoring. The right default for workloads primarily on GCP, especially when the team uses the other Google Cloud Observability components.
Datadog APM / New Relic / Honeycomb — richer trace analytics UIs, deeper RED-metrics dashboards, BubbleUp and high-cardinality exploration features that Cloud Trace does not have. The right choice when trace analytics is a core part of how the team operates production, especially for orgs that already pay for those vendors.
Because both sides accept OpenTelemetry, the instrumentation is portable. A team can start with Cloud Trace and move to a third-party APM later (or run both in parallel during a transition) without rewriting instrumentation. The lock-in tax that used to come with picking a tracing backend is largely gone.
- 100% sampling in production for any workload above modest RPS. Trace volume scales linearly with traffic; the storage bill follows.
- Head-based sampling only. The slow requests — the ones the team most wants to investigate — are sampled at the same rate as the fast ones, so most outliers are lost.
- Context propagation not configured across service boundaries. Traces are cut at every service hop; what arrives at the dashboard looks like dozens of unrelated tiny traces instead of one end-to-end story.
- Log entries without trace ID. Jumping from a slow span to "what was the application actually doing" requires manual time-window correlation across the log explorer, which is the opposite of the integrated experience tracing is supposed to provide.
- Custom or vendor-specific tracing libraries instead of OpenTelemetry. The instrumentation becomes locked to one backend; switching tracing providers means re-instrumenting the application.
- Tracing data accumulated but never used — no RED metrics derived, no alerts on percentile latency, no review during incident postmortems. The bill is paid; the value is not extracted.
- OpenTelemetry SDK in every application service. Standard, portable, with broad auto-instrumentation coverage for HTTP, gRPC, databases, queues.
- Head-based sampling at a low baseline (1-5%) plus tail-based sampling for errors and slow requests. The combination keeps cost predictable while catching the requests that actually matter.
- Context propagation via W3C Trace Context headers across every service boundary. Most OTel SDKs do this by default; verify it works end-to-end.
- Trace ID in every application log entry as a structured field. The bidirectional jump between traces and logs is what makes integrated observability useful.
- RED metrics (Rate, Errors, Duration) derived from traces, exported to Cloud Monitoring, alerted on percentile latency spikes (p95, p99).
- Trace exploration as a regular part of incident postmortems. A trace shows what other instrumentation cannot.
Knowledge Check
What is the relationship between a trace and a span?
- A trace and a span are exactly one and the same thing; the two terms are fully interchangeable
- A trace is the end-to-end record of one request through the system; a span is one operation inside that trace, with a parent-child relationship to the span that caused it
- A trace is the application-level record of the call, while a span is the entirely separate platform-level record emitted automatically by Google's underlying infrastructure
- A trace is a sampled set of many requests grouped together, and each span is one of those individual requests within the set
Why use OpenTelemetry rather than vendor-specific tracing libraries?
- OpenTelemetry is meaningfully cheaper to run than the commercial vendor SDKs, purely because it follows a community open-source licensing model that carries no per-seat or ingest fees
- OTel is the industry-standard SDK; instrumentation written against it is portable across backends (Cloud Trace, Datadog, New Relic, Honeycomb) without rewriting application code
- OTel is strictly required by Cloud Trace nowadays, because the older vendor-specific tracing libraries no longer integrate with the service at all
- OTel transparently provides automatic end-to-end encryption of all trace data in flight, a protection that the vendor SDKs simply do not offer
What is the principal weakness of head-based sampling alone?
- Head-based sampling is fundamentally incompatible with W3C Trace Context propagation and cannot be combined with it across services
- Tail latency outliers — the slow requests you most want to investigate — are sampled at the same low rate as fast ones, so most of them are dropped
- Head-based sampling reliably produces traces with broken, disconnected parent-child span relationships once a request crosses between services
- Head-based sampling forces every single request to allocate trace storage at the very start, which dramatically increases memory pressure on each service
Why include the trace ID as a structured field in every application log entry?
- Cloud Logging automatically deduplicates repeated log entries that share the same trace ID, which meaningfully reduces overall log storage cost
- It enables bidirectional navigation: from a slow trace span to the log entries written during it, and from a log line back to its enclosing trace — making investigation one click rather than manual correlation
- A populated trace ID field is strictly required on each entry before any log-based metric can be defined, extracted, or evaluated against those logs in Cloud Monitoring
- Including a trace ID on each entry automatically extends that log's retention, because Cloud Logging reclassifies trace-tagged entries as audit-relevant and keeps them around far longer than ordinary application logs
When does Datadog APM, New Relic, or Honeycomb earn its place over Cloud Trace?
- Whenever the workload is built on plain HTTP services, since Cloud Trace can only instrument and capture gRPC traffic between services
- Whenever the team has standardized on OpenTelemetry, because Cloud Trace ingests only its own proprietary wire protocol and rejects OTLP
- When trace analytics is core to how the team runs production — richer exploration UIs, RED-metrics dashboards, and BubbleUp-style high-cardinality features that Cloud Trace does not have
- Whenever the workload is multimodal, because only the third-party APMs are actually able to trace requests as they flow across mixed audio, image, and video processing pipelines from end to end
You got correct