OpenTelemetry sampling: head decisions, tail policies, and limits

Trace sampling decides which traces to retain and which evidence to discard. It can reduce collection, transport, or storage work, depending on where the decision happens. The trade-off is that a request needed for a later investigation may no longer be available.

Head sampling decides early, using information available when a span starts. Tail sampling decides later, using spans that have reached a sampling processor. Neither guarantees that every important trace survives. Choose the policy together with its failure behavior, routing, and metric strategy.

Estimate the volume before choosing a policy

Use measured span rates and encoded sizes. As an illustrative calculation, 10,000 spans per second × 1,000–2,000 bytes × 86,400 seconds produces 864 GB–1.728 TB per day, in decimal units, before compression and transport or storage overhead. Tens of thousands of spans per second are not merely a few gigabytes per day at those sizes.

Serialized span size, compressed network volume, and backend billable ingestion are different measurements. Sampling can reduce some of them, but a Collector making a tail decision has already received the upstream data. Confirm the point at which costs are incurred before projecting savings.

Healthy requests are also useful: they establish normal paths and performance distributions. Retaining failures and slow traces preferentially changes the population, so keep representative baseline data and separate complete metrics where needed.

Head sampling decides at span creation

An SDK sampler can use the trace ID, parent decision, and attributes available when a span starts. A parent-based ratio policy applies a probability to new root traces and normally respects a sampled or unsampled parent for descendants.

The early decision can avoid recording and exporting many spans, reducing downstream traffic and SDK work. It does not make instrumentation overhead zero: context propagation and some instrumentation logic still run, and metrics or logs have their own behavior.

The limitation is missing future information. A root span that looks ordinary at creation may later encounter an error or become slow. With a simple independent 10% root probability, a rare error also has only that probability of being selected, rather than guaranteed retention. Small samples can vary substantially from the intended percentage.

A Python SDK configuration can make the parent behavior explicit:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

provider = TracerProvider(
    sampler=ParentBased(root=TraceIdRatioBased(0.25))
)

This configures a provider; a complete application still needs to use that provider, configure a processor/exporter, and shut it down appropriately. The 25% probability applies to roots without a parent decision. TraceIdRatioBased alone does not express the same parent-following policy.

Supported SDK auto-configuration can use OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.25. These are generally read during initialization. Updating an environment variable does not imply a live change in already-running workers; runtime or remote reconfiguration requires explicit implementation support. See the SDK configuration specification.

Tail sampling decides from accumulated spans

The Collector's tail-sampling processor groups spans by trace ID and evaluates policies over the data available to it. Policies can retain traces containing errors, traces above a duration threshold, a named business operation, or an approved bounded customer category.

Unlike an early ratio decision, this can use observed outcomes. It must first receive and hold data, which adds memory, latency, and routing requirements. A configured wait is not proof the trace is complete: long-running or delayed spans may arrive after the decision, and upstream loss may leave permanent gaps.

All spans belonging to one trace must reach the same sampling instance for that instance to evaluate them together. Use a trace-aware routing design for multiple replicas and account for changes in the destination set. A generic load balancer distributing individual export requests can split traces. Stateful sampling also needs a rollout and failure plan; an instance restart loses in-memory trace state.

Configure an error, latency, and baseline policy

The following complete trace-only example targets Collector Contrib 0.162.0. It listens locally and writes selected test traces to the debug exporter:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
  batch:
    timeout: 1s

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [debug]

Validate it with the selected binary before sending controlled data:

otelcol-contrib validate --config=sampling.yaml
otelcol-contrib --config=sampling.yaml

These three positive policies combine as a union: an error match, a latency match, or the baseline decision can retain a trace. The order is not an exclusive “try errors, otherwise slow, otherwise baseline” rule. Advanced drop, composite, and other policy types have their own decision semantics. Read the versioned processor documentation when changing the policy set.

status_code here examines OpenTelemetry span status, not an HTTP response-code attribute. An HTTP 5xx response needs the appropriate instrumentation status or a separate attribute-based policy. Likewise, the latency policy uses the span timing range observed for the trace; missing spans can change that range.

The example intentionally does not set a customer policy because its attribute and privacy rules depend on the application. If you add one, use a stable, bounded category and test what happens when the attribute is missing. Merely naming a paid tier does not guarantee all of its requests reach the sampler.

Plan for incomplete traces and finite memory

decision_wait: 10s controls decision timing for the default strategy; it is not a declaration that every trace completes in ten seconds. A shorter wait reduces retained state but may miss late errors. A longer wait holds more data. Decision caches and late-span behavior are additional version-specific controls.

num_traces is a capacity in traces, not spans or bytes. A rough starting model for buffered span data is peak incoming spans per second × waiting time × in-memory span size, plus processor maps, trace metadata, decision caches, exporter queues, and other overhead. In-memory size differs from serialized size, and a few very large traces can dominate memory. A fixed “add 50%” rule is not a validated capacity plan.

Capacity pressure can evict or drop traces before the desired decision, and a memory limiter can refuse data earlier in the pipeline. Monitor refused data, early trace drops, decision latency, sampled versus dropped counts, queue pressure, and process memory. Test long traces, delayed exports, overload, and a destination outage with representative traffic.

Processors that need original connection context, such as Kubernetes association, generally belong before tail sampling because regrouping can lose that context. Keep batching after the sampling decision unless component requirements justify another order.

Verify the retained population in Guance

Replace debug output with an exporter matching the DataKit OTLP receiver, checking its actual protocol, port, path, and authentication requirements. Use known trace IDs for normal, slow, and error cases and confirm the expected retained traces and their spans at the destination.

Compare counts before and after sampling, and inspect exporter failures and processor drops. An alert firing is not evidence that all the spans needed to investigate it survived. Head-dropped data cannot be restored by a downstream tail sampler.

Common questions

Does trace sampling affect metric accuracy?

It depends on where metrics come from. Independently recorded application counters and histograms are not inherently sampled by the trace sampler. Metrics derived from spans after filtering or sampling describe that selected population. If span-derived totals must represent all traffic, compute them before the relevant sampling boundary and account for upstream loss. Do not treat error-biased retained traces as an unbiased latency distribution.

How do we change a head-sampling rate?

Use the SDK's supported startup configuration, then restart or roll out the affected processes as required. A live configuration mechanism is a separate feature that must be tested across workers. Verify parent behavior during mixed-version or mixed-policy rollouts.

Should applications sample when the Gateway already does?

For outcome-based tail decisions over all traffic, applications normally need to record and export all candidate spans. Early head sampling reduces that traffic but permanently narrows what tail policies can see. If combining them, document the upstream selection and use consistent parent behavior; independent downstream decisions can fragment traces.

How much memory does a sampler need?

Measure at peak trace arrival rate, with realistic spans per trace, trace duration, attributes, and late arrivals. Size trace capacity and process memory together and monitor eviction. There is no safe universal number derived only from request rate.

See the OpenTelemetry sampling overview and the Collector guide. The related context propagation, OTTL patterns, Collector monitoring, and OpenTelemetry introduction guides are currently in Chinese.