OpenTelemetry best practices for instrumentation, Collectors, and sampling

Follow a request from automatic instrumentation to storage. Set resource attributes, choose a Collector topology, and keep useful traces with the right sampling policy.

Best practices
An illustration of connected systems and telemetry flows

A checkout trace reaches your backend with a server span and a database call. It still tells you little about why the order was rejected. Later, when you lower the sampling rate, the trace for a failed checkout goes missing. These problems start at different points. One is about what your application records. The other is about what the pipeline keeps.

This guide follows one request from instrumentation to storage. It shows where to add business context, where to run a Collector, and when to sample traces.

Start with the instrumentation your runtime supports

Start with the supported instrumentation for your language. The Java Agent can instrument supported libraries at runtime. Python zero-code instrumentation and the Node.js instrumentation packages have their own supported libraries and signal coverage. Check the versions in your application and inspect what actually arrives in the backend. An HTTP span can tell you that an endpoint returned an error. It takes application code to record whether a payment was declined or an inventory reservation expired.

Add a custom span around the part of the request that is still hard to explain. Record a bounded result such as accepted, declined, or timeout. If automatic instrumentation already creates the span you need, use that span for the additional attribute where your library permits it. Check whether the SDK or distribution already creates and installs a provider before wiring one in yourself. OpenTelemetry zero-code instrumentation

Give services stable names and carry context across calls

A resource describes what produced the telemetry. Use service.name for the logical service and service.version for its version. Use the current semantic convention deployment.environment.name for the environment. Keep the service name stable across replicas and releases. For runtimes that support the standard SDK environment variables, a staging example is:

export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES=service.version=1.8.0,deployment.environment.name=staging

Then verify the fields as stored in your backend. Put event-specific information on the span and keep high-cardinality identifiers out of metric dimensions. At a service boundary, check that the sender injects and the receiver extracts the trace context. Baggage travels separately and does not automatically become a span attribute. Keep credentials and personal data out of it. Deployment resource convention · OpenTelemetry Baggage · Trace context propagation

Put the Collector where it solves a real problem

Direct export can work when the backend accepts the signals, protocol, and authentication the SDK uses. Use an agent close to the workload when you need to collect local sources or provide a nearby intake. A gateway is useful when several services need the same processing, routing, or export policy. Teams often use both. Give each hop a reason to exist and monitor it as part of the telemetry path. Collector deployment patterns

Batching can reduce export requests. A sending queue and retries can absorb some short interruptions, subject to the exporter, capacity, and retry settings. An in-memory queue loses its backlog when its process exits. A persistent queue can survive a restart when its storage remains available. Watch queue usage, rejected telemetry, and failed exports. Filter sensitive fields before crossing the relevant trust boundary and inspect the resulting payload. Collector resiliency · Monitor the Collector

Sample traces where you have enough information

A head sampler makes an early decision. A parent-based sampler can follow the incoming sampling decision for child spans, with a probability rule for new roots. A tail sampler can apply error or latency policies after receiving spans. It can only decide on spans that made it to the Collector. A span dropped in the SDK cannot be restored when a downstream error is discovered.

When several Collector instances perform tail sampling, route spans with the same trace ID to the same sampling instance. An ordinary request load balancer can split a trace across replicas. Allow for late spans, queue capacity, and memory before raising the decision wait. Keep service-level request and error metrics on their own aggregation path. If metrics come from spans, check which sampling or filtering occurs before aggregation. OpenTelemetry sampling · Trace-aware Collector routing

Follow a known request into the backend

Configure the DataKit OpenTelemetry receiver for the same OTLP transport as the exporter. For HTTP, match the signal-specific paths in the receiver configuration. Then send one known request from the entry service. In the Guance trace details, check that the synchronous spans share a trace ID, that their parent IDs follow the call, and that the service, environment, and version appear as expected.

If a hop is missing, check instrumentation coverage, context propagation, sampling, and export in that order. Compare both sides of the boundary before changing a sampling policy. If you replace a receiver or backend, validate any existing processing and sampling rules separately. To estimate automatic instrumentation overhead, compare startup time, latency, and resource use with and without the instrumentation under the same workload.

Further reading · What is OpenTelemetry? · Redact sensitive telemetry

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting