Instrument Go services with OpenTelemetry tracing

A useful Go trace connects the incoming HTTP request to the work it triggers: business logic, outbound HTTP calls, database queries, and background operations that belong to that request. Creating spans is only part of the job. Those spans need the same propagated context, a configured exporter, and enough time to leave the process during shutdown.

This guide uses the OpenTelemetry Go SDK and library instrumentation. Go also has automatic instrumentation options with their own runtime and library constraints; the SDK approach gives you explicit control over business operations and context boundaries. See the Go instrumentation guide.

Initialize the SDK and propagation together

Start in a new module or use your application's existing module:

go mod init example.com/orders-tracing
go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Commit the resolved go.mod and go.sum through your normal dependency process. Use compatible releases and a Go version supported by those releases.

Save this initialization code as tracing.go:

package main

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func initTracing(ctx context.Context) (*sdktrace.TracerProvider, error) {
    // Local demo: a Collector listens for plaintext OTLP/gRPC here.
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("127.0.0.1:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    res := resource.NewSchemaless(
        attribute.String("service.name", "order-service"),
        attribute.String("service.version", "1.0.0"),
        attribute.String("deployment.environment.name", "development"),
    )
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithResource(res),
        sdktrace.WithBatcher(exporter),
        sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.AlwaysSample())),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))
    return tp, nil
}

The resource identifies the service producing the spans. NewSchemaless is sufficient for these explicitly supplied attributes; if you use resource.New with detectors, handle its returned error.

The tracer provider records and exports spans. The text-map propagator separately extracts and injects W3C traceparent, tracestate, and baggage headers. Installing a provider alone does not configure HTTP propagation. Include only approved data in baggage because it can travel to downstream services.

The plaintext endpoint is for a local demonstration. For a remote Collector, configure the actual address, TLS, authentication, and network policy. A gRPC WithEndpoint value is a host and port, not an HTTP URL. Exporter construction also does not prove that the receiver is reachable; check subsequent export errors and received spans.

Instrument the HTTP server and its outbound call

Save this as main.go beside tracing.go:

package main

import (
    "context"
    "errors"
    "io"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/codes"
)

var client = &http.Client{
    Transport: otelhttp.NewTransport(http.DefaultTransport),
    Timeout:   5 * time.Second,
}

func ordersHandler(w http.ResponseWriter, r *http.Request) {
    ctx, span := otel.Tracer("example.com/orders").Start(r.Context(), "orders.list")
    defer span.End()

    target := os.Getenv("INVENTORY_URL")
    if target == "" {
        target = "https://example.com/"
    }
    err := fetchInventory(ctx, target)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "inventory request failed")
        http.Error(w, "Inventory unavailable", http.StatusBadGateway)
        return
    }
    _, _ = io.WriteString(w, "Orders are ready\n")
}

func fetchInventory(ctx context.Context, target string) error {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
    if err != nil {
        return err
    }
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    if resp.StatusCode >= 400 {
        return errors.New("inventory returned an error status")
    }
    // This demo only checks the response; a real client decodes its payload.
    _, err = io.Copy(io.Discard, resp.Body)
    return err
}

func main() {
    tp, err := initTracing(context.Background())
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer cancel()
        if err := tp.Shutdown(ctx); err != nil {
            log.Printf("tracing shutdown: %v", err)
        }
    }()

    mux := http.NewServeMux()
    mux.Handle("/orders", otelhttp.NewHandler(
        http.HandlerFunc(ordersHandler), "GET /orders",
    ))
    server := &http.Server{
        Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second,
    }
    failures := make(chan error, 1)
    go func() { failures <- server.ListenAndServe() }()
    stop, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer cancel()
    select {
    case <-stop.Done():
    case err := <-failures:
        if !errors.Is(err, http.ErrServerClosed) {
            log.Printf("HTTP server: %v", err)
        }
    }
    ctx, done := context.WithTimeout(context.Background(), 10*time.Second)
    defer done()
    if err := server.Shutdown(ctx); err != nil {
        log.Printf("HTTP shutdown: %v", err)
        _ = server.Close()
    }
}

The server wrapper creates an incoming request span and places its context in r.Context(). The business span inherits that context. The instrumented client creates a child span and injects propagation headers into the outbound request. The receiving service must extract those headers with compatible instrumentation.

Read or close every response body. Besides releasing resources, this lets the HTTP instrumentation finish tracking the response body. A client timeout limits this demo's downstream request; production response-size limits and parsing depend on the API. See the otelhttp package.

The shutdown sequence first stops accepting HTTP work, waits for active handlers, and then shuts down tracing with a fresh timeout. Reusing an already canceled signal context would prevent a useful flush. Graceful shutdown is still bounded: a deadline, export failure, process kill, or machine failure can lose buffered spans.

Add database and Redis instrumentation

HTTP spans reveal downstream requests, but database and Redis operations need their own instrumentation. For go-redis/v9, add the matching instrumentation package:

go get github.com/redis/go-redis/v9
go get github.com/redis/go-redis/extra/redisotel/v9

The following application fragment installs tracing before the client is used:

rdb := redis.NewClient(&redis.Options{Addr: "localhost:6379"})
if err := redisotel.InstrumentTracing(rdb); err != nil {
    return err
}
defer rdb.Close()
value, err := rdb.Get(ctx, "order:123").Result()

Import github.com/redis/go-redis/v9 and github.com/redis/go-redis/extra/redisotel/v9. Here ctx must come from the current operation, and the surrounding function must handle value and err, including the normal redis.Nil missing-key case. Review command-argument capture before enabling it for data that may contain secrets. Follow the go-redis instrumentation instructions.

For SQL, choose instrumentation compatible with your driver and database/sql, such as the external XSAM/otelsql project. These wrappers are not interchangeable across every database library. Keep QueryContext and ExecContext tied to the request context, and verify whether statements or arguments are captured. Avoid stacking two wrappers that both create a span for the same query.

Route spans through a Collector

The intended path is Go application → Collector → the configured DataKit OpenTelemetry receiver. The application's local port 4317 belongs to the Collector in this example; it is not an assumption about DataKit's listening port.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
processors:
  batch: {}
exporters:
  otlp_http/datakit:
    traces_endpoint: ${env:DATAKIT_OTLP_TRACES_URL}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_http/datakit]

Set DATAKIT_OTLP_TRACES_URL to the full trace ingestion URL supported by your DataKit receiver, including its trace path. Check the receiver configuration and the Guance OpenTelemetry integration. Do not assume that changing a gRPC destination to an http:// string switches protocols.

Current Collector documentation uses otlp_http; older distributions use otlphttp. Use the component name available in your installed distribution consistently in both exporters and the pipeline. Validate against that Collector version before rollout. A Collector running in another container also needs a reachable receiver bind address and appropriate port/network configuration; loopback only accepts local connections. See Collector configuration.

Batching reduces export overhead. It does not provide an unlimited or durable queue. Size queues and memory limits, decide retry/storage requirements, and monitor refused or failed exports for your production deployment.

Keep context attached to goroutine work

A goroutine does not automatically receive a request context. Pass it explicitly for work that belongs to the request:

go func(ctx context.Context) {
    ctx, span := otel.Tracer("example.com/orders").Start(ctx, "refresh-cache")
    defer span.End()
    refreshCache(ctx)
}(r.Context())

This is a fragment: refreshCache is your function. A server request context is canceled when the handler returns, so this pattern only suits work whose lifetime is intentionally bounded by the request. Coordinate completion when the response depends on that work.

For durable or detached work, define a separate job lifecycle, deadline, and shutdown behavior. Carry trace context through the queue or job envelope, then decide whether the consumer should continue the trace or link a new trace. Replacing the context with context.Background() without propagation discards the parent relationship; retaining a request context forever creates a different lifecycle problem. Trace context and cancellation need deliberate treatment at that boundary.

Verify the trace in Guance

Enable the DataKit OpenTelemetry receiver, start the Collector and Go application, and request /orders. Search Guance APM for order-service in the intended environment and time range.

Inspect one successful request and one controlled downstream failure. Confirm that the server, business, and outbound spans share a trace ID; their parent IDs form the expected tree; the failed operation has an error status; and the downstream service joins the trace when instrumented. Then examine the service topology and span waterfall to locate where time is spent. Missing spans can result from sampling, lost context, exporter failures, or a query filter—not just a slow dependency.

Before changing sampling, establish that this small flow works end to end. During an incident, compare APM data with application logs carrying the actual active trace ID, rather than treating an unrelated request ID as a trace identifier.

Common questions

Can Go tracing work without changing application code?

Automatic instrumentation exists, including options based on eBPF and compile-time instrumentation. Coverage and prerequisites vary. Evaluate the available Go approaches against your frameworks and deployment, and add business spans when automatic coverage lacks the operation you need.

How do I sample high-volume traffic?

For a root sampling ratio of 20% while respecting parent decisions, replace the sampler with sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.2)). The actual observed proportion varies, and upstream sampling decisions affect child services. Collector tail sampling can select received traces using later evidence, but cannot recover spans already discarded by head sampling.

Can I keep tracing disabled in local development?

You can omit the recording SDK and use a no-op provider when export is unnecessary. For debugging instrumentation, a local console exporter is often more informative. Keep propagator setup explicit if local services still need to forward incoming trace context.

Why is an outbound call missing from the trace?

Check that the instrumented client is actually used, the request was created with the active context, and the response body was consumed or closed. Then check sampling and export failures. A new background context or a second uninstrumented client can break an otherwise correct setup.

Related guides

Continue with OpenTelemetry Node.js tracing or contextual logging in Go to keep application events connected to request traces.

The broader series also covers Collector deployment on Kubernetes (Chinese), Go metrics (Chinese), the OpenTelemetry SDK (Chinese), context propagation (Chinese), and sampling (Chinese).