OpenTelemetry metrics in Go: instruments, Views, and OTLP

Instrument HTTP requests and orders with the OpenTelemetry Go Metrics SDK, drop a metric stream with a View, and verify OTLP/gRPC export locally.

Best practices
Illustration of inspecting telemetry across an application

A checkout endpoint can slow down while the number of orders it accepts stays flat. A request-duration histogram tells you how response times moved. A separate order counter tells you how many requests actually created an order. Go’s OpenTelemetry Metrics SDK lets you record both and send them over OTLP.

This guide follows two synthetic HTTP requests through the Go SDK into a local OpenTelemetry Collector. One succeeds and one returns 503. That difference gives us a quick way to check the automatic HTTP metrics, the business instruments, and the View that drops an unused stream.

Set up a local OTLP receiver

Save the following as collector.yaml and run otelcol-contrib --config=collector.yaml. The gRPC receiver listens on loopback. The debug exporter prints the payload so you can inspect it without a backend.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:14317
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [debug]

The example was checked with Go 1.26, OpenTelemetry Go SDK 1.47.0, the Go contrib instrumentation 0.72.0, and Collector Contrib 0.165.0. Install the dependencies in an empty directory.

go mod init example.com/otel-order-metrics
go get go.opentelemetry.io/otel/metric@v1.47.0 \
  go.opentelemetry.io/otel/sdk@v1.47.0 \
  go.opentelemetry.io/otel/sdk/metric@v1.47.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc@v1.47.0 \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.72.0 \
  go.opentelemetry.io/contrib/instrumentation/runtime@v0.72.0

Wire the MeterProvider to the handler

The MeterProvider gives the application a resource name, a periodic reader, and a View. The reader exports the metric points over OTLP/gRPC. The 30-second interval applies while the process is running. This short example sends its final collection through Shutdown, using a separate timeout after the HTTP test server closes. Run the program as go run . after saving it as main.go.

package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"log"
	"net/http"
	"net/http/httptest"
	"strings"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	otelruntime "go.opentelemetry.io/contrib/instrumentation/runtime"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
	"go.opentelemetry.io/otel/metric"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
)

func orderHandler(meter metric.Meter) (http.Handler, error) {
	created, err := meter.Int64Counter("orders.created", metric.WithUnit("{order}"))
	if err != nil {
		return nil, err
	}
	inFlight, err := meter.Int64UpDownCounter("orders.inflight", metric.WithUnit("{request}"))
	if err != nil {
		return nil, err
	}
	queueDepth, err := meter.Int64Gauge("orders.queue.depth", metric.WithUnit("{order}"))
	if err != nil {
		return nil, err
	}
	duration, err := meter.Float64Histogram("orders.process.duration", metric.WithUnit("s"),
		metric.WithExplicitBucketBoundaries(0.05, 0.1, 0.25, 0.5, 1, 5))
	if err != nil {
		return nil, err
	}

	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ctx := r.Context()
		started := time.Now()
		inFlight.Add(ctx, 1)
		defer inFlight.Add(ctx, -1)
		defer func() { duration.Record(ctx, time.Since(started).Seconds()) }()
		queueDepth.Record(ctx, 2)

		if _, err := io.Copy(io.Discard, r.Body); err != nil {
			w.WriteHeader(http.StatusBadRequest)
			return
		}
		if r.URL.Query().Get("fail") == "1" {
			w.WriteHeader(http.StatusServiceUnavailable)
			return
		}
		created.Add(ctx, 1, metric.WithAttributes(attribute.String("channel", "app")))
		w.WriteHeader(http.StatusCreated)
	}), nil
}

func run() (err error) {
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	exporter, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithEndpoint("127.0.0.1:14317"),
		otlpmetricgrpc.WithInsecure())
	if err != nil {
		return err
	}
	mp := sdkmetric.NewMeterProvider(
		sdkmetric.WithResource(resource.NewSchemaless(attribute.String("service.name", "order-service"))),
		sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exporter, sdkmetric.WithInterval(30*time.Second))),
		sdkmetric.WithView(sdkmetric.NewView(
			sdkmetric.Instrument{Name: "http.server.request.body.size"},
			sdkmetric.Stream{Aggregation: sdkmetric.AggregationDrop{}},
		)),
	)
	defer func() {
		shutdownCtx, stop := context.WithTimeout(context.Background(), 5*time.Second)
		defer stop()
		err = errors.Join(err, mp.Shutdown(shutdownCtx))
	}()

	if err := otelruntime.Start(otelruntime.WithMeterProvider(mp)); err != nil {
		return err
	}
	const route = "/orders"
	handler, err := orderHandler(mp.Meter("example.com/order-service"))
	if err != nil {
		return err
	}
	mux := http.NewServeMux()
	mux.Handle("POST "+route, otelhttp.NewHandler(
		handler, route, otelhttp.WithMeterProvider(mp),
	))
	server := httptest.NewServer(mux)
	defer server.Close()

	for _, suffix := range []string{"", "?fail=1"} {
		req, err := http.NewRequestWithContext(ctx, http.MethodPost,
			server.URL+route+suffix, strings.NewReader("demo"))
		if err != nil {
			return err
		}
		resp, err := server.Client().Do(req)
		if err != nil {
			return err
		}
		resp.Body.Close()
		fmt.Println(resp.StatusCode)
	}
	return nil
}

func main() {
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

WithInsecure() disables transport security. Here the receiver is a local test fixture. For remote ingestion, configure the receiver’s address, TLS and any required credentials for that deployment.

Read HTTP metrics and business metrics separately

The program prints 201 and 503. On the Collector, look for service.name=order-service and http.server.request.duration. With this version of otelhttp, the duration histogram records one observation per completed instrumented request. Its Count gives request volume for the same dimensions. The two responses have different status-code attributes, so expect a count of one for each. The application counter orders.created should equal one in total, with channel=app.

The HTTP middleware also records body sizes. The SDK View drops the metric stream named http.server.request.body.size before export. Response body size and request duration remain. If you need a metric with fewer dimensions, use a View attribute filter and confirm that the remaining labels still answer your query. Dropping an entire stream and removing attributes from a retained stream change different parts of the payload.

Choose an instrument for each measurement

Question Instrument What it records
How many orders did the handler create? Counter Nonnegative increments. Aggregate them with a query that understands the stream’s temporality and resets.
How many requests are being handled? UpDownCounter Add one on entry and subtract one on exit with matching attributes. A read after both requests sees zero.
What is the current queue depth? Gauge Record a value from the source of truth. The example uses the fixed value two. An ObservableGauge can read the queue during collection.
How long did processing take? Histogram Record elapsed seconds into a distribution. The example includes a bucket boundary at 250 ms.

Set histogram boundaries around the thresholds you use in alerts. P95 estimated from explicit buckets depends on their resolution. When combining distributions across replicas, keep bucket layouts compatible and make sure the backend combines the distributions before estimating a percentile.

Use stable metric names and declared units such as s or By. Keep attributes to dimensions you query, such as channel and region. A request ID or full URL can create a new time series for each request. Use a route template such as /orders/{id} instead. How metric cardinality grows

Send the same metrics to Guance

The local Collector shows the OTLP payload before any backend-specific conversion. To use DataKit, enable its OpenTelemetry input and check the receiver configuration and network path. Point the gRPC exporter at the configured gRPC listener. If you choose OTLP/HTTP instead, use the HTTP/Protobuf exporter and its configured metrics path. DataKit’s documented default HTTP metrics path is /otel/v1/metrics, and its default HTTP listener port is 9529. DataKit OpenTelemetry input

Send a known number of requests and check the received metric name, unit, attributes and histogram representation before setting up a monitor. For an investigation that also uses traces, align service and environment values across signals and confirm the spans were collected. An aggregate point covers a group of observations and usually cannot identify one trace. Configure Guance monitors

Questions that come up in production

What happens to a counter when the service restarts?

A cumulative OTLP point can carry a start timestamp. When the process restarts, the new stream starts accumulating again. Verify how the backend computes rates across the reset. If you configure delta temporality, the exporter reports interval changes and the backend needs to interpret them accordingly.

Can I aggregate a Summary across replicas?

Client-computed quantiles cannot generally be combined into a fleet percentile. Go’s Metrics API exposes Histogram as its distribution instrument. A backend can combine compatible histograms and estimate the percentile from that combined distribution.

How often should the SDK export?

The example uses 30 seconds. Start with an interval that fits your alert window, the available receiver capacity, and the data freshness you need. Shorter intervals send more batches. Measure the impact at your own volume.

How do I collect Go runtime metrics?

This example calls runtime.Start on the same provider, which registers runtime measurements such as go.goroutine.count and go.memory.gc.goal in the pinned contrib version. To collect the scheduler latency histogram go.schedule.duration, add runtime.NewProducer() to the reader as documented. Check names and availability for your pinned version. Go runtime instrumentation

References

Related reading

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting