Send OpenTelemetry metrics to Prometheus over OTLP
Configure the Prometheus OTLP/HTTP receiver and Collector, verify metric names and resource labels, and choose how to handle delta temporality.
The checkout service starts sending OpenTelemetry metrics to a new backend. The Collector accepts the batches, but the Prometheus dashboard stays blank. Start with the actual HTTP request. Prometheus accepts OTLP metrics at /api/v1/otlp/v1/metrics. If the request reaches that route, check the metric name and label set stored in Prometheus before changing the instrumentation.
Get the metrics request to the right URL
On Prometheus 3.x, enable the receiver explicitly. For a local smoke test, save the Prometheus configuration below as prometheus.yml and keep the listener on loopback.
prometheus --web.enable-otlp-receiver --web.listen-address=127.0.0.1:9090 --config.file=prometheus.yml
An SDK configured through environment variables can send metrics directly with OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://127.0.0.1:9090/api/v1/otlp/v1/metrics. The metrics-specific variable takes the complete URL. If you use the general OTEL_EXPORTER_OTLP_ENDPOINT instead, give it the base URL and the HTTP exporter appends the signal path. Configure separate destinations for traces and logs.
With Collector 0.159.0, this local pipeline receives OTLP/HTTP and forwards the metrics. Older Collector builds may call the exporter otlphttp.
receivers:
otlp:
protocols:
http:
endpoint: 127.0.0.1:4318
exporters:
otlp_http:
endpoint: http://127.0.0.1:9090/api/v1/otlp
service:
pipelines:
metrics:
receivers: [otlp]
exporters: [otlp_http]
The Collector adds /v1/metrics to endpoint. Use the complete URL if you set metrics_endpoint. The receiver can accept OTLP/gRPC from applications when configured for it, while the Prometheus-facing exporter still uses HTTP. Prometheus 2.47 introduced the experimental receiver under --enable-feature=otlp-write-receiver. Check the flags on the version you actually run. Prometheus OTLP guide · OTLP exporter specification · Collector HTTP exporter
Read the series that Prometheus actually stored
Suppose checkout emits a cumulative monotonic sum called checkout.requests. With the default translation in Prometheus 3.16, query checkout_requests_total. The strategy normalizes metric and label names and adds applicable type and unit suffixes. UTF-8 support alone does not change the translation strategy. A strategy change can alter the selectors in rules and dashboards, or collapse metrics that previously differed by type or unit.
Prometheus derives job from service.name and prefixes service.namespace when set. It derives instance from service.instance.id. With a namespace of shop, a service of checkout and an instance ID of checkout-1, query checkout_requests_total{job="shop/checkout",instance="checkout-1"}. In Prometheus 3.16, target_info is created for a resource that has a job or instance identity and at least one non-identifying resource attribute. A join on job and instance assumes a single matching info series. A resource change can leave old and new info series visible together for a while.
If the resource already carries an environment attribute and every alert needs to filter on it, promote it. Leave changing pod names in target_info if the alerts do not need them on every series. Under the default translation, the promoted label below is deployment_environment_name.
otlp:
translation_strategy: UnderscoreEscapingWithSuffixes
promote_resource_attributes:
- deployment.environment.name
A new label splits a series when it introduces a new combination of label values. Even values that do not add many simultaneous series can cause churn when workloads restart. Compare active series, growth in new series and memory headroom under your own workload. Prometheus OTLP label mapping · Prometheus 3.16 converter · High-cardinality observability
Pick temporality before moving the dashboard
A cumulative counter carries the running total since its start. A delta point carries the change over one interval. If a dashboard already calls rate() on cumulative counters, keep the SDK on cumulative output when possible, or convert the deltas at a designated Collector. A stateful converter needs a consistent path for each stream. Independent converters can produce different totals after a restart or a traffic shift.
Prometheus 3.2 added experimental otlp-deltatocumulative conversion. It holds state in memory, so a restart resets that state. Prometheus 3.4 added experimental otlp-native-delta-ingestion, which stores raw deltas. The flags are mutually exclusive, and raw deltas need queries suited to the deployed version's delta semantics. Keep cumulative output for existing cumulative-counter queries until the delta queries have been verified. Prometheus feature flags · Version changelog
Separate late batches from duplicate writers
Multiple Collector replicas can deliver batches out of order. If each replica forwards part of the same stream, inspect arrival delay and consider a bounded storage.tsdb.out_of_order_time_window. If two senders write the same samples or use the same instance ID for unrelated streams, first repair the routing or identity. A wider window only allows late samples within its range. It cannot reconcile conflicting writers.
Enable TLS and restrict who can write to the receiver before exposing it beyond loopback. Track exporter failures, retries and queue behavior, then check Prometheus for dropped samples and the series your alerts use. If you send data onward to longer-retention storage via Remote Write, monitor that queue separately. An OTLP push also does not produce the scrape-generated up series for your application. Monitoring the OpenTelemetry Collector
Reconcile the numbers before changing storage
For an existing Guance workspace, the DataKit OpenTelemetry input can receive OTLP metrics at its own configured endpoint. The DataKit Prom collector can scrape an application's existing /metrics endpoint. Keep trace routing and any trace-derived APM metrics separate from those application counters.
If you run both backends during a migration, compare the same service, time window and instrument. Check a known request-count change and a latency histogram before moving alerts. Guance's PromQL query mode can help express the comparison. Check the stored names, labels and bucket boundaries before reusing a query.
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
