How to monitor the OpenTelemetry Collector

Scrape the Collector’s internal metrics, diagnose receiver and exporter failures, and read queue pressure alongside process and container resources.

Best practices
Illustration of connected systems under inspection

The application is still serving requests, but new spans have stopped appearing in the trace backend. On the Collector, the receiver is accepting spans and the sending queue is filling. That narrows the investigation to the export path. If the Collector's own metrics have disappeared too, check whether the scraper can reach it before drawing conclusions about application traffic.

Monitoring an OpenTelemetry Collector starts with three views of the same time window: process resources, the flow of telemetry through the pipeline, and the Collector's logs. Keep the monitoring path reachable when the destination that receives your application data is unavailable. The configuration and metric names below were checked with the official OTLP Collector distribution, v0.165.0, using synthetic traces on loopback.

Expose the Collector's own metrics

The default Prometheus endpoint listens on 127.0.0.1:8888. For a Collector on the same network namespace as the scraper, you can make that choice explicit. Add this fragment under service alongside your existing pipeline definitions.

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: '127.0.0.1'
                port: 8888
                without_type_suffix: true
                without_units: true
    logs:
      level: info
      encoding: json

Run curl -fsS http://127.0.0.1:8888/metrics from that namespace. Check the endpoint, a process metric such as otelcol_process_memory_rss, and the names and labels of the component metrics you plan to query. Receiver or exporter series may only appear once those components handle data.

If your scraper runs outside the Collector's network namespace, choose a reachable bind address. For example, bind to 0.0.0.0 inside a Pod and restrict access with your network controls. Keep this endpoint separate from the OTLP receiver. On v0.123.0 and later, the old service.telemetry.metrics.address key no longer sets the address.

The example keeps the Prometheus type and unit suffixes disabled, matching the v0.165.0 defaults. If you set both options to false, the same endpoint exposes otelcol_receiver_accepted_spans_total and otelcol_process_uptime_seconds_total. Internal telemetry sent over OTLP retains the instrument names. Read the endpoint or inspect the receiving backend before copying dashboard expressions between deployments. Collector internal telemetry

Follow spans from the receiver to the exporter

Receiver and exporter counters show what happened at different stages. Track per-instance and per-component rates. Use the comparable log-record or metric-point counters for the other signals.

Metric in the example What it tells you
otelcol_receiver_accepted_spans The receiver passed spans into the pipeline.
otelcol_receiver_refused_spans The receiver could not pass spans into the pipeline. Inspect backpressure or a downstream rejection.
otelcol_exporter_enqueue_failed_spans The exporter could not place spans in the sending queue. Check capacity and the queue configuration.
otelcol_exporter_send_failed_spans The exporter reported a send failure. Check exporter logs and retry behavior to understand the outcome.
otelcol_exporter_sent_spans The exporter reported a successful send. Check the backend separately if spans still do not appear there.

Don't subtract export rates from ingest rates to calculate loss. Sampling and filtering intentionally change volume, fan-out can send one span to several exporters, and queueing shifts traffic across time windows. Some components also expose their own metrics or omit the shared helper metrics. If a processor is in question, check its incoming and outgoing counts where available, then check its documented behavior and the pipeline that actually uses it.

Read queue occupancy and resource pressure together

Plot otelcol_exporter_queue_size beside otelcol_exporter_queue_capacity for the same instance, exporter and signal. Where capacity is positive, dividing the two gives the fraction used. In v0.165.0, an exporter-helper sending queue measures both with its sizer. The default counts requests. items or bytes changes the unit. Check the config before comparing two queues, and expect these gauges only when the component exposes them.

A queue rising while send throughput falls points you toward the exporter and its destination. In our loopback check, a deliberately stalled backend filled a four-item queue. The next three spans failed to enqueue and the receiver returned HTTP 503. With retries disabled, the backend returned HTTP 400 for the first blocked request and HTTP 200 for the next one. Two previously accepted spans failed to send and two were sent. The same outage showed up differently at each stage. Exporter helper queue configuration · Collector resiliency

Put process RSS, heap allocation and CPU beside container memory limits, throttling and restart events. A rising heap, a rising RSS and an OOM restart tell you different things. Check sampling state, buffering, concurrency and expensive transforms against the observed load. In Kubernetes, use your container metrics and deployment metadata to add Pod context. Scrape labels come from discovery and relabeling or from your telemetry settings.

If you already collect metrics in Guance, DataKit's Prometheus collector can scrape the reachable endpoint. Confirm the stored metric names, measurement mapping, labels and interval in DataKit before building the dashboard. Pair those metrics with Collector logs from the same instance so that an authentication error can be separated from a backlog or a scrape outage.

Put the first useful signals on the dashboard

  • Resource pressure. Chart RSS and heap allocation alongside container memory, limits and restarts. Use the rate of otelcol_process_cpu_seconds for CPU, and verify any extra runtime metrics against the endpoint.
  • Flow and buffering. Plot accepted and refused rates per receiver, sent and failed rates per exporter, and queue occupancy per signal. Include scrape health so an unreachable Collector stands out from an idle pipeline.
  • Errors. Group Collector logs by exporter or error class. Correlate them with queue growth and send failures. For a latency panel, use an instrument that actually covers the path you want to measure.

Set thresholds from normal bursts, the backlog you can tolerate and the retry window. Alert separately on sustained queue pressure, actual enqueue failures and lost scrapes. A fixed percentage copied from another deployment can miss a short overflow or page you for a queue that always drains safely.

Match a symptom to the next check

Symptom First check
Queue rises and sent rate falls Check destination latency, throttling, network access and retry logs. Size the queue for a bounded disruption.
RSS rises and the container restarts Confirm the termination reason and the container limit, then check buffering, tail sampling state and in-flight requests.
CPU saturates during peak traffic Compare throughput and processor configuration. Regex, transforms, compression and export can each add work.
Receiver accepts data and the destination is empty Check whether the data is sampled or routed elsewhere, then check the exporter endpoint, errors and backend ingestion.

Practical questions

How often should I scrape? Start with an interval your monitoring system can sustain, then measure scrape size, duration and series count. Keep the interval short enough to resolve the backlog behavior you care about.

How do I monitor replicas? Keep a stable instance label, group replicas by their role, and keep per-instance panels alongside aggregates. A single saturated replica can disappear into a healthy fleet average.

Which log level should I use? Start with info. Enable debug for a bounded investigation if needed, then restore the previous level. Verify log volume and access before sharing output.

What if DataKit and the Collector are both in the path? Monitor both hops. The Collector can handle the OTLP pipeline while DataKit scrapes its internal Prometheus endpoint. If telemetry also flows from the Collector to DataKit, monitor that handoff separately.

Further reading · What is OpenTelemetry? · OTTL in the Collector · OpenTelemetry vs Prometheus

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting