Prometheus exporters: how to scrape and verify metrics

Choose a Prometheus exporter, test an NGINX scrape end to end, and tell exporter health from upstream health.

Best practices
Illustration of Prometheus metrics collection

The NGINX dashboard goes quiet, but Prometheus still marks the scrape target as up. A healthy exporter can keep serving /metrics even when it cannot read NGINX. To find the broken hop, check which endpoint Prometheus scraped and whether the exporter reached the system behind it.

A Prometheus exporter reads a system's existing statistics and exposes them in a format Prometheus can scrape. It can translate an HTTP status page, a database view or a device protocol. An application that already exposes useful metrics through a Prometheus client library can be scraped directly. For a system that does not expose the metrics you need, choose an exporter based on what it can read and what it costs to read it. Prometheus exporter directory

Match the exporter to the question

Exporter What it reads What to check
Node Exporter Operating system counters and gauges for CPU, memory, filesystems and networking Which collectors the OS supports and whether the exporter sees the intended host
Blackbox Exporter Results of active HTTP, TCP, ICMP or DNS probes Which network location runs the probe and which module defines success
MySQLd Exporter MySQL status and selected database or performance statistics Server version, enabled collectors and minimum permissions
PostgreSQL Exporter PostgreSQL statistics for connections, locks and replication, among others Which views are accessible and which collectors the installation enables
Redis Exporter Redis information such as memory, keyspace hits and misses, and slow-log statistics Redis version, permissions and collection cost
NGINX Prometheus Exporter Open-source NGINX stub_status or the NGINX Plus API Which source endpoint and exporter mode are configured
kube-state-metrics Kubernetes object state from the API Access to the object types you need. Use a separate source for CPU and memory usage.
SNMP Exporter Device counters exposed through SNMP Device MIB, module, authentication and network access

An exporter can run alongside the application or elsewhere on a network that can reach it. Blackbox and SNMP exporters can probe multiple targets from one deployment. Check the path from the exporter to the source as well as the path from Prometheus to the exporter. They can fail independently.

Scrape open-source NGINX on one machine

Use NGINX 1.30.6 built with http_stub_status_module and the NGINX Prometheus Exporter 1.5.1 binary on your PATH. The NGINX module exposes connection and request counters. The official exporter reads this page in open-source mode and uses the Plus API in NGINX Plus mode. VTS is a separate third-party path.

In the first terminal, make a temporary NGINX configuration. Both listeners in this example bind to loopback.

LAB_DIR=$(mktemp -d)
mkdir -p "$LAB_DIR/logs"
cat > "$LAB_DIR/nginx.conf" <<'NGINX'
pid logs/nginx.pid;
events { worker_connections 128; }
http {
  access_log off;
  server {
    listen 127.0.0.1:18080;
    location = /stub_status { stub_status; }
  }
}
NGINX
nginx -p "$LAB_DIR/" -e "$LAB_DIR/logs/error.log" -c nginx.conf
curl -fsS http://127.0.0.1:18080/stub_status

In a second terminal, start the exporter in the foreground.

nginx-prometheus-exporter \
  --nginx.scrape-uri=http://127.0.0.1:18080/stub_status \
  --web.listen-address=127.0.0.1:19113

Add this job under scrape_configs in a Prometheus instance running on the same machine. If Prometheus runs in a container, use the exporter address reachable from that container instead of its own loopback address.

scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: ['127.0.0.1:19113']

Run curl -fsS http://127.0.0.1:19113/metrics. Selected samples from a short local run look like this. Counts depend on the requests sent to NGINX.

nginx_up 1
nginx_connections_active 1
nginx_connections_handled 4
nginx_http_requests_total 4

Read the two health signals together

Prometheus generates up{job="nginx"} for the scrape of the exporter. The exporter supplies nginx_up for its read of NGINX. If the exporter replies successfully but its status request fails, you can see up=1 alongside nginx_up=0. Check the status path, network reachability and access rules from the exporter's location. If Prometheus has up=0, start with its scrape error and the exporter endpoint. For a blackbox probe, probe_success plays the role of the second signal. Prometheus jobs and instances

Stop the exporter with Ctrl+C in its terminal. In the first terminal, run nginx -p "$LAB_DIR/" -e "$LAB_DIR/logs/error.log" -c nginx.conf -s quit to stop the test NGINX process. In production, restrict access to the status and metrics endpoints to authorized collectors. Plan for authentication and TLS where the traffic crosses a trust boundary.

If your team also uses Guance, the DataKit Prometheus collector can scrape an existing exporter endpoint. Configure the reachable address and verify a gauge and a counter with their labels in the destination. Check what series you keep and how often you scrape if Prometheus and DataKit both read the same exporter.

Reduce volume where the work happens

If you only need CPU, memory and filesystem metrics from a supported Node Exporter installation, enable those collectors and leave the other default collectors disabled.

node_exporter --collector.disable-defaults \
  --collector.cpu --collector.meminfo --collector.filesystem

For an existing Prometheus job, a metric relabeling rule can drop samples before ingestion. This example removes Go runtime and Prometheus HTTP handler metrics from the Node Exporter scrape. Keep them if they are part of your exporter monitoring.

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_.*|promhttp_.*'
        action: drop

Metric relabeling runs after the scrape, so the exporter still collected the data and transferred the response. Prometheus-generated series such as up bypass the rule. Removing a label can also merge several series identities without summing their values. For a high-cardinality family, reduce dimensions at instrumentation or stop collecting the family before it reaches Prometheus. Prometheus metric relabeling

Choose a fallback when there is no exporter

Use a client library if you control the application. To check reachability from outside, use a blackbox probe. For a machine-specific batch job, Node Exporter's textfile collector can expose metrics written by a local script. Consider Pushgateway for the results of short-lived service-level batch jobs. Pushed series remain until deleted, so decide how to clean them up.

Set the scrape interval around the freshness your alerts require and the cost you observe. Compare scrape_duration_seconds with the timeout, and give range queries a window that contains enough samples for that job's cadence.

Further reading · Prometheus relabeling · Prometheus monitoring guide · High cardinality in observability

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting