Monitor REST APIs with Prometheus

Instrument the API to measure real request traffic, then probe it from outside the process to check reachability. Application metrics describe request rate, errors, and duration; an external probe describes the experience from the probe's network location. A health endpoint alone does not test every user journey.

Record request counts and latency

For a Python application, define metrics once and update them from request middleware:

from prometheus_client import Counter, Histogram

reqs = Counter(
    "api_requests_total", "API requests", ["endpoint", "method", "status"]
)
latency = Histogram(
    "api_request_duration_seconds", "API latency", ["endpoint"]
)

def record_request(route, method, status, elapsed_seconds):
    reqs.labels(endpoint=route, method=method, status=str(status)).inc()
    latency.labels(endpoint=route).observe(elapsed_seconds)

Pass the normalized route, such as /users/{id}, rather than a concrete user URL. Measure elapsed time with a monotonic clock, count failed requests as well as successful ones, and expose the client registry through a /metrics endpoint suitable for your framework. Multi-process servers need the client library's supported multi-process setup; a single worker's registry is not automatically the whole service.

For request rate, error fraction, and classic-histogram P95 latency:

sum(rate(api_requests_total[5m]))

sum(rate(api_requests_total{status=~"5.."}[5m]))
  / sum(rate(api_requests_total[5m]))

histogram_quantile(
  0.95,
  sum by (le) (rate(api_request_duration_seconds_bucket[5m]))
)

Add the relevant job/service selectors in a shared server. The error fraction needs a policy for zero traffic and for absent error series. Initialize bounded counter label combinations when appropriate rather than treating missing instrumentation as success. Histogram quantiles are estimates whose precision depends on bucket boundaries; keep le when aggregating classic buckets. Retain endpoint too if you want separate route percentiles.

Probe the endpoint with Blackbox Exporter

This is a Prometheus scrape configuration for an exporter with the http_2xx module configured:

scrape_configs:
  - job_name: api-probe
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['https://api.example.com/health']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Prometheus contacts the exporter, which contacts the URL. probe_success reports the probe outcome and probe_duration_seconds reports probe duration, which includes more than application handler time. up for this scrape says whether Prometheus could scrape the exporter response; it does not replace probe_success.

TLS probes can expose probe_ssl_earliest_cert_expiry, a Unix timestamp. Subtract time() and divide by 86400 to estimate days remaining. Configure probe authentication, trust, and reachability for the actual endpoint, and restrict who can invoke arbitrary targets through the exporter.

Common questions

Will endpoint labels create excessive series? They will if they include IDs, query strings, or arbitrary paths. Normalize routes and provide a bounded label for unmatched requests.

Should health checks be counted? They can be instrumented, but distinguish their route and filter them from customer-traffic SLOs when appropriate. Otherwise frequent successful health checks can dilute a user-facing error ratio.

References

• Python Prometheus client
• PromQL histogram quantiles
• Blackbox Exporter configuration