Prometheus Monitoring: The Complete 2026 Guide (Metrics, PromQL, Alerting, Remote Write)

Short answer: Prometheus is the de facto standard for metrics monitoring in cloud-native systems: it pulls metrics over HTTP from instrumented targets, stores them as time series, evaluates PromQL for dashboards and alerts, and ships data to long-term storage via remote write. This guide covers the four metric types, the PromQL patterns you will actually use, alerting rule design, and the production pitfalls — cardinality explosions, storage limits, and HA — with copy-paste configs throughout.

What Prometheus is (and is not)

Prometheus is a metrics database plus a collection engine, not a full observability platform. It does not store logs or traces; it does not provide dashboards beyond a basic expression browser (Grafana fills that role); and it is not designed for multi-year retention. What it does better than anything else: scrape, store, and alert on numeric time series with a label-based data model that matches how Kubernetes-era infrastructure actually works.

The four metric types, with examples

Type	Semantics	Example	PromQL pattern
Counter	Only goes up (requests, errors)	`http_requests_total`	`rate(http_requests_total[5m])`
Gauge	Goes up and down (memory, queue depth)	`node_memory_MemAvailable_bytes`	direct query
Histogram	Distribution in buckets (latency)	`http_request_duration_seconds`	`histogram_quantile(0.95, rate(...[5m]))`
Summary	Client-side quantiles	`go_gc_duration_seconds`	direct quantile read (not aggregable)

The label model is the superpower: every series is metric_name{label="value",...}. A single metric like http_requests_total{service="payments", status="500", pod="..."} replaces hundreds of hierarchical metric paths.

Related guidePrometheus Integration

Setup: from zero to scraping in 10 minutes

# prometheus.yml — minimal production shape
global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']        # node_exporter
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod                          # auto-discovery via annotations
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"

PromQL: the 6 patterns that cover 90% of work

# 1. Request rate per service
sum by (service) (rate(http_requests_total[5m]))

# 2. Error ratio (alert on this, not raw errors)
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# 3. p95 latency from a histogram
histogram_quantile(0.95, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

# 4. CPU saturation per node
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

# 5. Predict disk exhaustion (linear extrapolation)
predict_linear(node_filesystem_avail_bytes[6h], 24*3600) < 0

# 6. Top-k offenders
topk(5, sum by (pod) (container_memory_working_set_bytes))

Alerting rules that don't page you at 3 a.m. for nothing

groups:
  - name: service-health
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
        for: 10m                      # the single most important line: flapping killer
        labels: { severity: page }
        annotations:
          summary: "{{ $labels.service }} error ratio > 5% for 10m"

Three rules of alert hygiene: alert on ratios and symptoms (error rate, latency SLO burn) rather than causes (CPU); always use for: to suppress blips; route by severity so warnings never page.

Scaling: remote write, HA, and the cardinality trap

A single Prometheus comfortably handles hundreds of thousands of active series per host, but three walls arrive eventually:

Retention/local storage — Prometheus keeps data locally (default 15 days). Long-term storage is delegated via remote write to backends like Mimir, Thanos, VictoriaMetrics — or SaaS platforms that accept the protocol (Grafana Cloud, Guance, and others).
High availability — Prometheus has no clustering; HA means two identical replicas plus deduplication at the read layer (Thanos/Mimir/Cortex pattern).
Cardinality explosion — the #1 production failure. A label with unbounded values (user_id, request_id) multiplies series combinatorially; one bad label can take a server down in hours. Rule: audit labels in code review, and keep count by (__name__)({__name__=~".+"}) on your own dashboard.

# remote write to any compatible backend
remote_write:
  - url: https://<your-backend>/api/v1/write
    # Guance, Grafana Mimir, Thanos receiver all accept this protocol

Where Prometheus fits in a full stack

Prometheus owns metrics. Logs go to Loki/Elasticsearch or a platform backend; traces go to Tempo/Jaeger or OTLP endpoints; dashboards go to Grafana; and the correlation work — pivoting from a metric spike to the log line to the trace — is where all-in-one platforms (Datadog, Guance, New Relic) differentiate. A common 2026 architecture keeps Prometheus as the collection standard while remote-writing into such a platform, which is exactly what Guance's Prometheus integration supports: keep your exporters and PromQL assets, gain unified storage, correlation, and usage-based pricing.

FAQ

Q: Is Prometheus free?
Yes — fully open source under Apache 2.0, graduated CNCF project. The real cost is operational: storage, HA setup, and the engineer time to run it. Managed options (Grafana Cloud, Amazon Managed Prometheus, Guance remote write) trade money for that operational burden.

Q: Prometheus vs Grafana — what's the difference?
Prometheus collects and stores metrics; Grafana visualizes them (and many other sources). They are complements, not competitors — the standard stack is Prometheus + Grafana. "Grafana Cloud" is the vendor's managed bundle of both plus logs and traces.

Q: How many metrics can Prometheus handle?
A single well-sized server handles hundreds of thousands to low millions of active series. Beyond that, shard by team/service or move to a horizontally scalable backend (Mimir, Thanos, VictoriaMetrics) via remote write. Cardinality discipline matters more than hardware.

Q: Does Prometheus monitor logs or traces?
No — metrics only. The standard companions are Loki (logs) and Tempo (traces) in the Grafana ecosystem, or OpenTelemetry Collector feeding an all-in-one platform. Prometheus can export alerts derived from log-derived metrics, but raw log search is out of scope.

Q: What is remote write and why does it matter?
Remote write is Prometheus's protocol for streaming samples to external storage as they are scraped. It decouples collection (Prometheus, which you keep) from storage/query (a backend you choose freely). It is the standard bridge for adopting SaaS observability without abandoning existing exporters and PromQL dashboards.

Sources: Prometheus official documentation (metric types, PromQL, remote write), Prometheus 3.x release notes. Verified 2026-08-07. Published by Guance — the Prometheus integration accepts standard remote write.