Prometheus monitoring: how collection, metrics and storage work
Understand Prometheus scraping, metric types, Pushgateway, Graphite and StatsD, retention and availability—plus the limits that matter in production.
Prometheus is an open-source monitoring and alerting system built around time series. It collects numerical measurements, stores their history and lets you query them with PromQL. The project graduated from CNCF in 2018 and is widely used in Kubernetes environments.
For an API, those measurements might describe request counts, errors and latency. They help show when behavior changed and which instances were affected. Understanding how the values reach Prometheus matters just as much as drawing the graph: a failed scrape, a restarted counter and a missing request log mean different things.
How Prometheus collects a measurement
An application or exporter exposes an HTTP endpoint, commonly /metrics. Prometheus scrapes that endpoint on a configured interval, such as every 15 seconds. A text response can look like this:
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{method="GET", status="200"} 1027
http_requests_total{method="POST", status="500"} 3
A metric name and its complete label set identify a time series. Each sample has a value and timestamp; when the response does not supply a timestamp, the scrape time is normally used. Changing method or status creates another series, so labels let a query separate errors from successful requests without giving every combination a different metric name. Data model · Exposition format
Prometheus knows which targets it is configured to scrape. The generated up series is 0 when a scrape fails. That could be a stopped process, a network problem, rejected credentials or an invalid metrics response; it does not establish that the host is down. The pull protocol does not require a target to remember Prometheus's scrape position, but application metrics can still have state. A process restart can reset counters, and a later successful scrape does not reconstruct every missing observation. Jobs and instances
Where the Pushgateway fits
A service-level batch job may finish before Prometheus can scrape it. The Pushgateway can retain that job's result and expose it for a later scrape. This is a limited use case, rather than a default route for every short-lived process or serverless function.
The gateway retains pushed series until they are deleted. Its up value describes the scrape of the gateway, not whether the original job is still running. Job naming and cleanup therefore become part of the monitoring design. For a machine-specific batch job, other approaches, such as the Node Exporter's textfile collector, may fit better. Pushgateway guidance
Choose the metric type before the query
| Type | What it represents | Typical use |
|---|---|---|
| Counter | A cumulative value that increases and can reset, for example after a restart | Requests completed or errors observed |
| Gauge | A current value that can increase or decrease | Memory use, temperature or queue depth |
| Histogram | A distribution represented by classic buckets or native histogram samples | Estimating latency percentiles at query time |
| Summary | Observation totals and counts, with optional client-calculated quantiles | Local quantiles whose aggregation limits are understood |
A counter's current value is not a request rate. PromQL's rate() accounts for observed resets when estimating its increase over a window. For summaries, the precomputed quantiles cannot be combined into a service-wide percentile by averaging instances. Their _sum and _count series can still be aggregated according to their meaning, for example to calculate a mean duration. Histogram percentiles are estimates too; the representation and resolution affect their accuracy. Metric types · Histograms and summaries · Query functions
The components around Prometheus
Exporters expose measurements from systems that do not already speak the Prometheus exposition format. Node Exporter covers Linux host metrics; mysqld_exporter covers MySQL measurements and blackbox_exporter performs endpoint probes. Service discovery finds changing targets through systems such as Kubernetes, Consul or cloud APIs. It reduces the need to maintain a static list as instances come and go.
PromQL filters and combines the resulting series. Prometheus evaluates alerting rules; Alertmanager handles grouping, deduplication, routing and silences for the alerts it receives. Grafana provides a common dashboard interface, while Prometheus remains the metrics backend in that arrangement.
Graphite and StatsD occupy different parts of this picture. Graphite receives, stores and queries time series. Its traditional naming model encodes dimensions in dotted paths, and the 1.1 release series added tagged series. Prometheus uses metric names and labels as series identities and includes active collection and alert-rule evaluation. A Graphite-versus-Prometheus decision therefore involves the data model and operating workflow, not just a choice between push and pull. Graphite tags · Product boundaries
StatsD is a metrics reporting and aggregation component, not a complete equivalent to Prometheus. Existing StatsD instrumentation can feed statsd_exporter, which maps and exposes Prometheus metrics. Check the timer, counter, label and aggregation-window semantics when doing so; dashboards and alerts do not migrate simply because data starts arriving. StatsD Exporter
What Prometheus does well
Prometheus fits infrastructure and application metrics, particularly when targets change frequently. Kubernetes service discovery and the exporter ecosystem give teams practical collection paths; labels and PromQL let them investigate across instances rather than inspect one host at a time.
Each server operates independently with local storage. That can keep monitoring usable when another observability backend fails, provided Prometheus, its targets and their network paths are still available. Autonomy reduces external dependencies; it does not remove those operating requirements.
Storage, availability and precision need separate decisions
A local TSDB is not limited to 15 days. That is the default only when neither the time nor size retention option is set. Retention can be extended with suitable capacity planning; when both limits are configured, whichever takes effect first removes old data. A separate backend is another choice. Mimir can receive remote write, while a Thanos Sidecar can upload Prometheus TSDB blocks to object storage. Those are different data paths with different components to operate. Storage · Thanos Sidecar · Sending to Mimir
For an existing Prometheus installation, Guance's DataKit Remote Write input provides another receiving path. Collection can remain in Prometheus while the team configures and operates that receiver. Check metric names, label mapping, timestamps and supported metric representations before relying on the resulting data. Retention, queries, dashboards and alerts still need their own configuration. Guance Remote Write integration
The local TSDB is not a replicated distributed database. Independent Prometheus replicas can provide redundant collection, and Alertmanager supports an HA cluster for notification handling. A unified query view, replica deduplication or distributed storage requires additional design and often components such as Thanos, Cortex or Mimir. Alertmanager HA · Thanos Query
Prometheus stores metrics rather than full logs or traces, so those investigation paths need other data stores. Its periodically collected statistics also should not be the sole record for per-request billing or exact auditing. Scrape gaps and counter resets can leave incomplete evidence. Scope and accuracy
Three questions before deployment
Prometheus or Zabbix? Prometheus is a natural candidate for a Kubernetes and exporter-based metrics workflow. For established host and network-device monitoring, compare Zabbix's collection and templates against the systems you actually operate. Existing coverage, alert ownership and the team's operating process matter more than a universal winner; the two systems can also run alongside each other.
Can Prometheus monitor Windows? Yes. windows_exporter, formerly wmi_exporter, exposes Windows system and supported application metrics. Coverage depends on the enabled collectors, permissions and operating-system version. Comparing its metric count with Linux Node Exporter does not establish whether a Windows workload is adequately monitored. Windows Exporter
How large can one server be? There is no reliable universal ceiling. Active series, series churn, scrape interval, ingestion rate, concurrent queries, rules and disk performance all affect capacity. Size against the actual workload and measure CPU, memory, storage and scrape duration. A million active series is not, by itself, a stability guarantee.
Technical sources checked on November 4, 2026. This article explains documented behavior; it does not report a capacity benchmark.
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
