Manage Prometheus counters correctly

Keep a counter cumulative for the lifetime of the process or entity it describes. Increment it for events, use counter-aware queries for activity over time, and keep its labels bounded. Do not periodically subtract from or reset a counter just to produce daily totals.

Define and name the metric

from prometheus_client import Counter

http_requests = Counter(
    "http_requests_total", "Total HTTP requests", ["method", "status"]
)
http_requests.labels(method="GET", status="200").inc()

_total is the conventional counter suffix. The Python client normalizes this suffix when exposing a counter; do not assume every client library performs identical name rewriting.

Use labels such as normalized HTTP method and status when they support useful filtering. User IDs, request IDs, and arbitrary URLs usually create unbounded series and belong in logs or traces instead.

Let the query handle observed resets

rate(http_requests_total[5m])
sum(increase(http_requests_total[1h]))

Ordinary process-local counters can restart from zero without being restored from persistent storage. rate() and increase() adjust for sampled decreases, but they cannot guarantee a continuous curve or recover all events around an unseen reset. For a business total that must survive crashes exactly, use durable business records and choose instrumentation that accurately reflects that source.

Compute the counter function before aggregation so that individual resets remain visible. A raw counter value is still useful for inspection; rate and increase are specifically for changes over time, not mandatory for every counter query.

Manage storage and series growth

Local Prometheus storage defaults to 15 days when neither a time nor size retention policy is set. Configure retention according to capacity, or use an appropriate remote storage architecture for longer history. Mimir commonly receives remote write; Thanos can upload blocks through a sidecar or receive samples through Thanos Receive. They are different integration paths.

When retiring a metric, stop emitting it or remove the selected collection path. Existing history ages out under retention; do not delete the entire database merely to remove a deprecated counter.

Watch prometheus_tsdb_head_series and the rate of new series creation. High cardinality is a label-design problem that can affect any metric type, not uniquely counters. Histograms can multiply the cost further through bucket series.

Common questions

Can a counter decrease? Its application-level updates should not decrease it. Resets are a separate lifecycle event. Choose a gauge for a value that naturally moves in both directions.

Why does rate spike? Check the instrumentation, scrape gaps, interval alignment, resets, duplicate ingestion, and workload first. A wider window can smooth noise; a window of roughly four scrape intervals is a useful starting point, not a cure for incorrect data.

Is an info metric a counter? No. Information metrics conventionally expose a value of one with descriptive labels and can enrich queries through vector matching. They do not count metadata changes.

References

• Python client counters
• PromQL counter functions
• Prometheus storage