Group Prometheus query results by label

Use by to retain the labels that define each aggregation group. For example, sum by (status) (rate(http_requests_total[5m])) produces one request-rate series per status. Use without when it is easier to name the dimensions you want to remove.

Choose the grouping labels

# Requests per second by status.
sum by (status) (rate(http_requests_total[5m]))

# Split each status group by HTTP method.
sum by (status, method) (rate(http_requests_total[5m]))

# Combine instances while retaining other ordinary labels.
sum without (instance) (rate(http_requests_total[5m]))

sum(rate(http_requests_total[5m])) by (status) is an equivalent placement of the modifier. by drops labels that are not listed; without drops the listed labels. An ordinary aggregation also drops the metric name. Labels retained by without, such as job, can leave more groups than you expected.

Match the aggregation to the question

An average of per-instance latency averages is not generally a request-weighted service average. For histogram latency, aggregate the rates of the sum and count and divide them instead. Similarly, count counts series, not automatically unique hosts: extra labels can produce several series per host.

Keep reset handling and cardinality in mind

Take rate() before aggregation so that it can see each counter's resets. A valid subquery around a pre-aggregated counter does not restore the lost reset information.

More grouping dimensions can produce more output series. User IDs and request IDs are usually unsuitable metric labels; keep them in logs or traces. There is no universal safe limit of a few dozen values: measure the total series count, churn, query workload, and storage cost.

Common questions

Can one aggregation use both by and without? No. Choose one modifier for that aggregation. Nested aggregations can have their own separate grouping.

How can I make a crowded graph readable? Filter to the relevant population or use topk. In a range query, the top three can change at each step, so the complete graph may still contain more than three series.

Can I group by part of a label value? Use label_replace() to extract a bounded, meaningful component into a new label, then aggregate by it. A regex matcher alone filters values; it does not rename or group them.

References

• PromQL aggregation operators
• PromQL rate and label functions
• Prometheus metric naming practices