High cardinality in observability: how to count and control time series

Measure high-cardinality metrics with PromQL, account for histograms and label churn, and keep request-level detail in logs and traces.

Best practices
Illustration of logs, metrics, and traces converging for analysis

The checkout API looks normal after a deploy, yet Prometheus starts creating far more series. One new label explains the change. The request counter now includes the full URL path, so /orders/81723 and /orders/81724 land in separate series. For an error-rate alert, both requests belong to /orders/{order_id}. To investigate one order, search for its ID in a log or an available trace.

High cardinality means many distinct values or combinations. It matters in observability because a metric system has to keep track of each distinct series. To control the cost, find the combinations that actually exist and decide which ones your queries need.

Count combinations before estimating the cost

A dimension is a label key, such as method, status_code, or service. A label's cardinality is the number of distinct values observed in scope. A metric's series cardinality is the number of distinct label sets associated with that metric name. Check the time window and include target labels such as job and instance when they are present. Prometheus data model

Suppose a counter has seven methods, 30 status codes and 20 services. Holding all other labels constant and assuming all three are present, the maximum is 7 × 30 × 20 = 4,200 combinations. Real traffic is rarely a full cross-product. If only one method and one status occur per service, you observe 20 combinations. Add a million distinct user IDs and the theoretical ceiling becomes 4.2 billion. The actual count depends on which users use which routes and services, and whether other labels vary.

Account for the metric type and label churn

The metric type changes the calculation. For a conventional Prometheus classic histogram, 10 exported bucket series including +Inf, plus _sum and _count, means 12 series per label set for those components. At 4,200 sets, that is 50,400 series. A native histogram stores the distribution as a composite sample in one series per label set. Compare the actual scrape and storage footprint. A native histogram sample can carry more data than a scalar, and a deployment that retains both representations can store more series. Histograms and summaries

Also watch churn. Instance IDs or deployment hashes can change rapidly even when the number of live targets stays flat. Each new identity can create another series. Retained data and historical queries can therefore cover many more series than the current active count suggests. Capacity depends on the store, sample frequency, retention, churn and query workload. A fixed label-value threshold cannot account for all of those variables.

Find the label that changed

Start with an instant count of series visible to PromQL for the affected metric. Break it down by a stable label such as job, then inspect the candidate keys on the jobs that grew.

count(http_requests_total)
sort_desc(count by (job) (http_requests_total))

Run the expressions separately. Compare the same query before and after the deploy, within the data you still retain. This is a count at the evaluation instant, subject to Prometheus lookback and staleness. It does not count every series that existed over an interval. For a local Prometheus server, the TSDB stats API also lists head-block series counts by metric name and value counts by label name. Check what each view measures before comparing them.

Guance's metric management shows time series statistics for measurements and their tags. Use it to narrow a jump to the affected measurement and inspect the tag design. For example, check whether the scrape added an instance dimension, or whether an application update started attaching a full request path.

Fix the metric at the earliest useful point

Use route templates, bounded status groups and stable service names where those dimensions answer your operational questions. Change the instrumentation or the aggregation so the counter sums requests at the intended granularity before exporting. Then check that its totals and alert queries still match expectations.

During an incident, dropping an entire noisy metric family with metric_relabel_configs can protect ingestion while you repair the source. Check the recording rules and dashboards that depend on it first. Removing only user_id from counters that were independently accumulated for each user can give several samples the same identity. Prometheus does not sum those counters during relabeling. Similarly, a recording rule can make dashboard queries cheaper by creating a lower-cardinality series, while ingestion still pays for the originals. Prometheus relabeling

With an OpenTelemetry SDK that supports metric Views, keep the needed metric attributes before aggregation. Where a cardinality limit is implemented, the SDK can combine measurements that exceed it under otel.metric.overflow=true. Treat that point as a capacity signal. It has already lost the per-attribute breakdown. Check your SDK version and reader settings before relying on it. OpenTelemetry Metrics SDK

Keep useful detail in logs and traces

Keep the request ID or another permitted correlation key in structured records that you can actually retrieve. A dashboard can tell you which route became slow. A matching log or retained trace can show the individual request and its downstream calls. Sampling and retention affect whether the trace will still be there. Redact credentials and other sensitive data, and keep only the fields you need.

A columnar store can make selective queries efficient, but a high-cardinality field still has write, storage and query costs. In systems such as Loki, using request IDs as indexed stream labels creates many streams. Store them as structured metadata where supported, or query the parsed records. Measure with your own retention period and the filters people actually use. Loki label cardinality

Further reading · Logs, metrics, and traces · Prometheus relabeling in practice · Redact sensitive data in the Collector

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting