Prometheus label limits and cardinality

Understand modern UTF-8 name support, compatibility rules, configurable scrape limits, and the cost of label combinations.

Best practices
Illustration of metrics, monitoring, and technical investigation

The main operational constraint on labels is cardinality: every distinct metric name and label set identifies another time series. Name validation also depends on version. Prometheus 3 supports UTF-8 metric and label names, while the traditional ASCII conventions remain the safest choice across exporters, tools, and compatible backends.

Naming and value rules

For broad compatibility, use:

Label names:  [a-zA-Z_][a-zA-Z0-9_]*
Metric names: [a-zA-Z_:][a-zA-Z0-9_:]*

These are the legacy naming rules and current compatibility recommendations, not a universal ban on UTF-8 names in Prometheus 3. Names outside the traditional character set need the supported quoting/escaping syntax and end-to-end ecosystem support. Colons are reserved by convention for user-defined recording-rule metric names, rather than direct instrumentation.

Names are case-sensitive: Method and method are different labels. Label values can contain UTF-8 text. An empty label value is treated as equivalent to the absence of that label.

Names beginning with __ are reserved for Prometheus. Internal target labels such as __address__ are removed after target relabeling, but do not generalize that to “every internal label disappears everywhere”: __name__ represents the metric name in queries.

Limits can reject an entire scrape

Scrape configuration includes label_limit, label_name_length_limit, and label_value_length_limit, along with sample and body-size limits. Length limits are expressed in bytes, which matters for multibyte UTF-8 characters. Defaults may leave a specific limit disabled, but an Operator, managed service, or compatible backend can impose additional constraints.

When a configured label or sample limit is exceeded after metric relabeling, Prometheus can fail the whole scrape. Inspect the scrape error rather than assuming excess labels are silently truncated. There is no universal “ten labels maximum” rule.

Count combinations correctly

100 instances × 50 routes × 10 statuses = 50,000 possible series

That is an upper-bound combination count for one metric if all combinations occur, not 500,000. A classic histogram adds separate bucket, sum, and count series. Changing label values over time also creates churn, even if the simultaneously active count looks manageable.

There is no universal safe ceiling of a few million series. Capacity depends on labels, churn, scrape frequency, sample type, queries, and available resources. Monitor prometheus_tsdb_head_series, ingestion and series creation rates, memory, and query behavior against a workload-specific budget.

Find expensive metrics

For a bounded job selector, inspect the largest current series counts:

topk(10, count by (__name__) ({job="my-app", __name__=~".+"}))

To sort all selected counts, use sort_desc(count by (__name__) (...)); PromQL does not use a shell-style | sort_desc pipeline. Even a cardinality query can be expensive over a large server. The TSDB status API can provide another view of head-series cardinality.

Keep only labels that support meaningful filtering or grouping. Normalize routes and avoid unbounded user IDs, request IDs, timestamps, and full URLs; place those details in logs or traces. Test a bounded population before expanding instrumentation, and watch both current series and churn.

References

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting