Prometheus label limits and cardinality
Understand modern UTF-8 name support, compatibility rules, configurable scrape limits, and the cost of label combinations.
The main operational constraint on labels is cardinality: every distinct metric name and label set identifies another time series. Name validation also depends on version. Prometheus 3 supports UTF-8 metric and label names, while the traditional ASCII conventions remain the safest choice across exporters, tools, and compatible backends.
Naming and value rules
For broad compatibility, use:
Label names: [a-zA-Z_][a-zA-Z0-9_]*
Metric names: [a-zA-Z_:][a-zA-Z0-9_:]*
These are the legacy naming rules and current compatibility recommendations, not a universal ban on UTF-8 names in Prometheus 3. Names outside the traditional character set need the supported quoting/escaping syntax and end-to-end ecosystem support. Colons are reserved by convention for user-defined recording-rule metric names, rather than direct instrumentation.
Names are case-sensitive: Method and method are different labels. Label values can contain UTF-8 text. An empty label value is treated as equivalent to the absence of that label.
Names beginning with __ are reserved for Prometheus. Internal target labels such as __address__ are removed after target relabeling, but do not generalize that to “every internal label disappears everywhere”: __name__ represents the metric name in queries.
Limits can reject an entire scrape
Scrape configuration includes label_limit, label_name_length_limit, and label_value_length_limit, along with sample and body-size limits. Length limits are expressed in bytes, which matters for multibyte UTF-8 characters. Defaults may leave a specific limit disabled, but an Operator, managed service, or compatible backend can impose additional constraints.
When a configured label or sample limit is exceeded after metric relabeling, Prometheus can fail the whole scrape. Inspect the scrape error rather than assuming excess labels are silently truncated. There is no universal “ten labels maximum” rule.
Count combinations correctly
100 instances × 50 routes × 10 statuses = 50,000 possible series
That is an upper-bound combination count for one metric if all combinations occur, not 500,000. A classic histogram adds separate bucket, sum, and count series. Changing label values over time also creates churn, even if the simultaneously active count looks manageable.
There is no universal safe ceiling of a few million series. Capacity depends on labels, churn, scrape frequency, sample type, queries, and available resources. Monitor prometheus_tsdb_head_series, ingestion and series creation rates, memory, and query behavior against a workload-specific budget.
Find expensive metrics
For a bounded job selector, inspect the largest current series counts:
topk(10, count by (__name__) ({job="my-app", __name__=~".+"}))
To sort all selected counts, use sort_desc(count by (__name__) (...)); PromQL does not use a shell-style | sort_desc pipeline. Even a cardinality query can be expensive over a large server. The TSDB status API can provide another view of head-series cardinality.
Keep only labels that support meaningful filtering or grouping. Normalize routes and avoid unbounded user IDs, request IDs, timestamps, and full URLs; place those details in logs or traces. Test a bounded population before expanding instrumentation, and watch both current series and churn.
References
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan