Eight Prometheus best practices for production
Design Prometheus metrics, scope PromQL, handle missing data and plan alerting and storage as the workload grows.
A quiet traffic period and a failed scrape can both leave a dashboard blank. Those cases call for different responses, and you need enough detail in the metrics to tell them apart. These eight practices follow the data from instrumentation to queries, alerts and storage.
Name metrics so their units survive a query
Use a consistent style such as http_request_duration_seconds for duration and http_requests_total for a counter. Prefer base units such as seconds and bytes. Give application-specific metrics a namespace when it clarifies who owns them. Keep one quantity and unit per metric. If summing or averaging its series produces a meaningless value, check whether the metric mixes things that should be separate. Prometheus naming guide
Budget for label combinations and churn
A status label with a few states is predictable. Adding product_id adds combinations of product, status, replica and any other labels. The actual series count depends on the combinations emitted. IDs, timestamps and untemplated URL paths are usually poor metric dimensions because they grow or change quickly. Put request-level detail in logs or traces, and use route templates or bounded categories in metrics. High cardinality in observability
Count attempts and failures together
Count every attempt and increment the failure counter for the failed subset. The two counters should cover the same population, labels and time window. Initialize known series when the client supports it, including the failure series before the first failure. Then the complement of the failure fraction represents success if every counted attempt is either a success or a failure.
1 - (
sum(rate(requests_failed_total{job="api", env="production"}[5m]))
/ (sum(rate(requests_total{job="api", env="production"}[5m])) > 0)
)
Here > 0 filters the denominator without turning it into a Boolean. With zero traffic, the ratio is absent. A missing or undersampled failure series can also make the ratio absent, so check the source counters and scrape health before interpreting the result. Avoid filling an unknown failure rate with zero. Instrumentation guidance · PromQL operators
Scope the query before you aggregate
If production and staging share a backend, select the environment before summing. Keep service if the panel should separate services. Add job or cluster where the same metric is used across different workloads.
sum by (service) (rate(http_requests_total{env="production"}[5m]))
Apply rate() to each counter before aggregating so that resets can be detected per series. Confirm that the labels in the selector are present on your data. PromQL rate
Give a sustained condition time to become actionable
A for duration keeps an alert pending until the same alert label set remains present in the expression result for that duration at rule evaluations. When the label set stops matching before the timer completes, it leaves the pending state. Choose the duration together with the query range, rule evaluation interval and time you can afford to wait. A fast-moving outage can warrant a shorter delay than a resource trend.
groups:
- name: api-errors
rules:
- alert: ApiHighErrorRate
expr: sum by (service, instance, env) (rate(requests_failed_total{job="api", env="production"}[5m])) > 10
for: 10m
labels:
severity: warning
annotations:
summary: '{{ $labels.service }} / {{ $labels.instance }}'
This rule is an example for per-instance failures above ten per second. Tune the expression and duration to the service and check what Alertmanager will route. Prometheus alerting rules
Tell an unsuccessful scrape from missing data
absent(http_requests_total{job="api", env="production"})
up{job="api", env="production"} == 0
absent(up{job="api", env="production"})
Read the expressions separately. The first detects when no request-counter series match the selector. The second identifies currently discovered targets whose scrape failed. The third detects when no matching scrape-health series remain. For the up selectors to work, set env as a scrape target label, for example through target relabeling. An env label on the application metrics alone does not carry over to up. If one replica disappears from discovery while others still match, a job-wide absent() stays empty. Use an expected target inventory or a narrower selector if you need to detect that case. Jobs and instances · PromQL absent
Keep the labels a responder needs
Choose the level at which someone can act. Group by service and env for a service-wide alert, and include instance only if someone needs to act on each replica. Every distinct output label set can become a separate alert. Keep stable labels for routing or grouping, and put changing values or a runbook link in annotations. Check the grouping, matchers and receivers in Alertmanager alongside the rule. Alertmanager configuration · Reduce alert fatigue
Scale the path that is under pressure
Start by measuring active series, churn, ingestion, disk use and query cost. Prometheus local storage is a single-node database. The time-retention default is 15 days when neither a time limit nor a size limit is configured. Retention does not give you replica redundancy.
Use federation when an upper-level Prometheus needs to scrape current values for selected series, often local aggregates. Use Remote Write when a compatible receiver should receive samples for central storage or queries. Receiver retention, querying and availability depend on the destination. Account for sender queues, disk and network when you add the write path. Storage · Federation · Remote Write tuning
If you already operate Prometheus, the DataKit Remote Write receiver gives you a path to send selected metrics to Guance. Start with a small set and verify the resulting metric names and tags. Recording and alerting rules remain in their original evaluator until you configure and validate an equivalent elsewhere.
Questions that come up in practice
Which metric has the most series? Use Prometheus's TSDB status page for Head statistics, or scope a count of query-visible series to the job or environment you are investigating. Across all metrics this query can be expensive.
topk(10, count by (__name__) ({__name__=~".+"}))
When are recording rules useful? They help when dashboards or alert rules repeatedly run the same expensive expression. They store new derived series, and the original series still cost storage unless you change ingestion.
Is an environment label enough to isolate production? It scopes queries and routing. Use separate instances or tenant controls when you need resource, failure or access isolation.
Further reading · Prometheus relabeling · Kubernetes health checks
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
