Prometheus relabeling: choose targets, labels, and samples

Use Prometheus relabeling to filter scrape targets, normalize labels, shard Kubernetes targets, and drop samples before ingestion or Remote Write.

Best practices
Illustration of Prometheus target discovery and metrics processing

A Kubernetes pod exposes /metrics. Prometheus can reach it, but you want to scrape only opted-in pods and leave one noisy metric out of storage. Those are two different decisions. A target rule can prevent the HTTP request. A metric rule runs after the response arrives and can discard selected samples before ingestion.

Prometheus relabeling changes labels or drops objects at a particular stage. Start by deciding what must disappear from service discovery, local storage, or a remote destination. Then choose the configuration field.

Choose the stage first

Configuration When it runs What you can change
relabel_configs in a scrape job After discovery, before scraping Keep or drop targets, change the scrape address, copy discovery metadata onto public labels
metric_relabel_configs After scraping, just before ingestion Keep or drop scraped samples, or change their labels
write_relabel_configs Before sending samples to a particular Remote Write endpoint, after external labels Control what that endpoint receives while leaving local ingestion alone
alert_relabel_configs Before sending alerts to Alertmanager, after external labels Normalize alert labels or filter alerts before delivery

Dropping a sample with metric_relabel_configs still leaves the HTTP scrape and parsing work. It also leaves automatically generated series such as up outside the rule. A successful up therefore tells you that a scrape succeeded, even if the metric you are looking for was dropped. Prometheus configuration

Follow one target through relabeling

Rules run in order. In a Kubernetes pod scrape job, __address__ supplies the target address. __scheme__ and __metrics_path__ control the request, and labels such as __meta_kubernetes_namespace come from discovery. If a later query needs a discovery label, copy it to a public label before target relabeling finishes. Discovery metadata is removed after relabeling, and internal __* fields such as the scrape address stay out of the public target labels. __tmp is reserved for intermediate results. If you leave instance unset, Prometheus uses the final __address__. Sample relabeling has a different input and can read the metric name from __name__.

relabel_configs:
  - source_labels: [__meta_kubernetes_namespace]
    separator: ';'
    regex: '(production|staging)'
    replacement: '$1'
    target_label: env
    action: replace

Here replace reads the namespace and writes env only for an exact production or staging match. Other targets still proceed. Prometheus matches RE2 patterns against the entire input, treats missing source labels as empty strings and joins multiple source values with a semicolon by default. replace is the default action. Its default regex is (.*) and default replacement is $1. Use parentheses for a capture that you plan to reference. Relabeling reference

Pick the action that matches the job

Action Behavior
keep / drop Match the joined source values against a regex. keep rejects nonmatches and drop rejects matches.
replace Match the joined source values and write the replacement to a target label.
labelkeep / labeldrop Keep matching label names or remove matching label names. They operate on names.
labelmap Copy the values of matching label names to new names. The source labels remain at that step.
hashmod Hash the joined source values and write the remainder for a positive modulus.
keepequal / dropequal Compare the joined source values to the current target label value and keep or drop on exact equality. Use source_labels and target_label for these actions.
lowercase / uppercase Convert the joined source value and write it to the target label.

This rule keeps only targets in production. It does not set a label.

- source_labels: [__meta_kubernetes_namespace]
  regex: 'production'
  action: keep

To keep the pod name as useful query context, put it in pod. If two endpoints in one pod expose the same metric, rewriting both instance values to the pod name can collapse their identities.

- source_labels: [__meta_kubernetes_pod_name]
  target_label: pod
  regex: '(.*)'
  replacement: '$1'
  action: replace

To publish just the Kubernetes labels you use, map a small allowlist:

- regex: '__meta_kubernetes_pod_label_(app|team)'
  replacement: '$1'
  action: labelmap

This makes app and team available to queries after the discovery labels are stripped. A broad match can also copy a changing label such as a build identifier or overwrite a label you already set. For example, the next rule removes names on whichever label set it runs against:

- regex: 'build_sha|request_id'
  action: labeldrop

Use it only after checking that distinct series remain distinct. If a name and the remaining labels are identical, Prometheus cannot preserve the original series distinction.

Split a scrape job across two shards

With the same candidate targets and a modulus of two, one shard keeps remainder 0 and the other keeps remainder 1. Each target belongs to one shard. The actual counts depend on the input addresses and their hashes.

- source_labels: [__address__]
  modulus: 2
  target_label: __tmp_hash
  action: hashmod
- source_labels: [__tmp_hash]
  regex: '0'
  action: keep

Run identical preceding target rules on both shards so they hash the same key. A second replica with the same residue scrapes the same subset, which can be intentional for high availability. Change the residue to split work. Changing the modulus or hash input can move targets between shards.

Read opt-in annotations and override the port

Use these rules in a scrape job with Kubernetes pod discovery. The first rule requires the opt-in annotation. The second replaces the port when the port annotation contains digits and the discovered address has the expected host or bracketed IPv6 form. If that second pattern fails, the original address remains. Check the resulting target address and make sure the selected port actually exposes metrics.

- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
  regex: 'true'
  action: keep
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __address__]
  regex: '([0-9]+);(\[[^]]+\]|[^:]+)(:[0-9]+)?'
  replacement: '${2}:${1}'
  target_label: __address__

The Prometheus job owns these rules. If you also send metrics to Guance through DataKit, configure its KubernetesPrometheus collector separately. Enable the relevant annotation discovery option and check the annotations and selectors that the collector expects before using both collection paths.

Drop one expensive metric and verify the result

Use a specific name after finding a metric you can discard. This example filters http_request_debug_total and leaves the rest of the scrape intact.

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'http_request_debug_total'
    action: drop

Before reloading, validate the configuration. The second command runs the named job's discovery and shows the discovered and relabeled labels. It needs access to the discovery source. Adjust the job name to match your configuration. promtool reference

promtool check config prometheus.yml
promtool check service-discovery prometheus.yml kubernetes-pods

In the Prometheus UI, compare Service Discovery with Targets. If the target is present and up is one, query the retained metric and the name you filtered. That separates a target-selection problem from sample filtering. To reload a running server, use SIGHUP or the HTTP POST endpoint /-/reload if the lifecycle API is enabled.

Questions that come up when labels change

Why did the regex not match? Check the actual concatenated value. The regex matches the whole string. Use a more specific pattern or an explicit .* when you really want a substring match.

Why does $1 produce an empty string? The pattern needs a matching capture group. Use ${1} to separate the group number from the characters that follow it, and $$ for a literal dollar sign in the replacement.

Will this rename old data? Existing series keep their labels. A changed label set can start a new series on future scrapes, and queries that span the change can encounter both sets.

Further reading · Deploy Prometheus on Kubernetes with Helm · Prometheus monitoring guide

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting