Secure Prometheus metrics endpoints

Treat a metrics endpoint as an operational interface that needs an explicit access policy. Many exporters expose metrics without authentication unless configured otherwise. Labels, version strings, and business metrics can reveal sensitive information even when the endpoint is read-only.

Restrict the network first

Allow only the collectors that need to reach each metrics port. Use private listeners, host firewalls, and security groups as appropriate. For pod endpoints, use an enforced Kubernetes NetworkPolicy that selects the intended collector pods and destination ports.

A namespace name alone is not a security boundary. Check the namespace and pod selectors, the network plugin's policy support, and whether host networking changes enforcement. Kubelet normally runs on a node, so ordinary pod NetworkPolicy is not a sufficient control for port 10250.

Authenticate the endpoint

A reverse proxy can add authentication to an exporter. Inside an HTTPS-enabled Nginx server block:

location = /metrics {
    allow 10.0.0.0/8;
    deny all;
    auth_basic "metrics";
    auth_basic_user_file /etc/nginx/.htpasswd;
    proxy_pass http://127.0.0.1:9100;
}

Replace the example network with the narrow collector network. Ensure the exporter cannot be reached directly around the proxy. Basic authentication needs TLS to protect credentials in transit.

Configure the matching Prometheus scrape:

scrape_configs:
  - job_name: protected-node
    scheme: https
    static_configs:
      - targets: ['metrics.example.com:443']
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/secrets/metrics-password
    tls_config:
      ca_file: /etc/prometheus/certs/metrics-ca.crt

Use a trusted CA and a certificate valid for the target hostname. Keep the password file readable only by the service account that needs it.

Protect Prometheus itself

Prometheus supports TLS and basic authentication through a separate web configuration file. A TLS configuration can begin with:

tls_server_config:
  cert_file: /etc/prometheus/server.crt
  key_file: /etc/prometheus/server.key

Start Prometheus with --web.config.file=/etc/prometheus/web.yml. To add authentication, include a basic_auth_users mapping of usernames to generated bcrypt password hashes as described in the official basic-auth guide. The web server stores hashes; scrape credentials use the actual password or its file. Protect the UI and API as well as /metrics.

Limit what is exposed

Remove secrets and personal data from instrumentation itself. metric_relabel_configs can keep selected samples out of Prometheus storage, but cannot hide them from anyone who can already read the exporter's endpoint.

If DataKit performs collection, configure its endpoint, authentication, and TLS settings using the Prometheus collector guide. A localhost listener works only when collector and application share the relevant network namespace; containers or remote collectors need their own network-access design.

FAQ

Does Pushgateway need the same protection? Yes. It also accepts writes, so unauthorized access can inject misleading metrics.

How should I protect kubelet metrics? Require authenticated, authorized access, configure kubelet authentication and authorization appropriately, and restrict node-level network access. Grant the collector's service account only the permissions required by the chosen scrape path.

Sources

This guide is based on the documentation below; the examples have not been tested against a live deployment.

• Prometheus security model
• Prometheus HTTPS configuration
• Prometheus basic authentication
• DataKit Prometheus collection