Monitor custom Kubernetes Pod metrics with Prometheus

Expose application metrics, tell Prometheus how to discover the endpoint, and verify a known series. Pod annotations are only a convention: Kubernetes and Prometheus do not automatically start scraping merely because prometheus.io/scrape: "true" exists.

Expose and declare the metrics port

Instrument a counter such as orders_created_total and expose the registry on /metrics. Bind the endpoint to an address reachable from the scraper, not only to the application's loopback interface.

For the following discovery configuration, use one declared metrics port named metrics. Add these fields to the Pod template in your workload, retaining its existing containers and other configuration:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"
spec:
  containers:
    - name: app
      ports:
        - name: metrics
          containerPort: 8080
          protocol: TCP

This is a merge example, not a complete workload manifest: the real container still needs its image and application settings. In a Deployment, these fields belong under spec.template.

Configure Pod discovery explicitly

For an in-cluster Prometheus server using its service account:

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: 'true'
      - source_labels: [__meta_kubernetes_pod_phase]
        action: keep
        regex: Running
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        action: keep
        regex: metrics
      - source_labels: [__meta_kubernetes_pod_container_port_protocol]
        action: keep
        regex: TCP
      - source_labels: [__meta_kubernetes_pod_container_port_number]
        target_label: __meta_kubernetes_pod_annotation_prometheus_io_port
        action: keepequal
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        regex: '(.+)'
        target_label: __metrics_path__
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

This selects only Running Pods with the opt-in annotation and the declared TCP metrics port matching the port annotation. keepequal requires a supporting Prometheus version; validate with the server's matching promtool.

Pod discovery already constructs the address from the Pod IP and declared port. Keeping it avoids an IPv4-only address rewrite and prevents one annotated port from being scraped repeatedly through every discovered container port. Require one selected metrics endpoint per Pod for this convention. If the path annotation is absent, the (.+) rule does not replace the default /metrics with an empty string.

Grant the discovery service account the necessary Pod get/list/watch access in the namespaces being monitored. Also allow network access to the metrics port, configure authentication/TLS when required, and scope discovery to the intended namespaces. API discovery permission does not itself grant network access to application metrics.

Use PodMonitor with Prometheus Operator

If Prometheus Operator owns the deployment, a PodMonitor or ServiceMonitor is often a better fit than manually editing generated scrape configuration. A PodMonitor selects Pod labels and named ports; the Prometheus resource must also select that monitor and its namespace. Creating the resource alone is not proof it will be discovered. Avoid configuring both paths for the same endpoint unless duplicate scraping is intentional.

Verify collection and diagnose gaps

Check the Targets page for the final endpoint, scrape error, and labels. Generate a known application event, then query orders_created_total{namespace="your-namespace"} using the namespace you actually deployed. A discovered target is not proof that the expected metric is exposed.

If no target appears, check namespace scope, RBAC, Pod annotations, phase, declared port name/number, and relabel drops. If the target is down, check reachability, listener binding, TLS, authentication, and response format.

For Guance collection, configure DataKit's Prometheus exporter integration against a verified endpoint. Confirm the resulting metric name, namespace/service labels, and discovery scope with a known sample; a running DaemonSet does not prove all application metrics are collected.

Can these metrics drive HPA? Yes, with an appropriately configured adapter, such as Prometheus Adapter, exposing them through Kubernetes' custom or external metrics APIs. Prometheus collection alone does not create that API or define the scaling policy.

References

• Kubernetes Pod discovery and relabeling
• Prometheus Operator API
• Guance Prometheus exporter integration