Monitor disk usage in Kubernetes persistent volumes

For supported mounted volumes, kubelet exposes kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes. Divide used bytes by capacity to calculate utilization. The metrics include namespace and persistentvolumeclaim, which connect a storage warning to the claim that needs attention.

Scrape kubelet metrics

The following is a starting point for an in-cluster Prometheus deployment:

scrape_configs:
  - job_name: kubelet
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node

The service account needs discovery permissions and authorization to read the kubelet metrics endpoint. Network access and a serving certificate valid for the discovered address are also required. The cluster CA shown here must actually trust that certificate; do not resolve a trust failure by disabling certificate validation.

If you use kube-prometheus-stack, first inspect its existing kubelet ServiceMonitor and installed values. Avoid adding a second scrape of the same endpoint.

Calculate utilization

100 * kubelet_volume_stats_used_bytes
  / kubelet_volume_stats_capacity_bytes

This matches corresponding series using their labels. Display the PVC and namespace in the legend. In a multi-cluster setup, keep the cluster label too. If a shared volume is reported by several nodes, do not sum repeated observations as if they were separate capacity.

Alert on sustained usage

groups:
  - name: volume-capacity
    rules:
      - alert: PVDiskAlmostFull
        expr: (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) > 0.85
        for: 10m
        annotations:
          summary: 'PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} is over 85% full'

An 85% threshold sustained for ten minutes is an example. Account for growth rate, expansion lead time, and free space required by the application.

Understand coverage limits

Volume statistics depend on the volume implementation and, for CSI volumes, driver support. Missing data is not zero usage. Node Exporter filesystem metrics can reveal pressure on the host filesystem behind a hostPath volume, but they do not isolate that directory's usage or provide a PVC-specific substitute.

FAQ

Why are the metrics absent? Check the kubelet scrape, the mounted volume, and driver support. With appropriate API authorization, inspect the node summary using kubectl get --raw /api/v1/nodes/NODE_NAME/proxy/stats/summary, replacing NODE_NAME. Compare this with the kubelet /metrics output.

Can I monitor inodes? For filesystems that report them, use kubelet_volume_stats_inodes_used / kubelet_volume_stats_inodes. Many small files can exhaust inodes while byte capacity remains available.

Sources

This guide is based on the documentation below; the examples have not been tested against a live deployment.

• Kubernetes metrics reference
• Prometheus scrape configuration