Kubernetes Monitoring: The Complete 2026 Guide (Metrics, Tools, Alerts)

Short answer: Monitoring Kubernetes means watching four layers — cluster control plane, nodes, workloads (pods/containers), and applications — each with different metrics and tools. The practical baseline: kubectl top for ad-hoc checks, metrics-server for autoscaling data, Prometheus + kube-state-metrics for real monitoring, and an alert set covering the ten failure modes that actually take clusters down (CrashLoopBackOff, OOMKilled, pending pods, node pressure, API latency, and more). This guide gives you the layer-by-layer metric map, the tool decision, and copy-paste alert rules.

The four layers (and what "healthy" means at each)

Layer	Key metrics	Healthy looks like	Primary tool
Control plane	API server latency, etcd leader changes, scheduler pending pods	p99 API latency < 1s, etcd stable	Prometheus scrape of control-plane endpoints (managed K8s: cloud provider surfaces)
Nodes	CPU/memory/disk pressure, kubelet health	No `MemoryPressure`/`DiskPressure` conditions	node_exporter, cloud metrics
Workloads	Pod restarts, container throttling, request/limit usage	Restarts stable, throttling < 5%	metrics-server + kube-state-metrics + cAdvisor
Applications	RED (rate, errors, duration), saturation	SLO burn within budget	Prometheus client libs / OTel / eBPF

The layers fail differently: control-plane issues make everything unreachable (but your app may be fine); node pressure evicts pods; workload issues restart containers; application issues return errors while Kubernetes reports all green. Monitoring only one layer is how you get paged by users before your dashboards notice.

Built-in tools: what you get for free

kubectl top nodes                 # live CPU/memory per node (metrics-server required)
kubectl top pods -n prod --sort-by=memory
kubectl get events --sort-by=.lastTimestamp -n prod   # the incident timeline
kubectl describe pod <pod>        # restart reasons, probe failures, image pull errors

kubectl top reads from metrics-server, a lightweight aggregator that exists mainly to feed HPA/VPA autoscaling — it stores no history and exposes no query language. It answers "what is hot right now," never "what happened at 2 a.m." For anything beyond that, you need real metrics collection.

Related guideKubernetes Integration

The standard stack: Prometheus + kube-state-metrics

The production baseline in 2026 remains the kube-prometheus-stack Helm chart, which wires together:

Prometheus — scrapes cAdvisor (container CPU/memory via the kubelet), node_exporter, and app endpoints.
kube-state-metrics (KSM) — exposes object state: deployments, replicas, pod phases, restarts, resource requests/limits. The source for questions like "which pods are pending" or "who set no memory limit."
Alertmanager + Grafana — routing and visualization.

helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

Three KSM queries you will use weekly:

# Pods restarted in the last hour, by owner
sum by (namespace, pod) (increase(kube_pod_container_status_restarts_total[1h])) > 0

# Containers being CPU-throttled (limit too tight)
sum by (namespace, container) (rate(container_cpu_cfs_throttled_seconds_total[5m])) > 0.05

# Pods pending > 10 minutes (scheduler can't place them)
kube_pod_status_phase{phase="Pending"} == 1

The 10 alerts every cluster needs

Alert	Expression sketch	Why
CrashLoopBackOff	restarts increase > 3 in 15m	Most common workload failure
OOMKilled	container terminated reason = OOMKilled	Memory limit sizing
High throttle ratio	throttled/total periods > 5%	CPU limits too tight
Pending pods	phase=Pending for 10m	Capacity/scheduler failure
Node pressure	condition MemoryPressure/DiskPressure true	Eviction incoming
API server p99	apiserver request latency p99 > 1s	Control plane degradation
etcd leader flapping	leader changes > 1/h	Cluster brain instability
PVC near full	volume free < 15%	Stateful workload outage
Certificate expiry	cert expiry < 30d	The silent cluster killer
HPA at max replicas	current = max for 30m	Autoscaling ceiling hit

eBPF: the 2026 addition worth knowing

eBPF-based tools (Pixie, Cilium/Hubble, Beyla, groundcover) observe traffic and latency inside the kernel — no sidecars, no SDK changes, instant service maps. They are genuinely additive for "we instrumented nothing and need visibility now," but they complement rather than replace Prometheus metrics: you still need historical time series, custom app metrics, and SLO math. Budget for both.

Centralizing beyond one cluster

Single-cluster Prometheus becomes a liability when you run 5+ clusters or need multi-month retention: federation is fiddly, and per-cluster storage fragments incident analysis. The standard 2026 pattern is per-cluster collection (Prometheus agent mode or OpenTelemetry Collector) remote-writing to a central backend — Thanos/Mimir for self-managed, or a platform like Guance, which accepts Prometheus remote write and OTLP directly, correlates metrics with logs/traces in one console, and prices on usage rather than per-node. Keep your KSM dashboards and alert rules; change only where the data lands.

FAQ

Q: What is the best tool for monitoring Kubernetes?
For open-source: Prometheus + kube-state-metrics + Grafana (via kube-prometheus-stack) is the standard baseline. For managed platforms: Datadog, New Relic, Grafana Cloud, and Guance all offer Kubernetes integrations — compare them on per-node vs usage pricing, since per-node models compound in autoscaling clusters. eBPF tools like Pixie are complementary add-ons, not replacements.

Q: What is the difference between metrics-server and Prometheus?
metrics-server is a minimal, memory-only aggregator that feeds CPU/memory to the Horizontal Pod Autoscaler — no history, no query language, no alerting. Prometheus is a full monitoring database with PromQL, alerting, and retention. You need metrics-server for autoscaling and Prometheus (or an equivalent) for actual observability.

Q: Which Kubernetes metrics matter most?
At the workload layer: container restarts, CPU throttling ratio, and memory vs limits (OOMKill risk). At the node layer: memory/disk pressure conditions. At the control plane: API server p99 latency and etcd health. Application RED metrics sit on top. Alert on symptoms (restarts, pending, pressure) rather than raw utilization.

Q: How do I monitor multiple Kubernetes clusters?
Deploy lightweight collectors per cluster (Prometheus agent mode or OTel Collector) and remote-write to a central store — Thanos/Mimir self-managed, or a SaaS backend. Label every series with cluster and build dashboards that group by it. Avoid prometheus-to-prometheus federation for anything beyond trivial scale.

Q: How much does Kubernetes monitoring cost?
Self-managed OSS: infrastructure plus roughly 0.2–0.5 engineer at small scale. SaaS platforms: per-node pricing (Datadog ~$15/host infra + APM add-ons) compounds in autoscaling clusters — a 100-node cluster with bursts to 150 reprices at the 99th percentile. Usage-based platforms (Grafana Cloud, Guance) bill the data you send, which suits bursty fleets better.

Sources: Kubernetes and Prometheus official documentation, kube-state-metrics and kube-prometheus-stack project docs. Verified 2026-08-07. Published by Guance — the Kubernetes integration accepts Prometheus remote write and OTLP.