Kubernetes Monitoring Architecture: 4 Production Designs (2026)

Short answer: There are four Kubernetes monitoring architectures that actually work in production: (1) in-cluster Prometheus for one cluster and short retention, (2) Prometheus agent mode + remote write for centralization without federation pain, (3) hub-and-spoke multi-cluster (Thanos/Mimir or a platform backend) for fleet views, and (4) platform-native collection (DataKit, Datadog Agent, OTel Collector → SaaS) when you'd rather buy the storage layer than run it. The right choice depends on cluster count, retention requirements, and who owns the monitoring system's on-call. Below: each architecture's data flow, scaling limits, and failure modes.

The reference components (what every design is built from)

Every architecture below is a different answer to the same question: where does the data live, and who operates it?

Design 1: In-cluster Prometheus (the starter)

Pods/Nodes → kubelet(cAdvisor) ─┐
KSM ────────────────────────────┼→ Prometheus (in-cluster) → Grafana
node_exporter ──────────────────┘      ↓
                                  Alertmanager → PagerDuty/Slack

One Helm chart (kube-prometheus-stack), everything inside the cluster, 15-day local retention. Right for: 1–3 clusters, teams fine with short retention. Breaks when: you need 13-month capacity trends, the cluster dies and takes its own monitoring with it (a classic — your Prometheus pod is inside the blast radius), or a second cluster appears and dashboards fragment.

Related guideKubernetes Monitoring: The Complete 2026 Guide→

Design 2: Agent mode + remote write (the 2026 default)

Per cluster: kubelet/cAdvisor/KSM/node_exporter
      ↓ scrape
Prometheus agent mode (no TSDB, no rules — scrape & forward only)
      ↓ remote_write
Central backend: Mimir / Thanos Receive / Grafana Cloud / Guance
      ↓
Grafana (global), Alertmanager (central or local)

Prometheus agent mode strips local storage and query — the per-cluster footprint drops to a lightweight forwarder, and all retention/correlation happen centrally. Right for: most teams past 3 clusters; keeps all PromQL assets while centralizing storage. Watch: network egress cost on metrics volume, and central-backend cardinality governance (one bad label now poisons the global store).

Design 3: Hub-and-spoke multi-cluster (the fleet design)

Spoke clusters: collectors (agent mode / OTel DaemonSets) — labeled cluster=<name>
      ↓ remote_write
Hub: object storage + query layer (Thanos: Store Gateway + Querier + Compactor)
      ↓
Global Grafana, cross-cluster recording rules, tenant isolation

Thanos/Mimir-style designs add horizontal scaling (query and ingest scale independently), cheap object-store retention (13 months for the price of S3), and tenancy (platform teams give each product team a namespace of the telemetry world). Right for: platform-engineering orgs running 10+ clusters. Cost: you now operate a distributed database — this is a staffed platform, not a Helm chart.

Design 4: Platform-native collection (buy the hub)

Per cluster: vendor agent / DataKit / OTel Collector DaemonSet
      ↓ (vendor protocol or OTLP/remote write)
SaaS platform: storage, correlation, ML, dashboards, alerting

Related guideKubernetes Integration→

The decision matrix

Cross-cutting design decisions (whichever you pick)

1. Always label cluster, namespace, environment at collection time. Retrofitting identity into a central store is miserable.
2. Cardinality governance is architecture, not hygiene. A global store makes one team's bad label everyone's outage. Set per-namespace series limits at the collector.
3. Alert evaluation placement is a reliability decision. In-cluster Alertmanager fires even when the WAN is down; central alerting gives one noise-reduction point. Most mature setups do both: local paging alerts, central warning alerts.
4. The monitoring system needs monitoring. Meta-alerts on scrape failures, remote-write lag, and storage headroom — otherwise you discover your observability was blind during the incident.
5. Plan for eBPF as an addition, not a pillar. Kernel-level service maps (Cilium/Hubble, Pixie, Beyla) bolt onto any of the four designs for zero-instrumentation traffic visibility.

FAQ

Q: What is the best architecture for monitoring multiple Kubernetes clusters?
For 3–15 clusters: Prometheus agent mode (or OTel Collectors) remote-writing to one central backend — Mimir/Thanos self-managed or a SaaS platform. Beyond ~10 clusters with a platform team, a Thanos/Mimir hub-and-spoke with object-storage retention is the self-managed standard. In every case, label all series with cluster at collection time.

Q: Should Prometheus run inside or outside the Kubernetes cluster?
Both, layered: lightweight collectors inside each cluster (agent mode scrapes cAdvisor/KSM locally — you can't scrape a dead cluster's metrics from outside), storage and query outside (central backend). The naive "Prometheus inside the cluster it monitors" design loses observability exactly when the cluster dies.

Q: What is Prometheus agent mode?
A Prometheus build with the TSDB, rule evaluation, and query engine removed — it only scrapes and remote-writes. Memory footprint drops dramatically, and it becomes a pure per-cluster forwarder to a central store. It is the standard 2026 answer to "how do I centralize without federation."

Q: How does eBPF change Kubernetes monitoring architecture?
eBPF collectors (Cilium/Hubble, Pixie, Beyla) add kernel-level traffic maps and latency data with zero application changes — a DaemonSet addition to any design above. They complement rather than replace metrics collection: no custom app metrics, limited history, and no SLO math. Treat as the fourth signal source, not a new architecture.

Q: When should we buy a platform instead of running the hub?
When the on-call cost of running the monitoring system exceeds the vendor bill — typically when you lack a dedicated platform engineer, need multi-month retention without operating object storage, or want logs/traces/metrics correlated in one console. Compare per-host vs usage pricing against your fleet shape; keep collection on OTLP/remote-write so the decision stays reversible.

---

Sources: Kubernetes, Prometheus (agent mode), Thanos/Mimir, and OpenTelemetry project documentation. Verified 2026-08-07. Published by Guance — the Kubernetes integration accepts remote write and OTLP in Design 2/4 topologies.