Kafka Monitoring: The Complete 2026 Guide (Consumer Lag, JMX Metrics, Alerts)

Short answer: Kafka monitoring comes down to one user-facing truth and three broker-health layers. The truth: consumer lag — the gap between what producers wrote and what consumers read — is the single metric your users feel. The layers: brokers (under-replicated partitions, offline partitions, request handler idle ratio), JVM/OS (GC, disk), and producers/consumers (send failures, commit rates, rebalances). Instrument via JMX → Prometheus exporter or a platform agent, alert on lag growth rate rather than absolute lag, and never let UnderReplicatedPartitions stay non-zero overnight. Kafka 3.x/4.x (KRaft mode — ZooKeeper is gone) assumed throughout.

The metric that matters: consumer lag

Lag = log-end-offset − current-offset, per partition, summed per consumer group. Everything else in Kafka monitoring exists to explain why lag is growing.

# Built-in CLI — fine for ad-hoc, useless for alerting
bin/kafka-consumer-groups.sh --bootstrap-server b1:9092 --describe --group orders-consumer

# The output that matters: LAG column per partition
# GROUP           TOPIC     PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders-consumer orders    0          154832          154900          68

For monitoring you need lag as a time series (CLI snapshots don't alert). Options: Kafka Lag Exporter (Lightbend-origin, Prometheus-native), Burrow (LinkedIn, evaluates lag over windows — smarter about flapping), or a platform agent that reads __consumer_offsets directly (Guance DataKit, Datadog, Confluent). Our dedicated consumer-lag guide covers lag-per-second math and auto-remediation patterns.

Alert on lag growth rate, not absolute lag — 10K messages of lag is fine for a batch consumer draining 5K/min, catastrophic for a payment stream:

# Lag growing for 30 minutes = real problem regardless of absolute value
deriv(kafka_consumergroup_lag{group="orders-consumer"}[30m]) > 0

Broker health: the three layers

Layer 1 — Partitions (the cluster's vital signs):

Related guideKafka Integration

Metric	Healthy	Alert
`UnderReplicatedPartitions`	0	> 0 for 10 min (page)
`OfflinePartitionsCount`	0	> 0 ever (page immediately)
`ActiveControllerCount`	exactly 1 per cluster	≠ 1 (split brain)
`UncleanLeaderElectionsPerSec`	0	> 0 (data-loss risk accepted)

Layer 2 — Request pipeline (find saturation before users do):

Metric	Healthy	Watch
`RequestHandlerAvgIdlePercent`	> 0.3	< 0.2 = broker saturating
`NetworkProcessorAvgIdlePercent`	> 0.3	< 0.2 = I/O bound
`RequestQueueTimeMs` p99	low ms	rising queue = backpressure
`TotalTimeMs` p99 (Produce/Fetch)	stable baseline	drift = investigate

Layer 3 — JVM & OS: GC pause time (G1 pauses > 200 ms stall produce latency), open file descriptors (segment files multiply fast), and disk — Kafka's write model means a full disk is a cascading partition failure. Alert at 70% disk usage, not 90%.

Collection: JMX → Prometheus

Kafka exposes everything via JMX. The standard bridge:

# jmx_exporter config fragment (broker sidecar or javaagent)
lowercaseOutputName: true
rules:
  - pattern: kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value
    name: kafka_server_replicamanager_underreplicatedpartitions
  - pattern: kafka.server<type=BrokerTopicMetrics, name=BytesInPerSec><>Count
    name: kafka_server_brokertopicmetrics_bytesin_total

# Cluster throughput
sum(rate(kafka_server_brokertopicmetrics_bytesin_total[5m]))

# Log flush latency (disk health proxy)
histogram_quantile(0.99, rate(kafka_log_logflushstats_logflushtimeandcount_bucket[5m]))

Managed services abstract this away: MSK surfaces metrics to CloudWatch, Confluent Cloud has its own metrics API — and platforms like Guance collect broker JMX, consumer lag, and MSK/Confluent metrics into the same console as your application traces, which is what you want when "lag is growing" needs to become "the payments service deploy at 14:32 slowed deserialization."

The 6 failure paths (and their signatures)

Slow consumer cascade. One consumer instance stalls → its partitions' lag grows → rebalance makes it worse. Signature: lag concentrated in specific partitions; consumer group rebalance_rate elevated.
Disk fills, partitions go offline. Signature: OfflinePartitionsCount > 0, disk > 90%. Recovery: add disk or reduce retention, restart broker, wait for ISR catch-up.
ISR thrash from a flaky broker. UnderReplicatedPartitions oscillates; leader elections spike. Usually network or GC pauses on one broker.
Retention misconfiguration. retention.ms too long × traffic growth = the disk path above; too short = consumers lose data they hadn't read. Lag alerting is your safety net for the latter.
Hot partitions. One partition key dominates → one broker's disk/network saturates while others idle. Signature: per-partition byte rates skewed > 3×.
GC pauses masquerading as broker failure. Long G1 pauses trigger exactly the ISR thrash pattern above. Correlate GC metrics before blaming the network.

FAQ

Q: What is the most important Kafka metric?
Consumer lag — the gap between produced and consumed offsets per consumer group. It is the only metric your users directly experience (as processing delay). Alert on lag growth rate sustained over 15–30 minutes rather than absolute values, and track lag in time-units (estimated seconds behind) for business relevance.

Q: How do I monitor Kafka consumer lag?
For ad-hoc checks: kafka-consumer-groups.sh --describe. For monitoring: run Kafka Lag Exporter or Burrow and alert on the derived time series, or use a platform (Confluent, Datadog, Guance) that reads __consumer_offsets natively. Our consumer-lag guide covers growth-rate alerting and drain-time math in detail.

Q: What does UnderReplicatedPartitions mean?
The count of partitions whose in-sync replicas have fallen behind the leader — a durability warning: if the leader fails now, you may lose data or availability. It should be 0; transient blips during broker restarts are normal, but anything sustained means a broker is unhealthy, undersized, or network-impaired.

Q: Is ZooKeeper still needed for Kafka monitoring?
No — Kafka 3.3+ runs production-grade KRaft mode, and Kafka 4.x removes ZooKeeper entirely. Monitoring shifts accordingly: watch controller metrics (ActiveControllerCount, KRaft metadata lag) instead of ZooKeeper ensemble health.

Q: Prometheus vs Confluent vs platform tools for Kafka monitoring?
Prometheus + JMX exporter + Grafana is the free, full-control baseline. Confluent Control Center/Cloud metrics are the best Kafka-native UX but Kafka-only. Platforms (Datadog, Guance, New Relic) win when Kafka is one component of a stack and you need lag correlated with the consumer application's traces and deploys. Choose by where your incidents actually get diagnosed.

Sources: Apache Kafka documentation and KIP archives (KRaft), JMX exporter project docs, Burrow/Kafka Lag Exporter project docs. Verified 2026-08-07.