Redpanda vs. Kafka for log buffering: capacity, durability, and cost

A queue can keep accepting logs while the analysis backend is unavailable. Choosing Kafka or Redpanda starts with a less glamorous question: does the organization already operate a messaging platform with an owner, capacity plan, and recovery procedure?

If a reliable Kafka service already exists, adding a log topic may require less work than deploying another cluster. If neither platform is in place, compare the full operating cost of both. And if the problem is a brief network interruption with one destination, a collector's persistent buffer may be enough.

The queue buys time; it does not create downstream capacity. New logs keep arriving during an outage, and the backend must process more than the incoming rate after recovery if the backlog is to shrink.

Published by Guance. This comparison uses Kafka 4.3 configuration documentation and Redpanda's current streaming documentation, reviewed October 14, 2026. The capacity examples are arithmetic, not measurements from a Kafka or Redpanda cluster.

Size the buffer and the recovery path

Assume logs enter the queue at 20 MiB/s, and the downstream backend cannot accept data for two hours. The accumulated message content is:

20 MiB/s × 7,200 s = 144,000 MiB = 140.625 GiB

With three replicas, that corresponds to approximately 421.875 GiB of replicated content, provided the input rate already describes the effective bytes stored in the broker log. If the measurement is uncompressed collector output, measure the compression effect first. Indexes, record overhead, existing retained data, partition imbalance, and free-space headroom still need to be added.

Now suppose the recovered backend accepts 30 MiB/s, while new logs continue arriving at 20 MiB/s:

Net drain rate = 30 MiB/s − 20 MiB/s = 10 MiB/s
Catch-up time = 144,000 MiB ÷ 10 MiB/s = 14,400 s = 4 hours

The system will operate with a backlog for another four hours. That extra ingestion may also compete with normal queries for the analysis backend's resources. A recovery plan therefore needs both broker storage and sustained downstream headroom.

Retention needs a separate calculation. Kafka's time and size policies can delete eligible segments regardless of whether a consumer has processed them. A retention window equal to the two-hour outage leaves little margin for delayed restart, a slow partition, or another interruption. In a steady FIFO recovery faster than arrivals, the age of the oldest unprocessed record decreases; you do not automatically need six hours of retention merely because outage plus drain time is six hours. Retaining the entire incident window for a later replay is a different requirement. Kafka topic settings

If the backend has no catch-up headroom, increasing its capacity or reducing low-value input is more useful than indefinitely enlarging the queue. Use a local collector buffer when its durability and replay window meet the requirement. Add a replicated message service when you need protection beyond one node or independently progressing consumers.

Compare acknowledgment and persistence together

Under Kafka's conventional in-sync replica (ISR) model, acks=all waits for the current ISR to acknowledge the write. min.insync.replicas sets the minimum ISR size that permits that write; it is not simply the number of acknowledgments requested.

For example, with replication factor three and min.insync.replicas=2, all three replicas acknowledge when all three remain in the ISR. If only one is in sync, an acks=all write fails. Eligible Leader Replicas (ELR) changes relevant semantics, so verify whether it is enabled before applying this explanation to a particular cluster. Kafka min.insync.replicas

Redpanda documents a different persistence path: by default, an acks=all response waits for replication and disk synchronization on a majority of the partition's replicas. Enabling write caching can allow acknowledgment before the disk synchronization completes; cluster and topic settings control the flushing behavior. Redpanda producer configuration

Compare replication factor, acknowledgment policy, flush settings, storage behavior, and failure assumptions together. A process restart, loss of one machine, and simultaneous power loss across replicas are different tests. Matching the name of the acks setting is not enough to establish equivalent durability or performance.

Broker acknowledgment also covers only the producer-to-queue portion of the path. If a consumer commits its offset before the log platform has durably accepted the data, a crash can lose the opportunity to retry. If it commits after acceptance, a crash between acceptance and commit can produce a duplicate on replay. Establish acknowledgment and retry behavior at every boundary, and decide how the destination handles duplicates. More replicas do not fix an incorrect offset-commit order.

Find the busiest partition before adding consumers

In a conventional Kafka consumer group, a partition is assigned to one consumer at a time. Adding consumers beyond the partition count does not automatically increase that group's processing parallelism. Adding partitions also has an operating cost. This discussion concerns conventional consumer-group assignment, not newer queue-style sharing models. Kafka consumer design

A message key affects that distribution. With a consistent keyed partitioner and unchanged partition count, using service as the key keeps records for one service together. That can preserve useful per-key order, but one high-volume service can overwhelm its partition while other partitions remain lightly loaded.

A more distributed key can spread the work while changing the ordering guarantee. Decide whether downstream processing actually needs service-wide order. Many logging workflows can tolerate records arriving out of order if timestamps and identifiers are preserved; other consumers may depend on a particular sequence.

Size-based retention has the same imbalance problem. Kafka's retention.bytes limit applies per partition. A setting of 20 GiB across 12 partitions suggests a nominal 240 GiB of logical log capacity, but a hot partition receiving 10 MiB/s accumulates 20 GiB in about 34 minutes:

20 GiB × 1,024 MiB/GiB ÷ 10 MiB/s
  = 2,048 seconds
  ≈ 34.1 minutes

Unused space in another partition does not extend that hot partition's retention allowance. Segment boundaries and cleanup behavior affect the actual deletion time, so this is a risk estimate, not a precise expiration timer.

Changing brokers does not automatically repair a skewed key distribution. Check the most heavily loaded partitions, their consumer lag, and the retention budget before deciding a migration is necessary.

Test the client features you actually use

Redpanda supports the Kafka protocol, which can preserve much of an existing client investment. A successful test that produces and consumes one message demonstrates only a small part of compatibility.

The current Redpanda compatibility reference lists concrete exceptions. For example, one SASL user cannot simultaneously have both SCRAM-SHA-256 and SCRAM-SHA-512 credentials, and Kafka's request_percentage quota is not supported. A small, dedicated log pipeline may not depend on either feature. Shared-platform authentication and administration tools might. Redpanda Kafka compatibility

Inventory client versions, authentication, ACL automation, Connect plugins, administrative tools, transactions, and retry behavior. Exercise the integration during an unavailable broker, a rebalance, and a downstream outage as well as under steady traffic. A multi-purpose messaging platform requires a wider test than an isolated logging topic.

Do not compare a new Kafka deployment with an obsolete ZooKeeper architecture. Kafka 4.x uses KRaft and no longer supports ZooKeeper mode. Migrating an older cluster is a separate project whose work depends on the starting version and state. Kafka 4.0 upgrade guidance

Licensing belongs in the comparison too. Redpanda Community Edition is source-available under its Business Source License (BSL); this is different from Kafka's Apache licensing. Enterprise features and the proposed deployment model may change the license required. Check the terms for the exact release and use case rather than treating all downloadable software as equivalent. Redpanda licensing

Decide whether an independent queue earns its place

A team with an existing Kafka platform can reuse its people, tools, and recovery knowledge. Starting a Redpanda cluster adds another system to run. For a team with no messaging platform, both options begin with the full deployment and ownership cost; the answer may be different.

Include the analysis backend in the budget. In the earlier example, clearing the backlog in four hours requires sustained processing above the arrival rate. Temporary scale-out and permanently reserved capacity are different ways to pay for that requirement, not ways to avoid it.

Revisit the queue when connecting to Guance

DataKit supports direct log collection and a KafkaMQ integration. If the queue still serves replay or multiple consumers, it can remain while you change the downstream collection path. Check the KafkaMQ message formats, parsing, authentication, and consumption configuration against the actual topic. Kafka protocol compatibility alone is not proof that a Redpanda-to-DataKit deployment has passed recovery testing. See DataKit logging and KafkaMQ.

If the queue was introduced only to compensate for an unreliable self-managed write endpoint, reassess it after replacing that endpoint. Test whether direct collection meets peak and recovery requirements. Removing a queue can retire replica storage, upgrades, monitoring, and on-call work, but only if its other responsibilities have disappeared too.

Keep the queue when it still provides replay, independent consumers, or a required durability boundary. Consider direct collection when it meets the same acceptance criteria with fewer components. That decision should come from measured delivery and recovery behavior, not a general claim that one broker is faster or cheaper.

The calculations here do not measure compression, CPU, disk performance, or throughput for either product. For the original series' calculation notes, see capacity calculations and validation scope (Chinese).