Fluentd vs. Kafka: where each belongs in a logging pipeline

Fluentd collects, parses, and routes events. Kafka stores event streams in partitioned topics so independent consumers can read them at their own pace. A logging system may use both, but adding Kafka is useful only when you need the buffering, replay, or consumer separation it provides.

Compare their responsibilities

Neither has a meaningful universal events-per-second figure. Message size, parsing, compression, acknowledgments, and storage durability change the workload substantially.

When Fluentd alone is enough

If you need to collect logs and send them to Elasticsearch or object storage, a Fluentd pipeline with appropriately sized file buffers may be sufficient. Set limits and an overflow policy, monitor queued data, and test recovery after a destination outage. A disk buffer helps with process restarts but is not unlimited storage or an unconditional no-loss guarantee.

When to add Kafka

Kafka is useful when several teams consume the same events independently, when you need replay after a processing change, or when producers must continue through a longer downstream outage. Retention and disk capacity must cover the intended recovery window. Consumers still need enough spare throughput to catch up.

One possible architecture is:

Applications → Fluent Bit → Kafka → Logstash → Elasticsearch → Kibana

Fluentd can fill the collection or processing role instead, using a compatible Kafka plugin. Fluent Bit may suit node-level collection when its supported inputs and transformations meet the requirement. These are architecture choices, not interchangeable configurations.

FAQ

Will a crash lose logs?

That depends on the whole path. Fluentd file buffers need persistent, healthy storage and compatible retry/overflow settings. Kafka durability depends on replication and producer acknowledgment settings, among other factors. Uncollected application logs, ephemeral disks, and full queues remain possible loss points. Test the failure cases your service must tolerate.

Does Kafka replace Logstash parsing?

Kafka brokers do not automatically parse application logs into a search schema. Kafka Streams, Flink, or other consumers can implement processing, but that introduces application logic to operate. Use Logstash or another processor when its parsing model fits your workload.

References: Kafka concepts and Fluentd buffering.