Kafka Connect, Filebeat, and Logstash: different jobs in a data pipeline

Kafka Connect integrates Kafka with external systems. Filebeat collects and forwards events, especially logs from files. Logstash processes events from multiple inputs and sends them to one or more destinations. They often belong in the same architecture, but each should have a clear job.

Compare responsibilities

Connect is a framework: a source connector imports data into Kafka, while a sink connector exports it. “Bidirectional integration” does not mean one connector automatically synchronizes both ways or resolves conflicting updates.

Filebeat is commonly deployed as an Agent on hosts or alongside container log collection. Logstash and Connect run on the JVM, but language alone does not establish resource costs. Measure the connectors, filters, event sizes, and queue settings you actually need.

Match the tool to the task

Use a suitable Kafka Connect source connector, such as a Debezium connector, for database change data capture. Use a compatible sink connector for Kafka-to-Elasticsearch or object storage. Availability, licensing, and delivery semantics depend on the connector; these connectors are not all bundled with Kafka.

Use Filebeat to follow log files and send events to Elasticsearch, Logstash, or Kafka. Its Kafka output is built in. Use Logstash when you need richer parsing, enrichment, or conditional delivery to multiple destinations.

A possible arrangement is:

Application logs -> Filebeat -> Kafka -> Connect sink -> Storage
                                    -> Logstash -> Elasticsearch

Give separate downstream applications their own consumer groups when both must receive the full stream. Kafka lets consumers work at different rates within its retention and capacity limits; it does not create unlimited buffering.

Can one replace another?

Connect can replace a processing service when the sink connector and SMTs handle the required work. Complex text parsing may fit Logstash or a stream-processing application better. Filebeat can often send structured logs directly to Elasticsearch without either component.

A small deployment does not need Kafka simply because it collects logs. Filebeat-to-Elasticsearch, Filebeat-to-Logstash, or DataKit-to-Guance may be sufficient. Evaluate peak backlog, outage duration, replay requirements, and operational ownership before adding a broker.

References: Kafka Connect and Filebeat's Kafka output.