Why use Logstash when Elasticsearch has a REST API?

You can write directly to Elasticsearch through its REST API. Logstash is useful when you want a separate service to collect, transform, buffer, or route events. The decision depends on where those responsibilities should live and who will operate them.

Direct ingestion is not limited to storing untouched JSON: application code can transform events, and Elasticsearch ingest pipelines can parse and enrich documents before indexing.

What Logstash adds

Parsing and enrichment. Grok, date, mutate, GeoIP, and other filters can turn source-specific records into a consistent event schema. Structured fields enable targeted queries and aggregations. Elasticsearch ingest processors and other collectors can also perform some of this work, so Logstash is one implementation choice rather than a universal requirement.

Multiple inputs. File, network, message-broker, database, and HTTP plugins let one processing layer accept different sources. Each plugin has its own event-framing and acknowledgment behavior; a UDP input cannot offer the same recovery guarantees as an acknowledged input.

Buffering. A persistent queue or external broker can absorb an outage or burst while capacity remains. Logstash's default memory queue is not durable. Queue size, checkpointing, producer retries, and source-file retention determine what happens when a downstream service stays unavailable.

Routing and multiple outputs. Conditional branches can send events to different indices, clusters, or archive destinations. Consider whether a slow output can stall other work in the same pipeline and whether separate pipelines are warranted.

When direct ingestion fits

Direct ingestion can work for structured events with a manageable number of producers and a clear client-side recovery strategy. Low traffic makes capacity planning simpler, but high-volume direct ingestion is also possible with a properly designed bulk client.

A robust client must inspect individual bulk-item results, retry transient failures with backoff, handle permanent mapping failures, and consider duplicate writes. An HTTP 200 response from the bulk endpoint does not mean every item succeeded. A durable local queue may be needed if losing events during an outage is unacceptable.

Revisit the bottleneck before scaling

Filebeat can collect and perform useful parsing and processing without Logstash. If Logstash is required and becomes slow, measure filter cost, output latency, worker utilization, and queue growth. Scale workers or instances where appropriate; increasing batch size can also increase memory use and latency. A broker buffers excess traffic but does not raise Elasticsearch's sustainable indexing capacity.

References: Elasticsearch ingest pipelines and Logstash output retry behavior.