Fluentd vs Logstash: what happens when the destination stops

The logging backend is unavailable for twenty minutes while applications keep writing. Once it returns, the on-call engineer wants three answers: was the data retained, when will the backlog clear, and did retries create duplicates? Normal throughput becomes meaningful in that context.

Fluentd and Logstash both receive, transform and forward events. Neither alone replaces a storage, search and alerting platform. The choice is usually about existing plugins, transformation logic and the part of the delivery path that must survive a downstream failure—not a broad claim about being lightweight or powerful.

Guance Editorial Team · Documentation reviewed October 21, 2026. Capacity examples are arithmetic illustrations, not measured throughput or recovery benchmarks.

Follow an event through processing

A Fluentd event has a tag, time and record. Inputs create events, filters process them, and matching rules route them to outputs. Tags can drive routing, but a consistent business taxonomy still requires agreement across sources. Event lifecycle.

Logstash organizes processing into inputs, filters and outputs, with codecs available at input and output boundaries. A pipeline might decode text, parse a timestamp, convert field types and deliver to a destination. Elasticsearch is not its only possible output. Processing model.

Both models cover many common jobs. Replacing established Logstash filters means revalidating dates, nulls, types and failure handling, not merely translating syntax. Likewise, a working Fluentd deployment does not need rewriting simply because its destination is Elasticsearch.

Use records that have caused trouble: timestamps with offsets, multiline exceptions, JSON fields that sometimes contain arrays, and unparseable raw text. Establish equivalent meaning before comparing resource use. Otherwise the faster pipeline may simply be delivering different data.

Buffer placement defines its protection

Fluentd buffering belongs to outputs that support it. Events are managed in chunks using memory or file buffers; not every output mode follows the same path. Defaults can depend on the output plugin. File buffering is appropriate when data must survive process restarts, provided its path and underlying storage persist. Buffer configuration, output modes.

Logstash's PQ sits between input and filtering/output and is disabled by default. Checkpoint gaps can lose data during abnormal shutdown, its local disk is not replicated, and inputs without acknowledgments do not gain end-to-end protection merely by enabling it. Persistent queues.

Ask what each acknowledgment establishes. An application writing a file, a collector reading it, a receiver accepting a request and a search backend exposing a record are different milestones. Adding a local queue strengthens part of that path, not every failure boundary.

Permanent loss of a collector machine may require rereadable upstream data, an external replicated queue or another recovery mechanism. Moving from memory to disk changes which failures are covered; it does not create unlimited durability.

ACKs and retries change the delivery contract

Fluentd's out_forward defaults to at-most-once. Enabling require_ack_response true makes it wait for the destination in_forward acknowledgment and provides at-least-once behavior. That acknowledgment is from the next hop, not a blanket statement that the final search system has durably committed the record. Forward output.

At-least-once also requires thinking about duplicates. A receiver may finish its work while its response is lost, prompting another send. Duplicate logs become a business problem if someone uses event counts as transaction counts. Decide whether stable event IDs and deduplication are necessary, and where they belong.

Retry exhaustion has version-specific behavior. From Fluentd v1.19.0, corresponding to fluent-package v6, built-in file buffers evacuate queued chunk files to a recovery directory before clearing the queue. Earlier versions discarded them in that situation. Memory buffers do not offer this file recovery, and third-party buffers need implementation support. Include the recovery path, disk capacity and restoration procedure in operations. Buffers and retries.

Logstash PQ recovery can also replay events not acknowledged as complete. More frequent checkpoints improve durability at a disk-write cost. Choose settings against actual event size, storage and destination behavior rather than treating a copied setting as a free reliability guarantee. PQ durability.

A full buffer sends pressure somewhere

Fluentd offers overflow policies that raise an exception, block or discard the oldest chunk; raising an exception is the default. The outcome still depends on whether the input can pause, retry or reread its source. Blocking a collector does not make a remote UDP sender stop. Overflow behavior.

A full Logstash PQ applies backpressure to inputs, with protocol-specific consequences. Monitor queue age and remaining capacity, not just process availability: an error delivered half an hour late may be intact but useless for timely alerting. PQ backpressure.

At a raw arrival rate of 10 MB/s, twenty minutes without output produces 12 GB using decimal units. That excludes serialization, metadata, file layout, peak headroom and other pipelines. A 12 GB queue setting is not a validated capacity plan.

Recovery needs spare throughput too. If arrivals remain at 10 MB/s and the destination accepts 15 MB/s on the same measurement basis, the net drain rate is 5 MB/s. Clearing 12 GB would ideally take another forty minutes. At only 10 MB/s of output, the backlog never shrinks. These calculations describe a capacity constraint, not either tool's expected performance.

Choose plugins and runtimes by version

Fluentd uses C and Ruby, with a plugin ecosystem involving Ruby gem dependencies. Fluent Bit is a related but separate project that can run independently or alongside Fluentd. It is not a smaller Fluentd into which arbitrary configuration can be copied. The projects' overview explains positioning; its simplified resource figures are not a capacity guarantee for a particular pipeline. Fluentd and Fluent Bit.

Logstash runs on the JVM, but its plugins are not restricted to Ruby: Elastic documents a Java plugin API. Evaluate the plugin's supported versions, maintenance, field semantics and failure behavior before sizing runtime resources. Java plugins.

Licensing also has scope. Fluentd's main project is Apache 2.0. Logstash's repository describes Apache 2.0 outside x-pack, subject to exceptions, and Elastic License within x-pack. Elasticsearch licensing changes should not simply be projected onto every Logstash source file, nor should every distribution be assumed to have identical terms. Fluentd license, Logstash license.

Make recovery understandable before replacing the pipeline

An established Logstash deployment should first validate its PQ, destination responses and recovery. An established Fluentd deployment should clarify file buffering, ACKs and overflow policy. Runtime labels and plugin counts cannot substitute for that work.

A migration needs a concrete benefit, such as removing difficult transformations or supporting a new source protocol. Acceptance should explain the result of a destination outage, collector restart, exhausted buffer and retry—not merely a successful normal send.

Guance ingestion has the same boundaries

When evaluating a backend as well, Guance DataKit can collect logs and use Pipeline processing for fields. Compare transformed records and query results using the same samples before migrating. Collection and backend storage are separate responsibilities; a managed backend does not remove source-file, network or delivery-failure handling. Guance log collection.