Configure a Logstash persistent queue

A Logstash persistent queue stores events on local disk between the input and filter stages. It can recover queued work after a restart and absorb a downstream outage, but it does not guarantee that every event survives every failure.

Enable the queue

In logstash.yml:

queue.type: persisted
path.queue: /var/lib/logstash/queue
queue.max_bytes: 8gb
queue.checkpoint.writes: 1024

Ensure the Logstash user can write to the directory and that the disk has enough space. With multiple pipelines, account for each pipeline's queue capacity rather than treating queue.max_bytes as a process-wide disk budget.

The processing path is:

input -> persistent queue -> filters -> outputs -> acknowledgment

An event is acknowledged after its filter and output processing completes. A crash can replay work whose output succeeded before its acknowledgment was persisted, so downstream consumers must tolerate duplicates. Disk space is reclaimed at page granularity after all events on a page are acknowledged.

Plan for the outage you need to absorb

At 10,000 events per second and 500 bytes per raw event:

10,000 x 500 x 3,600 = 18,000,000,000 bytes per hour
10,000 x 500 x 86,400 = 432,000,000,000 bytes per day

That is about 18 GB per hour or 432 GB per day in decimal units, before queue serialization overhead and headroom. Measure serialized event sizes; the raw payload is not the queue's exact disk footprint. The illustrative 8 GB configuration is smaller than this workload's one-hour requirement.

When the queue fills, inputs experience backpressure. Acknowledgment-capable senders such as Beats can retry according to their own buffering and retry behavior. A UDP sender has no equivalent end-to-end acknowledgment, so congestion can still lose data.

Set durability expectations

Events not yet covered by a checkpoint can be lost in a crash or hardware failure. queue.checkpoint.writes: 1 forces a checkpoint after every write for stronger durability at a potentially substantial I/O cost. It does not replicate the disk or protect against losing the host's storage.

Use a supported local filesystem; NFS is not supported for persistent queues. Benchmark the actual workload and storage rather than relying on a universal 10–20% throughput penalty. Faster storage and batching affect performance, but the durability settings remain part of that trade-off.

Drain and verify pending work before switching queue modes or replacing the queue's storage. A dead-letter queue serves a different purpose: it holds certain processing failures for later investigation rather than acting as this input buffer.

References

• Elastic: persistent queues