Redis or RabbitMQ as a buffer before Logstash?
Compare Redis lists and RabbitMQ queues by delivery guarantees, persistence, routing, and integration support instead of unsupported throughput rankings.
Choose the queue by the failure modes you need to tolerate and the clients you can actually connect. Redis lists offer a simple buffer. RabbitMQ provides broker-level routing and acknowledgment mechanisms. Neither makes a logging pipeline lossless merely by being present.
Compare the mechanisms
| Concern | Redis list | RabbitMQ queue |
|---|---|---|
| Storage | Memory-backed data structure, with optional RDB snapshots and AOF logging | Queue storage and replication depend on queue type and configuration |
| Consumption | A pop removes the item; no processing acknowledgment in the simple list workflow | Manual consumer acknowledgments and redelivery are available |
| Routing | Application-chosen keys; Pub/Sub is a separate, non-durable model | Exchanges, bindings, and routing keys |
| Recovery | Persistence policy, memory limits, and the consumer's handoff matter | Publisher confirms, queue durability, replication, and acknowledgment timing matter |
| Capacity | Measure memory and persistence overhead | Measure disk, replication, queue backlog, and consumer throughput |
RDB is snapshotting; AOF records writes. They are not the same persistence mechanism. With AOF everysec, recent accepted writes can still be lost in a crash. A list consumer can also crash after removing an item but before handing it to durable downstream storage.
RabbitMQ's publisher confirms and consumer acknowledgments cover different hops. Persistent messages, an appropriate durable queue, confirms, and carefully timed acknowledgments work together. A broker acknowledgment does not prove Elasticsearch has indexed the document. Retries can create duplicates, so consumers may need idempotent document IDs or deduplication.
Check the shipping path
Filebeat has a built-in Redis output, but no built-in RabbitMQ output. These are different architectures:
Filebeat -> Redis list -> Logstash -> Elasticsearch
AMQP-capable producer -> RabbitMQ -> Logstash -> Elasticsearch
A bridge or additional shipper is required if Filebeat must feed RabbitMQ. Inspect the Logstash input's acknowledgment and persistent-queue behavior; do not infer end-to-end processing guarantees from the broker alone.
Consider Kafka and simpler alternatives
Kafka is useful when multiple independent consumers need retention and replay. Its durability depends on producer acknowledgments, replication, in-sync replica settings, and storage, while consumer recovery depends on offset commits. It also adds operational work.
For modest traffic and short outages, Filebeat can send directly to Logstash or Elasticsearch. Source-file retention, disk queues, and a Logstash persistent queue may meet the requirement without another broker.
There is no universal messages-per-second threshold at which RabbitMQ becomes too slow, or proof that Redis always wins. Benchmark your payloads, persistence settings, burst duration, and recovery workload. Monitor backlog age as well as queue length, and define what should happen when the buffer fills.
References: Redis persistence, RabbitMQ reliability, Filebeat outputs, and Logstash RabbitMQ input.
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan