What Redis does in an ELK logging pipeline

Redis is an optional buffer between a shipper and Logstash. It can absorb a temporary burst, hold events during a downstream interruption, and separate collection from processing. It does not increase Elasticsearch's sustained indexing capacity or guarantee that every event reaches it.

A common supported route is:

Filebeat -> Redis list -> Logstash -> Elasticsearch -> Kibana

Match the producer and consumer

A minimal local Filebeat output is:

output.redis:
  hosts: ["localhost:6379"]
  key: "logs"
  datatype: list

Disable any other Filebeat output when using this one. The corresponding Logstash input is:

input {
  redis {
    host => "localhost"
    key => "logs"
    data_type => "list"
    codec => json
  }
}

The setting is datatype in Filebeat and data_type in Logstash. Configure authentication, network access, and supported transport security for the actual environment rather than exposing a local demonstration unchanged.

Multiple Logstash consumers of the same list share work; each does not receive a complete copy. The standard Logstash Redis input supports lists and Pub/Sub channels, not Redis Streams consumer groups. Using Streams requires an integration that explicitly supports its commands and acknowledgment model.

Understand the buffer's limits

When arrival rate briefly exceeds processing rate, the list grows. Processing catches up only if it later exceeds the arrival rate. Size the queue for the intended outage window and monitor its age, length, memory consumption, persistence, and errors.

With no persistence, a Redis restart loses the in-memory backlog. RDB snapshots can lose writes since the last snapshot. AOF narrows the recovery window according to its fsync policy, but does not remove every failure scenario.

There is also a handoff risk: a list item is removed when consumed, before Elasticsearch necessarily accepts it. A consumer crash in between can lose that event. A Logstash persistent queue can reduce some downstream risks, but it does not turn Redis list consumption into an end-to-end acknowledgment protocol.

When memory fills, eviction policies can remove data; noeviction instead rejects affected writes, which shifts recovery responsibility to the producer. Redis is therefore a bounded buffer with a defined failure policy, not an unlimited outage store.

When another design fits better

Use Kafka when retained replay and independent consumer groups are central requirements, while configuring its acknowledgments, replication, and retention deliberately. For a small deployment, Filebeat can often send directly to Logstash or Elasticsearch. Its backpressure and queues still depend on source-file retention and available disk or memory.

Add Redis when its buffering role solves a measured problem and the delivery tradeoffs are acceptable, not simply because older ELK diagrams include it.

References: Filebeat Redis output, Logstash Redis input, and Redis persistence.