Collect Docker application logs with Filebeat and Logstash

A common pipeline is application stdout/stderr → Docker's configured log driver → Filebeat → Logstash → Elasticsearch. The file-harvesting example below specifically assumes a Linux Docker host using json-file; it is not a universal path for every driver or Docker Desktop installation.

Applications can also write files on a mounted volume when required. Give those files explicit collection and rotation rules rather than expecting Docker's stdout driver to discover them.

Configure a container input through autodiscovery

For a compatible modern Filebeat version, an example filebeat.yml is:

filebeat.autodiscover:
  providers:
    - type: docker
      host: unix:///var/run/docker.sock
      hints.enabled: true
      hints.default_config:
        type: filestream
        id: docker-${data.container.id}
        parsers:
          - container:
              stream: all
              format: docker
        paths:
          - /var/lib/docker/containers/${data.container.id}/*-json.log

processors:
  - add_docker_metadata:
      host: unix:///var/run/docker.sock

output.logstash:
  hosts: ["logstash:5044"]

Use the actual Docker data-root and an address reachable from Filebeat. This is a private-network example without TLS; configure transport authentication and encryption for the intended environment. Disable a conflicting Elasticsearch output when selecting Logstash output.

Persist Filebeat's data directory so its registry survives recreation. If Filebeat runs in a container, it needs read access to the selected host log paths and access to Docker metadata. Mounting the Docker socket read-only does not make the Docker API read-only; grant that access only to a trusted collector or use a restricted API proxy.

Docker documents its internal json-file files as daemon-managed and cautions against external interference. Do not modify them. Where host-file harvesting is an accepted deployment choice, validate it against the selected Docker/Filebeat versions; a supported external log destination can avoid relying on Docker's internal storage layout.

Receive events in Logstash

A minimal pipeline for checking delivery is:

input {
  beats { port => 5044 }
}
output {
  stdout { codec => rubydebug }
}

Replace the diagnostic output with the intended authenticated Elasticsearch output after validating events. Configure TLS on the Beats connection when needed. Filebeat index templates and ingest assets may require separate setup when events pass through Logstash.

Combine multiline events at the source

Docker label hints such as co.elastic.logs/multiline.pattern are interpreted by Filebeat, not automatically by Logstash. Configure multiline handling in the individual input before streams from different containers are mixed. Do not attach the multiline codec to a Logstash Beats input: that input rejects it, and Logstash fails to start. Handling multiline records in Filebeat also avoids combining unrelated sources.

Prefer one JSON event per line when the application supports it, with an exception stack encoded as a string. This also avoids guessing event boundaries from indentation.

FAQ

How do I prevent unlimited local logs?

For new json-file containers, set max-size and max-file, for example 10m and 3. Existing containers need recreation to adopt changed daemon defaults. Check collector lag before reducing retention.

Can this run as a DaemonSet?

A DaemonSet is a Kubernetes workload, not a deployment method for an ordinary Docker host. Kubernetes commonly exposes CRI log files under /var/log/containers; use its autodiscovery provider, parser, permissions, and runtime paths instead of copying the Docker configuration unchanged.

How do I validate the pipeline?

Run Filebeat's configuration and output tests, then emit a uniquely identifiable event and a multiline exception from two containers. Confirm separation, metadata, timestamps, restart behavior, and rotation handling. DataKit logging guidance describes an alternative collector path when Guance is the destination.

References: Filebeat autodiscovery hints, multiline handling, Logstash output, Beats input multiline restrictions.