Collect logs from a directory with Fluentd

Use a wildcard in Fluentd's in_tail path to follow multiple log files. /var/log/app/*.log selects one directory; /var/log/app/**/*.log includes matching files through the directory tree. Each selected file has its own reading position.

Configure the file input

<source>
  @type tail
  path /var/log/app/**/*.log
  exclude_path ["/var/log/app/**/*.gz"]
  pos_file /var/lib/fluentd/app.pos
  tag app.*
  path_key source_file
  refresh_interval 60s
  follow_inodes true
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

<match app.**>
  @type stdout
</match>

Create a writable directory for pos_file, and do not share the same position file between different Tail inputs. The file is strongly recommended for restart continuity, although the option is not mandatory. Without it, a restart can reread or skip existing content depending on the starting-position settings.

The parser above preserves each line as a message field. Replace it with the appropriate parser when the application writes JSON or another structured format.

Understand discovery and rotation

refresh_interval controls discovery of matching paths. A newly created file may wait for the next refresh, and a short-lived file removed before discovery can be missed. It is not a universal maximum ingestion delay.

follow_inodes true helps avoid duplicate reads when wildcard paths include renamed files. It does not eliminate every rotation race: retain rotated files long enough to finish reading and test the actual logrotate policy.

read_from_head true requests a backfill for files with no saved position. Existing offsets still take precedence. Since Fluentd 1.14.3, newly added files discovered after startup are read from the beginning even when read_from_head is false; that setting chiefly affects startup files. Check the in_tail reference for your version.

Keep file identity useful

In tag app.*, Fluentd expands the asterisk from the file path, replacing path separators with dots. The matching pattern app.** accepts the resulting hierarchy. path_key source_file also keeps an explicit path field, which is often easier to query or route on than reconstructing it from a tag.

For thousands of files, check open-file limits, watch limits, memory, and output-buffer capacity. Exclude archives and irrelevant directories deliberately. An input glob is not permission to read a file: Fluentd's service user still needs directory traversal and file-read access.

If different files need different destinations, add explicit tag or field-based routing and test one event from each path. This makes the routing contract visible as the directory grows.