Rotate Fluentd file output by time slice

Fluentd's file output groups buffered events by time and writes them to time-based paths. This provides hourly or daily output files without renaming an actively written file with logrotate. It does not automatically delete old output.

Write one path per hour

The following Fluentd v1 configuration uses UTC event-time buckets:

<match app.**>
  @type file
  path /var/log/fluentd-agg/app.%Y-%m-%d.%H
  append true
  <buffer time>
    @type file
    path /var/lib/fluentd/buffer/app
    timekey 1h
    timekey_wait 10s
    timekey_use_utc true
  </buffer>
  <format>
    @type json
  </format>
</match>

Prepare writable output and buffer directories. The output adds its default .log suffix, producing a path such as app.2024-01-15.10.log. append true allows multiple flushed chunks for that hour to share the file. Without it, the default behavior produces separate chunk files with numeric suffixes, so “one file per hour” is not guaranteed.

For daily slices, use timekey 1d and remove .%H from the path. Keep the time zone explicit so the boundary is predictable across hosts.

Understand the flush delay

timekey_wait 10s allows a delay after a time window ends before its chunk becomes eligible to flush. It helps accommodate late events; it does not prove that the file is permanently complete. Much later events or replayed records can still target an older time bucket.

For example, a record with an event timestamp in the 10:00–11:00 UTC window belongs to that hour even if it arrives later. Check parsing and event timestamps if files appear under an unexpected date.

Compress and retain completed output

To gzip the final output, add compress gzip at the output level, next to path and append. Keep <format> @type json </format> to retain JSON formatting. Buffer compression is a separate setting; changing the formatter to gzip is not the way to compress files.

Use a retention job for files that are no longer being written or replayed. Preview the selection before deleting anything:

find /var/log/fluentd-agg -type f -name 'app.*.log*' -mtime +30 -print

Review the paths and your archive requirements before adding deletion to an approved cleanup job. Keep buffer directories out of that cleanup path.

Can I rotate by size?

chunk_limit_size limits buffered chunk size, not an exact final-file rotation threshold. With append true, many chunks can accumulate in one time-based file. If separate size-bounded archives are required, design that retention/archive stage explicitly rather than treating the buffer setting as logrotate's size option.

References: file output and buffer configuration.