Send one Fluentd source through multiple filters and outputs

A single Fluentd source can feed several filters and destinations. Matching filters run in sequence before the first matching output. To deliver the same event to multiple outputs, use copy with a <store> for each destination; repeating <match app.**> does not create a broadcast.

Chain the filters

This example accepts local Forward events, adds a host field, and removes DEBUG records before output:

<source>
  @type forward
  bind 127.0.0.1
  port 24224
</source>

<filter app.**>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
  </record>
</filter>

<filter app.**>
  @type grep
  <exclude>
    key level
    pattern /^DEBUG$/
  </exclude>
</filter>

The filters apply to events tagged app or any descendant such as app.access. Put the output after them. A matching output earlier in the configuration prevents later filters from seeing the event.

Copy the result to two destinations

For a small local test, send records to stdout and a buffered file:

<match app.**>
  @type copy
  <store>
    @type stdout
  </store>
  <store>
    @type file
    path /var/log/fluentd/archive
    <buffer time>
      @type file
      path /var/lib/fluentd/buffer/archive
      timekey 1h
      timekey_wait 10s
    </buffer>
    <format>
      @type json
    </format>
  </store>
</match>

Prepare writable directories. The file output flushes buffered time slices; it is not an immediate duplicate of each stdout line. In a real deployment, replace the stores with configured Elasticsearch and S3 outputs if those are your destinations. Both require their plugins, connection settings, credentials, and appropriate buffer paths.

Combine routing with filtering

A rewrite_tag_filter output can re-emit events under tags such as error.app and normal.app. Give each new route its own filters and output or copy block. Avoid rewriting back into the original matching pattern, which can create a loop.

FAQ

Can I reorder filters for performance?

Yes, by changing their configuration order, but first consider the data dependency. Dropping unwanted records before an expensive transformation can reduce work. A filter cannot use a field that a later transformation has not added yet.

Are copy stores independent?

Not completely. Buffered destinations can flush on their own schedules, but a store's emit or formatting error can prevent a later store from running. Buffer limits and overflow behavior also matter. ignore_error is an explicit trade-off for a less critical store, not automatic isolation. By default stores share record objects; use the documented copy_mode when a destination mutates records and isolation is needed.

References: copy output and event routing order.