Change fields in Fluentd JSON output

Use Fluentd's record_transformer filter to change the record before it reaches an output. The output formatter then controls serialization. Adding a field and choosing JSON output are separate operations.

Transform the record before writing it

This example reads JSON Lines, removes two top-level secrets, adds context, and extracts a nested user ID:

<source>
  @type tail
  path /var/log/app.log
  pos_file /var/lib/fluentd/app.pos
  tag app.logs
  <parse>
    @type json
  </parse>
</source>

<filter app.logs>
  @type record_transformer
  enable_ruby true
  auto_typecast true
  remove_keys password,token
  <record>
    hostname ${hostname}
    environment production
    user_id ${record.dig("user", "id")}
    message ${record["message"].to_s.strip}
  </record>
</filter>

<match app.logs>
  @type file
  path /var/log/processed/app
  <format>
    @type json
  </format>
</match>

Ensure the position-file and output directories are writable. Fluentd's file output buffers and names its output files; the path is an output prefix, not a promise that every record immediately appears in a file with that exact name.

enable_ruby true is necessary for arbitrary Ruby expressions such as strip and dig. Here dig tolerates a missing user or id, but assumes that a present user is a hash. Validate unexpected shapes before applying this configuration. to_s deliberately converts a missing message to an empty string; remove that conversion if missing messages should be treated as errors instead.

Preserve the types you need

With auto_typecast true, a value consisting of a single placeholder can keep the underlying value's type. An integer user ID therefore remains numeric. Adding literal text around a placeholder produces a string. Do not rely on enable_ruby false as a general solution to type preservation.

For conditional fields, Ruby expressions can select a value:

<record>
  level_name ${record["status"].to_i >= 500 ? "error" : "ok"}
</record>

That expression assumes HTTP-like numeric statuses. A missing or invalid status becomes zero through to_i, so validate it if that would incorrectly label an event.

Check the entire redaction boundary

remove_keys password,token removes those top-level fields. It does not scrub copies inside message or a nested object. Inspect representative output records, including missing fields and nested secrets, before describing the result as redacted.

The record_transformer reference documents placeholders and type handling. Keep business fields as structured values until the formatter writes JSON, instead of manually constructing a JSON string inside a field.