Parse nested JSON fields in Fluentd

A JSON object nested inside another object is already parsed when Fluentd decodes the outer JSON. A field containing a string of JSON, however, needs a second parser. Start by inspecting the actual value type rather than adding a parser to every nested field.

Parse a JSON-encoded string

Consider this log line:

{"level":"INFO","request_body":"{\"order_id\":123,\"amount\":99.5}"}

The outer parser produces a request_body string. A parser filter can decode that string and merge its fields into the event:

<source>
  @type tail
  path /var/log/app/*.log
  pos_file /var/lib/fluentd/app-json.pos
  tag app.json
  <parse>
    @type json
  </parse>
</source>

<filter app.json>
  @type parser
  key_name request_body
  reserve_data true
  reserve_time true
  <parse>
    @type json
  </parse>
</filter>

<match app.json>
  @type stdout
</match>

<label @ERROR>
  <match **>
    @type stdout
  </match>
</label>

Prepare a writable position-file directory and test with non-sensitive input. reserve_data true retains the original fields, while reserve_time true keeps the event's existing timestamp. The successful output includes top-level order_id and amount alongside level and the original string.

If parsed keys might collide with existing names, use hash_value_field parsed_request in the parser filter to place the decoded object under a separate key. Add remove_key_name_field true only when you no longer need the original string after successful parsing.

Access an already nested object

For data such as {"user":{"profile":{"city":"Paris"}}}, no second JSON decode is needed. With Ruby enabled, a transformer can read a nested value:

<filter app.json>
  @type record_transformer
  enable_ruby true
  <record>
    user_city ${record.dig("user", "profile", "city")}
  </record>
</filter>

This assumes each intermediate value has the expected hash shape. Missing keys return nil; a value of the wrong type can still raise an error. Decide whether malformed records should be rejected, quarantined, or normalized.

Fluentd's record_accessor syntax, such as $.user.profile.city, is built into plugins that explicitly support that helper. It is not a separate plugin to install, and it is not interchangeable with Ruby expressions in every parameter.

FAQ

What happens when the inner value is invalid JSON?

The parser filter normally emits an error event to @ERROR. With reserve_data true, ordinary parse failures that return no parsed values, and missing-key cases, can also retain the original record on the normal route. Unexpected parser exceptions may follow only the error route. Thus the example exposes errors but does not guarantee exclusive quarantine: test malformed JSON, a missing field, and a value of the wrong type, then add explicit routing if invalid records must never reach the normal destination. See the parser filter implementation for these branches.

Can I access array elements?

Supported record-accessor paths can select an index such as [0]. Expanding an array into several events is a different operation and requires a suitable transformation/plugin with an explicit event model.

References: parser filter, record accessor, and record transformer.