Parse a JSON string inside a Fluentd record

When a log contains JSON inside a string field, parse it in two steps: extract the outer fields, then decode the string with a parser filter. For example:

2024-01-15 10:30:00 INFO request={"order_id":123,"amount":42.5}

The date and time together form the timestamp. A shorter line containing only a date would need a different expression.

Parse the envelope, then the payload

<source>
  @type tail
  path /var/log/app.log
  pos_file /var/lib/fluentd/app.pos
  tag app.mixed
  <parse>
    @type regexp
    expression /^(?<time>\S+ \S+) (?<level>\w+) request=(?<request>.*)$/
    time_format %Y-%m-%d %H:%M:%S
  </parse>
</source>

<filter app.mixed>
  @type parser
  key_name request
  reserve_data true
  reserve_time true
  hash_value_field payload
  <parse>
    @type json
  </parse>
</filter>

<match app.mixed>
  @type stdout
</match>

<label @ERROR>
  <match **>
    @type stdout
  </match>
</label>

Set the outer parser's timezone explicitly if the input's timezone differs from the collector's environment. The greedy .* captures the complete JSON text, including spaces. This assumes the JSON is the last part of the line and does not span physical lines.

reserve_data true keeps outer fields in the successfully parsed record. reserve_time true preserves the event time established by the outer parser. hash_value_field payload places the inner object under payload, avoiding collisions between inner and outer field names.

Decide how errors should travel

Parser errors are emitted to @ERROR by default. With reserve_data true, missing fields and parser nonmatches can also preserve the original record on the normal route; in this example, payload can be an empty object. Other parser exceptions may only reach the error route. The example makes errors visible on stdout, but it can therefore print the same failed input on both routes. Choose a deliberate failure policy and test malformed JSON, a missing request field, and an invalid field type.

To flatten the inner fields at the top level, omit hash_value_field. To prefix their names instead, use inject_key_prefix, such as request.. Both are documented parser filter options; a separate transformer is unnecessary for these operations.

Handle escaping one layer at a time

If the outer record itself is JSON, parse that layer first. Its JSON decoder removes the escaping that belongs to the outer string, leaving valid text for the next parser. Standard escaped quotes and newlines are handled by JSON decoding.

Do not blindly remove backslashes from double-encoded or malformed data: that can change legitimate values. Inspect the encoding contract, decode only the layers that actually exist, and add another parser only when the next value is still a JSON string.