Parse JSON logs into Elasticsearch fields with Fluent Bit

Use a JSON parser to turn each log line into a record before Fluent Bit sends it to Elasticsearch. A nested JSON object is already structured: Elasticsearch can address fields such as request.method without moving them to the top level. Use the Nest filter's lift operation only when your desired schema requires that change.

Load a JSON parser

In classic configuration mode, define parsers in a separate file, for example /etc/fluent-bit/parsers-custom.conf:

[PARSER]
    Name    app_json
    Format  json

Reference that file from the main configuration:

[SERVICE]
    Parsers_File /etc/fluent-bit/parsers-custom.conf

[INPUT]
    Name    tail
    Tag     app.logs
    Path    /var/log/app/*.log
    DB      /var/lib/fluent-bit/app.db
    Parser  app_json

[FILTER]
    Name         nest
    Match        app.logs
    Operation    lift
    Nested_under request
    Add_prefix   request_

[OUTPUT]
    Name               es
    Match              app.logs
    Host               elasticsearch
    Port               9200
    Index              app-logs
    Suppress_Type_Name On

Prepare writable storage for the Tail database. The output assumes a reachable Elasticsearch endpoint; add the authentication, CA trust, and TLS settings required by your cluster. Suppress_Type_Name On avoids mapping-type metadata for Elasticsearch versions that no longer accept it, including Elasticsearch 8.

Given this input:

{"request":{"method":"GET","path":"/api"},"status":200}

The filter produces these fields:

{"request_method":"GET","request_path":"/api","status":200}

The prefix reduces collisions with existing top-level names. If you omit the filter, request remains an object. A string containing JSON, such as "request":"{...}", requires an additional parsing step; lift cannot decode that string.

Reverse the transformation

Operation nest collects matching top-level fields under a new object. For example, Wildcard request_* with Nest_under request and Remove_prefix request_ rebuilds a request object. Check for an existing destination key before applying it.

FAQ

Why is a number indexed as text?

Check whether the producer emits 200 or "200", then inspect the index mapping. Correct the producer or use a documented conversion processor before indexing. Changing later events does not generally change an existing field's mapping; a corrected mapping and a new index or reindexing may be needed.

How do I verify the result?

Use stdout temporarily with non-sensitive test events to inspect the transformed record. Then inspect an indexed document and its mapping. Parsing a JSON record successfully does not guarantee every field is searchable or aggregatable; mappings control those capabilities.

References: JSON parser, Nest filter, and Elasticsearch output.