Truncate long message fields with a Fluent Bit Lua filter

A Lua filter can shorten a selected string field after Fluent Bit has ingested the event. Define the limit clearly: Lua string lengths count bytes, not Unicode characters, and an appended marker also consumes space.

Keep the result within a byte budget

Save this as /etc/fluent-bit/truncate.lua. It assumes valid UTF-8 input and avoids splitting a multibyte character at the cutoff:

function truncate_msg(tag, timestamp, record)
    local max_bytes = 1000
    local marker = "...[truncated]"
    local msg = record.msg
    if type(msg) ~= "string" or #msg <= max_bytes then
        return 0, timestamp, record
    end

    local cut = max_bytes - #marker
    while cut > 0 do
        local next_byte = string.byte(msg, cut + 1)
        if next_byte < 128 or next_byte >= 192 then
            break
        end
        cut = cut - 1
    end
    record.msg = string.sub(msg, 1, cut) .. marker
    return 2, timestamp, record
end

Return code 0 leaves the event unchanged. Code 2 replaces the record while preserving its original timestamp; code 1 replaces both. Keep the configured byte limit larger than the marker.

Apply the filter to the right field

This example tails unstructured lines and explicitly names the resulting field msg:

[INPUT]
    Name    tail
    Path    /var/log/myapp/*.log
    Tag     myapp.*
    Key     msg

[FILTER]
    Name    lua
    Match   myapp.*
    script  /etc/fluent-bit/truncate.lua
    call    truncate_msg

[OUTPUT]
    Name    stdout
    Match   myapp.*

Without Key msg, unparsed Tail records normally use log, and a filter looking only at msg would do nothing. For JSON records, parse the input first and target the actual field.

The Lua filter documentation describes the callback contract. A Logstash truncate filter cannot be installed as a Fluent Bit filter. Also distinguish field truncation from version-specific Tail options that handle oversized input lines: a Lua filter cannot rescue a line rejected before it reaches the filter.

Preserve an original only when you need it

Truncation can reduce network and storage use for large bodies or statements, but it may remove the part needed to diagnose an error. Choose the budget from representative events and consider excluding important fields.

All outputs receiving this modified record see the shortened message. Adding a second output after the filter does not create an original archive. Preserve the source file or branch/copy the original before modification, with appropriate retention and access controls.

For multiple fields, apply the same guarded logic to an explicit list. Test empty strings, missing and numeric values, exact-limit input, and multibyte text before deploying the script.