Preserve multiline log events with rsyslog

Join stack-trace lines while the collector still knows which file they belong to. In rsyslog, imfile can use startmsg.regex to recognize the first line of an event and append continuation lines to it.

Syslog is a message protocol, not a universal “one physical line per event” rule. Problems arise when the sender or transport framing splits a logical event into separate messages.

Define the beginning of each event

For application logs whose first line begins with an ISO-style date:

module(load="imfile")

input(type="imfile"
      File="/var/log/myapp/app.log"
      Tag="myapp:"
      startmsg.regex="^[0-9]{4}-[0-9]{2}-[0-9]{2}"
      readTimeout="5"
      escapeLF="on")

This is an input fragment for an existing rsyslog configuration with a destination rule. Load the module only once. Set a writable state directory and an appropriate persistence policy for your installation.

A matching line begins a new message. Nonmatching lines continue the current message. readTimeout allows an unfinished event to be emitted when no subsequent start line arrives; verify supported behavior in the installed version. escapeLF represents embedded line feeds as #012, so check the receiver's expected representation.

Use the documented imfile options, rather than assuming an invented legacy directive such as $InputFileStartRegex is supported. Older configurations using $InputFileName and $InputFileTag need a version-appropriate migration for regex-based multiline handling.

Test boundaries, size, and idle periods

Feed a complete exception, a second ordinary event, and a final exception followed by silence. Confirm each becomes exactly one event. Also test an unexpectedly long stack trace and a line inside the trace that resembles a date.

If every line matches the start expression, nothing is joined. If no line matches, records can accumulate or be emitted at other limits; do not expect a safe automatic fallback to one event per line. Tune the regex from actual input.

A global setting such as global(maxMessageSize="64k") changes the message-size budget, but the rest of the path may have smaller limits. Check both endpoints and avoid treating a larger setting as a guarantee against truncation.

Preserve boundaries during transport

One UDP datagram can carry a syslog message containing more than one line. If the sender instead emits one datagram per stack line, the receiver generally cannot reconstruct interleaved events reliably. Large datagrams also risk fragmentation and loss.

For TCP, newline-delimited framing can conflict with literal newlines; octet-counting framing carries a length explicitly when both peers support it. Another option is JSON with escaped newline characters in a stack field.

DataKit's file log collector supports start-of-event multiline configuration. Whichever collector you use, verify the complete exception and the following event at the destination, including flush delays and length limits.