Fix Fluentd’s “pattern not match” warning

A pattern not match warning usually means the input line does not fit the configured parser. Start with the exact raw line and the parser that received it. Broadening a regex before identifying the mismatch can hide a format change and silently remove useful fields.

Compare the input with the parser

Inspect Fluentd's warning and the original log file. Use sanitized copies when sharing samples; avoid pasting production records into public regex tools.

Check these common differences:

• Timestamp format: %d/%b/%Y:%H:%M:%S %z fits 10/Oct/2000:13:55:36 -0700. A different month representation, fractional seconds, or time-zone syntax needs a matching format.
• Separators and optional fields: an extra space, empty field, or quoted value can break a regex that worked on one sample.
• Event boundaries: a stack-trace continuation line is unlikely to match a pattern written for the first line. Configure multiline assembly before expecting one logical event to parse.
• Outer wrappers: container logs may wrap application text in a log or message field. Parse the wrapper and then the inner field rather than applying the application regex to the entire wrapper.

Use Ruby-compatible regex tooling, since Fluentd's regexp parser uses Ruby syntax. Test successful lines, failed lines, and edge cases. A regex match does not prove that timestamp conversion also succeeds.

Preserve unmatched input while investigating

For in_tail, enable unmatched-line emission alongside the existing parser:

<source>
  @type tail
  path /var/log/app/app.log
  pos_file /var/lib/fluentd/app.pos
  tag app.raw
  emit_unmatched_lines true
  <parse>
    @type regexp
    expression /^(?<message>.+)$/
  </parse>
</source>

<match app.raw>
  @type stdout
</match>

Prepare a writable position-file directory and use a test log path. This intentionally permissive example captures nonempty lines as message; adapt the expression to the real format. A line that still fails is emitted with unmatched_line, so downstream handling must allow that alternative shape.

Another temporary choice is <parse> with @type none, which preserves the raw line as message. Switching to it removes structured parsing and may change event timestamps. It is a diagnostic fallback, not a completed parser fix.

FAQ

Are failures harmless if I only see warnings?

No. With in_tail's default emit_unmatched_lines false, unmatched lines can be dropped. Repeated warnings also add overhead. Check delivery, not only whether the process remains alive.

Can I parse only selected tags?

Yes. A parser can run in a source's <parse> section, or in a parser filter that matches selected tags and parses a named field. These are different error paths; the parser filter can route invalid records to @ERROR.

References: tail input, regexp parser, and parser filter.