Build and debug Logstash Grok patterns

Grok combines regular-expression patterns into a parser. %{PATTERN:field} matches a pattern and stores the captured value in a field. Omitting :field matches the text without creating that named capture.

Start with the right building blocks

Do not assume a pattern named FLOAT exists in the bundled pattern set. NUMBER with :float is the usual way to capture a floating-point value. Grok supports int and float capture conversions; other types need an appropriate filter.

Match a complete example

2024-01-15 10:30:00 INFO User 42 logged in from 192.168.1.1

A working filter is:

filter {
  grok {
    match => {
      "message" => '^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} User %{INT:user_id:int} (?<action>logged in) from %{IP:client_ip}$'
    }
  }
}

The action is two words, so %{WORD:action} would not consume logged in. Anchors make unexpected prefixes or trailing text fail visibly rather than allowing a partial match. A timestamp capture is still a string; a date filter is needed to set @timestamp.

Use composite patterns deliberately

COMMONAPACHELOG and COMBINEDAPACHELOG cover the corresponding common and combined access-log formats, including compatible Nginx formats. They do not cover every custom access log. SYSLOGLINE handles a supported syslog shape, not every possible syslog protocol framing.

Composite captures can change with ecs_compatibility. Test using the same mode as the deployed pipeline. For variable formats, keep DATA bounded by a reliable delimiter and avoid several overlapping GREEDYDATA captures.

Check failures and escaping

Unmatched events normally receive _grokparsefailure; a timed-out expression can receive _groktimeout. The original message remains unless another operation removes it. Route these cases deliberately.

For a literal opening bracket, a single-quoted Logstash string can contain \[. Do not blindly double every backslash: escaping depends on the surrounding configuration string and config.support_escapes. Test the exact configuration file, not only a regex copied into a different tool.

Guance Pipeline can test extraction against the same sample, but its functions and conversion behavior are not interchangeable with a Logstash filter. Validate each target pipeline separately.

References: Grok filter and bundled patterns.