Handle grok matches that fail in Logstash

A grok filter that fails to match normally adds _grokparsefailure and lets the event continue. It does not delete the original message. Keep that text available so you can diagnose the unexpected format.

Try specific patterns before broad ones

When a source has several known variants, put them in one match array:

filter {
  grok {
    match => { "message" => [
      "^%{COMBINEDAPACHELOG}$",
      "^%{COMMONAPACHELOG}$"
    ] }
  }
}

The default break_on_match => true stops after a successful pattern. An unanchored common-log pattern can match the beginning of a combined-log line and skip its extra fields, so specificity and anchoring matter more than simply putting the shortest pattern first.

Keep a failure route

Conditional output plugins belong inside the output section:

output {
  if "_grokparsefailure" in [tags] or "_groktimeout" in [tags] {
    file {
      path => "/var/log/logstash/grok_failures.log"
      codec => json_lines
    }
  } else {
    stdout { codec => rubydebug }
  }
}

The failure branch includes _groktimeout because a timed-out match can carry that tag without _grokparsefailure. The stdout branch is for testing; replace it with the normal successful-event output when ready. Give the failure destination appropriate permissions and retention, and keep it outside any directory that the same pipeline reads. Otherwise you can create a feedback loop.

Name failures by parsing stage

filter {
  grok {
    match => { "message" => "^%{COMBINEDAPACHELOG}$" }
    tag_on_failure => ["_access_log_parse_error"]
  }
}

Update downstream conditions to match your custom tag. Setting tag_on_failure => [] suppresses the marker; it neither fixes the parse nor stops other filters from changing the event. With several grok stages, a later success also does not automatically remove an earlier failure tag.

Measure the fraction of eligible events with the relevant failure tag, using the same source and time range for numerator and denominator. For unrelated formats, route first so each event runs only the applicable patterns. Keep common successful patterns early where their specificity allows, and test timeout behavior for expensive expressions; timeouts use a separate tag by default.

References

• Elastic: filters grok
• Elastic: event dependent configuration