Parse fields from a Filebeat message

Choose the parser from the log format. Filebeat supports decode_json_fields for JSON stored in a string and dissect for predictable delimiters. For grok patterns, use an Elasticsearch ingest pipeline or Logstash; grok is not a Filebeat processor.

Decode a JSON message

This example keeps decoded fields under parsed, avoiding accidental replacement of Filebeat metadata:

processors:
  - decode_json_fields:
      fields: ['message']
      target: 'parsed'
      overwrite_keys: false
      add_error_key: true

For {"user":"tom","action":"login"}, the resulting fields include parsed.user and parsed.action. Setting target: '' merges decoded fields into the event root. Use that only after deciding how collisions should behave; overwrite_keys: true permits incoming values to replace existing ones.

Dissect stable text

processors:
  - dissect:
      tokenizer: '%{timestamp} %{level} %{service} %{msg}'
      field: 'message'
      target_prefix: 'app'

For 2024-01-01 INFO order Order accepted, this extracts app.level as INFO, app.service as order, and the trailing text into app.msg. It does not automatically turn the date into @timestamp; timestamp parsing is a separate step.

Dissect follows delimiters rather than regular-expression patterns. It is a good fit when the structure is consistent. Benchmark your own messages rather than assuming a fixed speedup over grok.

Apply grok in Elasticsearch

Create an ingest pipeline in Dev Tools:

PUT _ingest/pipeline/access-line
{
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:client_ip} %{WORD:method} %{URIPATHPARAM:url_path} %{INT:status:int}"]
      }
    }
  ]
}

This example accepts 192.0.2.1 GET /health 200. Select the installed pipeline in the Filebeat Elasticsearch output:

output.elasticsearch:
  hosts: ['https://es.example.com:9200']
  pipeline: 'access-line'

Add the authentication and certificate settings required by your cluster. If the output is Logstash, configure parsing or pipeline selection there instead.

Handle failures deliberately

Failure behavior differs between processors. JSON decoding can attach an error key; dissect has ignore_failure; an ingest-pipeline failure needs appropriate failure handling or can reject indexing. Inspect failed samples before deciding to drop them.

References: Filebeat JSON decoding, dissect, and Elasticsearch grok processor.