Send JSON logs through Filebeat and Logstash to Kibana

Collect one JSON object per line, decode it once, and send the resulting fields to Elasticsearch. Kibana then queries those indexed documents; it does not receive logs directly from Filebeat.

This local example uses Filebeat's filestream input, a Logstash Beats input, and ordinary daily indices. Install matching Elasticsearch templates and retention rules, and add TLS and authentication for a secured deployment.

Decode JSON in Filebeat

filebeat.inputs:
  - type: filestream
    id: app-json-logs
    paths: ["/path/to/your/app.json"]
    parsers:
      - ndjson:
          target: "app"
          add_error_key: true

output.logstash:
  hosts: ["localhost:5044"]

An input line might be:

{"level":"INFO","msg":"User signed in","ts":1696150204,"status":"200"}

The app namespace prevents application fields from colliding with collector metadata. Older log input examples use json.keys_under_root; those settings are not the filestream parser syntax. The legacy log input is deprecated and disabled by default in Filebeat 9.0, so check migration guidance before carrying an old configuration forward.

Normalize and forward in Logstash

input {
  beats { port => 5044 }
}
filter {
  if [app][ts] {
    date { match => ["[app][ts]", "UNIX"] }
  }
  if [app][status] {
    mutate { convert => { "[app][status]" => "integer" } }
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    ilm_enabled => false
    data_stream => false
    index => "app-logs-%{+yyyy.MM.dd}"
  }
}

Do not run the JSON filter again on fields Filebeat has already decoded. UNIX expects seconds; use UNIX_MS for milliseconds. The daily index suffix follows the event's UTC @timestamp, so replaying the old sample above writes to its historical date, not today's index.

Find the events in Kibana

Create a data view matching app-logs-*, choose @timestamp as its time field, and open Discover. Set a time range that includes the sample's actual date. Check app.level, app.msg, and the numeric app.status.

If fields are missing, first confirm indexing succeeded and inspect the document and mapping. Reload data view metadata as needed for your Kibana version. A refresh cannot repair an Elasticsearch rejection.

Malformed JSON with add_error_key produces error.message and error.type: json, not an error.json field. Monitor these events and Logstash's _dateparsefailure tags. Inconsistent field types across sources need a consistent schema or separate mappings; changing the data view does not resolve them.

If Filebeat's parsing and processors are sufficient, it can send directly to Elasticsearch. Keep Logstash when its transformations, enrichment, or routing are useful. For daily indices, create templates before ingestion; Filebeat's Logstash output does not automatically install them.

References: filestream JSON parsing, legacy log input, and date filter.