Route Logstash events to multiple Elasticsearch indices

Set a routing value during filtering, then interpolate it in the Elasticsearch output's index setting. Keeping the value under @metadata makes it available to the output without adding a routing-only field to the stored document.

Route application and error logs

This example expects application lines beginning with an ISO 8601 timestamp and error lines beginning with a timestamp followed by ERROR:. Adapt the patterns to your actual log format; standard Nginx error logs have a different format.

input {
  file {
    path => "/var/log/myapp/*.log"
    start_position => "beginning"
    type => "app_log"
  }
  file {
    path => "/var/log/myapp-errors/*.log"
    type => "error_log"
  }
}
filter {
  if [type] == "app_log" {
    grok {
      match => { "message" => "^%{TIMESTAMP_ISO8601:log_time} %{GREEDYDATA:log_message}$" }
    }
    mutate { add_field => { "[@metadata][index_name]" => "app-logs" } }
  } else if [type] == "error_log" {
    grok {
      match => { "message" => "^%{TIMESTAMP_ISO8601:log_time} ERROR: %{GREEDYDATA:error_message}$" }
    }
    mutate { add_field => { "[@metadata][index_name]" => "error-logs" } }
  } else {
    mutate { add_field => { "[@metadata][index_name]" => "other-logs" } }
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    ilm_enabled => false
    data_stream => false
    index => "%{[@metadata][index_name]}-%{+yyyy.MM.dd}"
  }
}

Configure credentials and TLS for a secured cluster. The fallback covers unexpected types, while parse failures retain their original message and failure tag for investigation.

Choose the lifecycle model deliberately

The example creates ordinary indices named by the event's @timestamp in UTC. Parsing a timestamp into log_time alone does not change that value; add a date filter if the index date should follow source event time.

Daily names do not configure retention automatically. Plan templates, shard counts, and deletion policies separately. If using ILM rollover aliases, do not expect dynamic index interpolation to override the rollover alias. Data streams use another naming and lifecycle model.

One output or several?

A single output works well for several index prefixes in the same cluster. Separate conditional outputs make sense for different clusters or materially different output settings. Independent output blocks can send the same event more than once; use mutually exclusive branches when each event should have one destination.

Always set the routing field before interpolation. Missing fields can remain as unresolved %{...} expressions or cause indexing failures, depending on plugin handling. Inspect the resulting target and failure logs instead of assuming the event landed in a fallback index.

References

• Elastic: outputs elasticsearch
• Elastic: event dependent configuration
• Elastic: inputs file