Route different log sources to separate Elasticsearch indices

Label each Filebeat input, then use that label to choose an Elasticsearch index. When Filebeat sends through Logstash, the Logstash output determines the destination. With a direct Elasticsearch output, Filebeat can select it itself.

The examples below use ordinary indices, not data streams or automatic rollover aliases. They are local demonstrations; configure TLS, authentication, permissions, templates, and retention for your deployment.

Label each Filebeat input

filebeat.inputs:
  - type: filestream
    id: app1-logs
    paths: ["/var/log/app1/*.log"]
    fields:
      log_type: app1
    fields_under_root: true

  - type: filestream
    id: app2-logs
    paths: ["/var/log/app2/*.log"]
    fields:
      log_type: app2
    fields_under_root: true

output.logstash:
  hosts: ["localhost:5044"]

fields_under_root: true puts log_type at the event root. Without it, Logstash must refer to [fields][log_type] rather than [log_type]. Choose a custom field name that does not overwrite information your application or collector already produces.

Route events in Logstash

input {
  beats { port => 5044 }
}
filter {
  if [log_type] == "app1" {
    grok {
      ecs_compatibility => "disabled"
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
}
output {
  if [log_type] == "app1" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      ilm_enabled => false
      data_stream => false
      index => "app1-%{+yyyy.MM.dd}"
    }
  } else if [log_type] == "app2" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      ilm_enabled => false
      data_stream => false
      index => "app2-%{+yyyy.MM.dd}"
    }
  } else {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      ilm_enabled => false
      data_stream => false
      index => "unknown-%{+yyyy.MM.dd}"
    }
  }
}

Use the Apache parser only if app1 actually emits that format. The fallback index exposes missing or unexpected labels instead of silently mixing those events into an application's index. For many routes into the same cluster, setting an index name in @metadata and using one Elasticsearch output can reduce connection overhead.

Route directly from Filebeat

Without Logstash, configure conditional index selection:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "unknown-%{[agent.version]}-%{+yyyy.MM.dd}"
  indices:
    - index: "app1-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.equals:
        log_type: app1
    - index: "app2-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.equals:
        log_type: app2

Custom indices disables Filebeat's ILM handling. With ILM enabled, a custom single index setting is ignored. Install matching templates separately and choose one coherent lifecycle strategy. Filebeat's Logstash output does not automatically install its Elasticsearch template.

Decide whether separate indices are worthwhile

Separate indices let you apply different retention periods and index permissions, target searches more narrowly, and limit the scope of an accidental deletion. They do not isolate workloads from every cluster-wide failure.

Daily naming does not delete old data. Configure retention independently. In Logstash, %{+yyyy.MM.dd} uses legacy Joda formatting; %{{...}} uses Java Time. The date comes from the event's @timestamp, expressed in UTC, not necessarily the machine's local calendar day.

If every event reaches one index, inspect a sample with stdout { codec => rubydebug }, check the field path, and confirm the updated Filebeat configuration was loaded. If small indices create too many shards, combine low-volume sources or use an appropriate rollover policy. Count primary and replica shards from the actual templates rather than assuming two shards per index.

References: Filebeat Elasticsearch output, Logstash Elasticsearch output, and filestream input.

Related reading

• Copy a directory with Dockerfile ADD
• Delete a remote Git tag