Understand the Logstash configuration format

A Logstash pipeline uses its own declarative configuration language. It is not JSON or YAML. The main sections are input, optional filter, and output; plugin settings use => between a key and its value.

input {
  # Input plugins
}
filter {
  # Optional parsing and transformation
}
output {
  # Destination plugins
}

Settings such as logstash.yml and pipelines.yml are YAML, but pipeline .conf files use the syntax shown here.

Define sources in input

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
  tcp {
    port => 5140
    codec => line
  }
}

These are independent inputs feeding one pipeline. An event does not pass through every input in turn. start_position affects a previously unseen file; an existing sincedb position wins. For TCP, agree on event framing with the sender; the example expects newline-delimited records.

Parse and transform in filter

For a pipeline dedicated to combined Apache-format access logs:

filter {
  grok {
    ecs_compatibility => "disabled"
    match => { "message" => "^%{COMBINEDAPACHELOG}$" }
  }
  if "_grokparsefailure" not in [tags] and "_groktimeout" not in [tags] and [timestamp] {
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      locale => "en"
    }
    if "_dateparsefailure" not in [tags] {
      mutate { remove_field => ["message"] }
    }
  }
}

This parser is separate from the generic syslog/TCP input example: only apply it to sources that actually use this format. The guards preserve raw messages when parsing fails. Keep raw data longer if your troubleshooting or retention requirements call for it.

Filters run in configuration order for an event. That does not guarantee ordering between events processed by multiple workers. Other common filters include JSON, mutate, and GeoIP; conditional branches can select by source or field value.

Configure destinations in output

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "app-%{+yyyy.MM.dd}"
    ilm_enabled => false
    data_stream => false
  }
  stdout { codec => rubydebug }
}

This is a local ordinary-index example. Configure templates, retention, TLS, and credentials separately. Debug output can expose sensitive event data; retain it only where appropriate. Each event reaches every matching output branch unless it was dropped earlier.

Understand multiple files and validation

When path.config points to a directory or glob, matching files are combined in alphabetical order into one pipeline. Names such as 00-inputs.conf, 10-filters.conf, and 30-outputs.conf organize that combined configuration; they do not create separate pipelines. Use pipelines.yml for independent pipelines.

Check syntax with:

bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/

A passing syntax check does not validate Grok patterns against your data or prove network outputs work. Run representative valid and invalid events through a test pipeline before reloading production configuration.

Reference: Logstash pipeline structure.