Can Logstash report the source file line number?

The Logstash file input does not attach the physical source-file line number to each event. A Ruby counter can count events processed during one run, but that number changes meaning after a restart, resume, rotation, or multiline merge.

Count events in a controlled single-worker pipeline

Set these pipeline options first:

pipeline.workers: 1
pipeline.ordered: true

Then, for a diagnostic pipeline with ECS v8 enabled:

input {
  file {
    path => "/var/log/myapp/*.log"
    start_position => "beginning"
    ecs_compatibility => "v8"
  }
}
filter {
  ruby {
    init => "@counters = Hash.new(0)"
    code => '
      path = event.get("[log][file][path]")
      if path
        @counters[path] += 1
        event.set("event_sequence", @counters[path])
      end
    '
  }
}
output {
  stdout { codec => rubydebug }
}

Each path starts at one when first encountered in that process. An existing sincedb record can resume halfway through a file even with start_position => "beginning". The sequence then starts at one for the remaining events, not at the file's first physical line.

Instance variables are shared by filter workers. This example requires a single worker to avoid concurrent updates, and ordered processing for its intended diagnostic meaning. Counters reset on restart, are keyed by path rather than file identity, and can grow without bound as new paths appear. Do not use this state as a production deduplication key.

Extract a position explicitly present in a log

If the message already reports a location, extract it:

filter {
  grok {
    match => { "message" => "^%{DATA:source_file}:%{POSINT:source_line:int}: %{GREEDYDATA:msg}$" }
  }
}

For app.rb:128: request failed, source_line is 128. Java %L and a Go caller field normally describe a source-code location, not the position in the output log file. Name and document the field accordingly.

Choose a locator that fits the task

For investigation, keep the path and timestamp, and use a byte offset if your collector actually supplies one. Logstash records offsets internally in sincedb but does not expose a native per-event offset or line-number field. Filebeat's log.offset is a different collector's field.

For ordering, combine timestamps with a reliable sequence generated at the source where available. Timestamps alone do not establish a total order. For deduplication, a fingerprint of appropriate stable content can be more useful, provided its collision and identity semantics fit the data.

Multiline decoding turns several lines into one event, so the counter advances once. Disabling multiline alone does not solve restart or rotation semantics. True physical line numbers require a reader that defines encoding, newline, file identity, and persisted position behavior, or an explicit source-provided position. Measure Ruby overhead in your own workload rather than assuming a fixed performance penalty.

Related reading

• Handle LF and CRLF in Git
• Combine multiline logs in Logstash

References

• Elastic: inputs file
• Elastic: filters ruby
• Elastic: filters fingerprint