Troubleshoot a Logstash file input that is not reading

Start by finding out whether Logstash discovers the file, reads new bytes, and emits events. A quiet output can mean a path or permission problem, an existing sincedb position, an incomplete line, or a blocked downstream pipeline.

Enable useful diagnostics

bin/logstash -f /etc/logstash/conf.d/ --log.level debug

Use a controlled test instance rather than launching a second process against a running service's data directory. Inspect file-input discovery messages and output errors, then restore the normal log level after troubleshooting.

Check paths and permissions

ls -l /path/to/your/log/file.log

The file input requires absolute paths. Confirm that your pattern matches the intended files and that the Logstash service account can read them and traverse every parent directory. /var/log/**/*.log matches recursively.

For a live file in Tail mode, append a complete newline-terminated record. EOF alone does not terminate a partial line in that mode. Also check ignore_older, open-file limits, and downstream backpressure.

Understand the saved position

Sincedb stores per-file byte positions under <path.data>/plugins/inputs/file unless you set sincedb_path. Changing start_position to beginning only affects files without a known position:

input {
  file {
    path => "/path/to/your/log/file.log"
    start_position => "beginning"
  }
}

For an isolated Unix test, combine that setting with sincedb_path => "/dev/null" and start a fresh process. This prevents position persistence between runs; it does not continuously rewind a file within the same run. Replays can create duplicate documents.

Do not delete every sincedb file on a production host. Stop the affected pipeline, back up its specific state file, and scope any reset to the source you intend to replay.

Account for rotation

The plugin tracks file identity, rather than only the pathname. Tail mode handles truncation by resetting the position when it detects a smaller file. Rename rotation and copy-truncate need different path choices; including copied archives can duplicate old content.

close_older manages idle file handles. file_completed_action applies to Read mode and is not a general fix for Tail-mode rotation. In Read mode its default can delete completed files, so select that mode and its completion policy deliberately.

References

• Elastic: inputs file
• Elastic: running logstash command line