Can I remove the message field in Logstash?

Remove the original message after successful parsing, preserve failed events, and understand what field removal changes in Elasticsearch.

Best practices
Illustration of log monitoring and analysis

Yes. Use the mutate filter's remove_field option to remove message from an event. Do this after extracting the fields you need, and retain the original text when parsing fails.

Remove it after a successful parse

The filter's common remove_field option runs when that filter succeeds, so it can be attached directly to grok:

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    remove_field => ["message"]
  }
}

For a separate cleanup step, the following pattern is useful when this grok filter is the only producer of the default parse-failure and timeout tags in the event:

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  if "_grokparsefailure" not in [tags] and "_groktimeout" not in [tags] {
    mutate { remove_field => ["message"] }
  }
}

Grok timeouts use _groktimeout; checking only for _grokparsefailure can miss that failure path. Keep the filter-attached removal in the first example when possible.

An unconditional mutate { remove_field => ["message"] } deletes the field even when parsing failed. In pipelines with several parsers, use a parser-specific success or failure tag so one stage does not accidentally decide another stage's cleanup policy.

Decide what you still need

Remove the raw text only when the parsed fields preserve the information needed for investigation and retention requirements. Keeping it gives you a fallback for unrecognized formats and a way to reprocess events later.

Full-text search continues to work on remaining fields mapped and indexed for that purpose. It cannot search text that you removed. Field removal affects events processed by the new configuration; it does not erase old Elasticsearch documents or copies retained elsewhere.

Storage savings depend on the message, mappings, compression, and the fields retained. Compare equivalent batches under the same index settings instead of treating the raw message length as a guaranteed saving.

References

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting