Can I remove the message field in Logstash?
Remove the original message after successful parsing, preserve failed events, and understand what field removal changes in Elasticsearch.
Yes. Use the mutate filter's remove_field option to remove message from an event. Do this after extracting the fields you need, and retain the original text when parsing fails.
Remove it after a successful parse
The filter's common remove_field option runs when that filter succeeds, so it can be attached directly to grok:
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
remove_field => ["message"]
}
}
For a separate cleanup step, the following pattern is useful when this grok filter is the only producer of the default parse-failure and timeout tags in the event:
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
if "_grokparsefailure" not in [tags] and "_groktimeout" not in [tags] {
mutate { remove_field => ["message"] }
}
}
Grok timeouts use _groktimeout; checking only for _grokparsefailure can miss that failure path. Keep the filter-attached removal in the first example when possible.
An unconditional mutate { remove_field => ["message"] } deletes the field even when parsing failed. In pipelines with several parsers, use a parser-specific success or failure tag so one stage does not accidentally decide another stage's cleanup policy.
Decide what you still need
Remove the raw text only when the parsed fields preserve the information needed for investigation and retention requirements. Keeping it gives you a fallback for unrecognized formats and a way to reprocess events later.
Full-text search continues to work on remaining fields mapped and indexed for that purpose. It cannot search text that you removed. Field removal affects events processed by the new configuration; it does not erase old Elasticsearch documents or copies retained elsewhere.
Storage savings depend on the message, mappings, compression, and the fields retained. Compare equivalent batches under the same index settings instead of treating the raw message length as a guaranteed saving.
References
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan