In the last 24 hours, two people running Wazuh 4.14.7 in production hit the same defect from different directions, and neither of them saw it on a dashboard. They found it by going looking in a log.
The defect is simple to state. The alerts index template gives each field path one type. Some integrations send more than one shape of data under the same path. When an event arrives in the shape the template did not expect, the indexer rejects that document with mapper_parsing_exception, Filebeat gets a 400 back, and a 400 is not the kind of error a shipper retries. The event is gone. Nothing alerts on it, because the thing that would alert is the thing that never got stored.
Three paths so far. data.ms-graph.status is mapped as keyword, and some Microsoft Graph events send an object there. data.office365.ModifiedProperties is also keyword, and some Office 365 events send an object; one operator counted 35 of those in a single Filebeat log. The third runs the other way: an override that mapped data.data as an object then met events that send the string "[]" in the same place.
The useful part is that this is one class of bug, not three. Any path that can carry two shapes will drop the minority shape silently, so the audit is a count of failures by field name, not a hunt for a specific field:
grep -o 'failed to parse field \[[^]]*\]' /var/log/filebeat/filebeat* | sort | uniq -c | sort -rn
An empty result is only meaningful if Filebeat logs at a level that records the rejection, so check that one known-bad event shows up before trusting a zero.
On the fix, put the override in a template of its own with a higher order instead of editing the wazuh template, which the documented upgrade procedure reinstalls. On 4.14.7 two legacy templates with the same pattern merge and the higher order wins the field; that part I measured. Remember a template only applies when an index is created, so today's index keeps the old mapping and keeps dropping until tomorrow's index exists. And before you flip a path from keyword to object, run an exists query on it: every hit is an event that indexed fine as a string, and flipping the type moves the 400s onto those. If both shapes are real, enabled: false accepts both (one of the two operators measured this on status), at the price of not being able to query inside that field.
If you run the grep and get a path that is not on this list, I'd like to hear which one.
Top comments (0)