Processing Heterogeneous IoT Logs with Fluent Bit and Elasticsearch

Christophe Chaussat — Thu, 10 Sep 2026 15:41:55 +0000

A lightweight two-stage approach for parsing, preprocessing and storing semi-structured home automation data

The full configuration example is available on GitHub.

Home automation systems tend to generate a surprisingly heterogeneous collection of logs and measurements.

In this project, most data originates from Domoticz, but additional information is produced by ESP8266/D1 devices, alarm scripts, monitoring tools and other IoT components. Some records are plain text, some contain timestamps and metadata, while others contain JSON embedded inside a log line.

Rather than forcing every producer into a single format, I use a two-stage processing architecture based on Fluent Bit and Elasticsearch.

The complete chain is:

                    IoT / home automation sources
                              |
             +----------------+----------------+
             |                |                |
          Domoticz          ESP/D1         Other scripts
             |                |                |
             +----------------+----------------+
                              |
                         Fluent Bit
                              |
                    parsing + preprocessing
                              |
                              v
                        Elasticsearch
                              |
                       ingest pipeline
                              |
                     final JSON decoding
                              |
                              v
                         dom_v3 index
                              |
                   +----------+----------+
                   |                     |
                 Kibana               Grafana

The complete pipeline normally has a latency of less than one minute in the author's installation.

1. Why two processing stages?

The main design decision is to avoid making Fluent Bit responsible for every transformation.

Fluent Bit performs the first level of processing:

reading the various log files;
recognizing several recurring line formats;
extracting timestamps and metadata;
decoding JSON when it is directly available;
removing intermediate fields;
forwarding the resulting records to Elasticsearch.

Elasticsearch then performs the final JSON decoding through an ingest pipeline.

This separation keeps the Fluent Bit configuration relatively simple while giving Elasticsearch responsibility for the final document transformation.

2. Fluent Bit parsers

The incoming data belongs to several recurring families. Instead of trying to recognize all of them with one very large regular expression, the configuration uses several smaller parsers.

A typical Domoticz format is:

[PARSER]
    Name    domoticz_parser_1
    Format  regex
    Time_Key    dom_timestamp
    Time_Format %Y-%m-%d %H:%M:%S.%L
    Time_Offset +0200
    Regex   ^(?<dom_timestamp>.*)  (?<dom_source>\w+)\: (?<msgtxt>.*)$
    Decode_Field json msgtxt

This extracts the timestamp and source and decodes the JSON contained in msgtxt.

Another Domoticz format is handled separately:

[PARSER]
    Name    domoticz_parser_2
    Format  regex
    Time_Key    dom_timestamp
    Time_Format %Y-%m-%d %H:%M:%S.%L
    Time_Offset +0200
    Regex   ^(?<dom_timestamp>.*)  \((?<dom_hardware>\w+)\) (?<dom_category>.*) \((?<dom_device>\w+)\)$

Generic JSON-bearing records are covered by additional parsers, including a final fallback for pure JSON.

The complete examples are provided in fluent-bit/parsers.conf.

3. Collecting the different sources

Fluent Bit's tail input monitors the individual log files. Each source receives its own tag and persistent database:

[INPUT]
    Name   tail
    Tag    domoticz
    Path   /path/to/domoticz.log
    DB     /var/spool/fb_domoticz.db

The real installation contains several additional sources for ESP/D1 devices, alarms, metrics, monitoring and Tuya-related data.

The tag provides a simple way of retaining the origin of each record.

4. Applying several parsers

The Domoticz records are processed with the parser filter:

[FILTER]
    Name      parser
    Match     domoticz
    Key_Name  log
    Parser    domoticz_parser_1
    Parser    domoticz_parser_2
    Parser    domoticz_parser_3
    Parser    domoticz_parser_4
    Parser    domoticz_parser_5
    Reserve_Data On

The individual expressions act as recognizers for recurring formats. This is easier to maintain than one large regular expression attempting to describe every possible record.

Reserve_Data On keeps the original record available while parsed fields are added.

Intermediate fields can then be removed:

[FILTER]
    Name        record_modifier
    Match       domoticz
    Remove_key  msgtxt

5. Sending the preprocessed data to Elasticsearch

The resulting records are sent to a common Elasticsearch index:

[OUTPUT]
    Name                es
    Match               *
    Host                127.0.0.1
    Port                9200
    HTTP_User           elastic
    HTTP_Passwd         <set-securely>
    tls                 On
    tls.verify          Off
    Include_Tag_Key     On
    Tag_Key             fb_tag
    Index               dom_v3
    Type                _doc
    Suppress_Type_Name  On
    Retry_Limit         5

The example deliberately does not contain a real password.

The tls.verify Off setting is specific to the author's local installation and should not be copied blindly. A network-accessible Elasticsearch instance should normally use proper certificate verification.

6. Final JSON processing in Elasticsearch

The second stage is implemented as an Elasticsearch ingest pipeline:

{
  "description": "dom_v3 ingestion pipeline v1",
  "processors": [
    {
      "json": {
        "ignore_failure": true,
        "add_to_root": true,
        "field": "log"
      }
    }
  ]
}

The purpose is to decode JSON that remains in the log field and add the decoded fields to the document root.

For example, a simplified intermediate document can contain:

{
  "log": "{\"temperature\":21.7,\"humidity\":48}",
  "dom_timestamp": "2026-09-10 10:23:41.123",
  "dom_source": "Sensor",
  "fb_tag": "domoticz"
}

After the ingest pipeline, the document can contain:

{
  "dom_timestamp": "2026-09-10 10:23:41.123",
  "dom_source": "Sensor",
  "temperature": 21.7,
  "humidity": 48,
  "fb_tag": "domoticz"
}

This example is deliberately simplified; the actual records may contain substantially more nested information.

7. Why not do everything in Fluent Bit?

It would be possible to push more processing into Fluent Bit. In this application, however, the separation proved more convenient.

Fluent Bit is used for:

collection;
line-oriented recognition;
regular-expression parsing;
timestamp extraction;
first-level JSON decoding;
metadata and filtering.

Elasticsearch is used for:

final document transformation;
JSON decoding;
indexing and mapping;
subsequent search and visualization.

This keeps each component focused on the type of work it handles well.

8. Mappings matter

JSON decoding does not automatically guarantee that every field will have the desired Elasticsearch type.

Measurements such as temperature and humidity should be indexed as numeric fields, while timestamps should use an appropriate date mapping.

This is especially important when the resulting data is queried or visualized by Grafana.

For a production deployment, index mappings should therefore be designed explicitly for the metrics expected from the installation.

9. Why this approach works well for home automation

Home automation installations rarely produce perfectly homogeneous data.

A single installation may contain:

application logs;
sensor measurements;
alarm events;
device status;
network information;
JSON APIs;
diagnostic messages;
custom scripts.

Trying to force every producer to emit exactly the same format can require unnecessary changes to individual applications.

The approach described here instead accepts heterogeneity at the input and progressively transforms it into structured Elasticsearch documents.

Adding another source generally requires another Fluent Bit input and, when necessary, another parser.

10. Result

The architecture has proved sufficient for the author's home automation environment:

Various IoT sources
        |
        v
   Fluent Bit
        |
        +-- regex parsing
        +-- timestamp extraction
        +-- JSON preprocessing
        +-- metadata
        +-- filtering
        |
        v
  Elasticsearch
        |
        +-- ingest pipeline
              |
              +-- final JSON decoding
        |
        v
      dom_v3
        |
   +----+----+
   |         |
 Kibana    Grafana

The complete chain normally has a latency of less than one minute in this installation.

The main advantage is not raw processing complexity, but keeping the architecture understandable:

Fluent Bit collects and recognizes.

Elasticsearch structures and stores.

Kibana and Grafana visualize.

Conclusion

For heterogeneous IoT and home automation data, combining Fluent Bit with Elasticsearch ingest pipelines provides a practical way of progressively converting unstructured log streams into structured data.

The key design principle is to avoid making any single component responsible for the complete transformation.

The resulting architecture is small enough for a home automation environment while remaining flexible enough to accommodate additional sensors, scripts and data sources.

Acknowledgements

This article was prepared by Christophe Chaussat, based on his own home automation installation, configuration and experiments.

ChatGPT (OpenAI) assisted with the structuring, editing and technical writing of this documentation. The technical architecture and configuration described here are based on the author's own installation and experiments.

Licensing

The configuration and software examples available on GitHub (cchaussat/fluent-bit-elasticsearch-iot-logging) are released under the MIT License.

This article and the accompanying documentation are released under the Creative Commons Attribution 4.0 International License (CC BY 4.0).

DEV Community: Christophe Chaussat