Integration of Manticore with Fluentbit

Introduction

Fluent Bit is an open source and multi-platform tool for logs processing and distribution.
Nowadays data comes from various sources and Fluent Bit can help you aggregate and process all your log data.
Now, Manticore also supports the use of Fluent Bit as a processing pipeline. This allows the collected and transformed data to be sent to Manticore.

Let's examine a simple example of indexing dpkg.log, a standard log file of the Debian package manager. The log itself has a simple structure, as shown below:

2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none>

Configuration

Here is an example of the Fluent configuration file one can use to work with Manticore:

[SERVICE]
    flush       1
    daemon      On
    log_level   info

[INPUT]
    name tail
    path /var/log/dpkg.log
    inotify_watcher false
    read_from_head true

[OUTPUT]
    name es
    match *
    host 127.0.0.1
    port 9308
    index  dpkg_log

Note that our example is meant to be run in Docker, so we start FluentBit in the daemon mode and with the INPUT inotify_watcher option disabled to avoid possible issues with the Docker environment which can lead to errors. Also, we assume that Manticore is started on the default http port 9308.

Results

Now you can just run Fluentbit using the config above. The data from the dpkg log will be passed to Manticore and properly indexed.

Here is the resulting schema of the created table and an example of the inserted document:

mysql> DESCRIBE dpkg_log;
+-------------+--------+----------------+
| Field       | Type   | Properties     |
+-------------+--------+----------------+
| id          | bigint |                |
| @timestamp  | text   | indexed stored |
| log         | text   | indexed stored |
+-------------+--------+----------------+

mysql> SELECT * FROM dpkg_log LIMIT 3\G
*************************** 1. row ***************************
id: 7856533729353662465
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 startup archives install
*************************** 2. row ***************************
id: 7856533729353662466
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 install base-passwd:amd64 <none> 3.5.47
*************************** 3. row ***************************
id: 7856533729353662467
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 status half-installed base-passwd:amd64 3.5.47

Conclusion

The integration of Manticore with Fluent Bit provides a powerful and efficient solution for handling and indexing log data, making it more accessible and manageable for various applications. With this simple configuration and clear examples provided, even those new to these tools can quickly get started and benefit from the robust capabilities of Manticore and Fluent Bit working together. Whether you're dealing with standard logs or more complex data sources, this collaboration simplifies the process and opens up new possibilities for effective data management.