This is a list of the top 5 open-source log shippers to fit your needs.
Many of these can be considered as “Logstash alternatives” in different ways. For those of you who don’t know it yet, Logstash is highly popular among DevOps for the possibility of ingesting data from different sources, dynamic data transformation, and much more.
However, it might not be the right choice for everyone. If you are looking for open-source alternatives and to choose what’s best for you - this guide is here to help you.
Here is the quick list:
- Logstash
- Vector
- Filebeat
- FluentD
- Promtail
For each log shipper, I also took some of the core criteria, like number of plugins and configuration difficulty, and you can find the full comparison table at the end of this blog post.
Logstash
The issue with Logstash
Let’s start with the king of log collectors. If you know about Logstash, feel free to scroll lower for other options.
As commonly known, Logstash is part of the famous ELK stack and acts as a workhorse in it. It is used to collect, parse, and ship your logs. However, due to its high workload, it shows low figures in performance tests (see ref.1 below) compared to other, more lightweight options.
As noted in the blog post by Logz.io, one of the reasons for low performance is that Logstash requires JVM to run, this dependency causes significant memory consumption. This is especially true when you have many data pipelines and advanced filtering. That is one of the reasons why Filebeat was created, and I will discuss more on that later on, however feel free to dive into the blog post by Logz.io, which compares Filebeat and Logstash in detail.
As for the performance comparison of Logstash vs others, Vector has an insightful benchmark TCP To Blackhole Performance Test (ref.1) that compares Vector, Logstash, FluentD, and FluentBit.
$ bin/compare -t tcp_to_blackhole_performance
| Metric | fluentbit | fluentd | logstash | vector |
|:----------------|:----------|:----------|:----------|:----------|
| IO Thrpt (avg) | 64.4MiB/s | 27.7MiB/s | 40.6MiB/s | 86MiB/s W |
| CPU sys (max) | 4 | 3.5 W | 6.1 | 6.5 |
| CPU usr (max) | 53.2 | 50.8 W | 91.5 | 96.5 |
| Load 1m (avg) | 0.5 W | 0.8 | 1.8 | 1.7 |
| Mem used (max) | 614.8MiB | 294MiB | 742.5MiB | 181MiB W |
| Disk read (sum) | 9MiB | 2.6MiB W | 2.6MiB | 2.6MiB |
| Disk writ (sum) | 14.8MiB | 13.7MiB | 11.6MiB | 11MiB W |
| Net recv (sum) | 3.9gib | 1.7gib | 2.4gib | 5.1gib W |
| Net send (sum) | 7.9MiB | 5.7MiB | 2.6MiB | 9MiB |
| TCP estab (avg) | 663 | 664 | 665 | 664 |
| TCP sync (avg) | 0 | 0 | 0 | 0 |
| TCP close (avg) | 1 | 2 | 7 | 4 |
-------------------------------------------------------------------------------------------------------------
W = winner
fluentbit = 1.1.0
fluentd = 3.3.0-1
logstash = 7.0.1
vector = 0.2.0-6-g434bed8
Reference 1, TCP to Blackhole Performance Test, Source: Vector Repo
Logstash Pros & Cons
Here is a list of pros & cons for Logstash (in comparison to other log shippers in this guide):
| Pros | Cons |
|---|---|
| Input plugins: Many | Configuration difficulty: High |
| Output plugins: Many | Resource-usage: High |
| Configuration capabilities: High | Documentation: Complex |
| Community activity: High (68 PRs over the last month) |
Vector
Vector is a tool used to collect, transform, and route all your logs and metrics. It was created by TimberIO in 2019, and acquired by Datadog in 2021.
Even though Vector is an end-to-end agent & aggregator, it can still be used as a log shipper/collector, and with its impressive benchmarks, it could become a significant tool in your stack.
The great part about Vector is that it is written in Rust, which is known for its performance, memory safety, and is designed for high-intensity work. Vector also provides distributed, centralized, and stream-based deployment.
To compare Vector based on features with other log shippers, the Vector repo has a good side-by-side comparison (reference 2 below) with Beats, Fluentbit, FluentD, Logstash, Splunk UF, and Splunk HF, which I added below for the reference.
Reference 2, Vector’s Features, Source: Vector Repo
The previously discussed benchmark especially caught my eye and it's a great foundation to compare Vector's performance against other log shippers. In short, Vector wins Logstash, FluentD, and Fluentbit in IO Thrpt (avg), Mem used (max), Disk writ (sum), and Net recv (sum) in TCP to Blackhole test.
Vector Pros & Cons
Here is a list of pros & cons for Vector (in comparison to other log shippers in this guide):
| Pros | Cons |
|---|---|
| Resource-usage: Low | Input plugins: Fewer |
| Configuration difficulty: Low | Output plugins: Fewer |
| Configuration capabilities: High | |
| Documentation: Simple | |
| Community activity: High (220 PRs over the last month) |
Filebeat
Another option is Filebeat. This one was created with a goal in mind to be a lightweight alternative to Logstash, if you don’t need advanced tuning. Even though you can use both Logstash and Filebeat, you do have an option to send your logs directly with Filebeat to your centralized logging platform. As you get more familiar or require additional features, you can always connect Logstash later on to your process.
If you want to collect logs on remote machines, Filebeat is a great option. And if you don’t need to make transformations to your data then you are free to send it straight to Elasticsearch! But more often than not, if you need more than just timestamp and message fields, you will need Logstash.
Overall, it is a much simpler option compared to Logstash. This makes it more reliable for new users, as you have less gears and handles to spin.
For scaling Filebeat, most likely than not you would still require Logstash in your stack. However, in this case, it is important to have a well-designed / architectured cluster.
For your stack, you might not only use Filebeat. This is especially useful if you are considering scaling. Below is a representation of what your process might look like at scale with Beats, Logstash, Elasticsearch, and Kibana.
Source: Elastic.co, Guide on Deploying and Scaling Logstash
Filebeat Pros & Cons
Here is a list of pros & cons for Filebeat (in comparison to other log shippers in this guide):
| Pros | Cons |
|---|---|
| Resource-usage: Low | Input plugins: Fewer |
| Configuration difficulty: Low | Output plugins: Fewer |
| Documentation: Simple | Configuration capabilities: Low |
| Community activity: High (220 PRs over the last month) |
FluentD
This one was built with the idea in mind to structure data in JSON as much as possible. FluentD claims that this method allows for a Unified Logging Layer (in other words, unifying logging infrastructure).
Due to a vast number of plugins, FluentD is a great choice for those that have data from different or unique sources.
Fluentbit is also an option to consider if you are looking for something that fits more small devices & distributed systems. Just like FluentBit, it was created by Treasure Data, but later on in 2015. There is a Side by Side comparison of FluentD and Fluentbit by Logz.io, which I advise you to check out.
There is an insightful article with multiple tests comparing FluentD and FluentBit, one of which is Forwarding 5,000 1KB events per second, which resulted in 80% CPU and 120MB Memory for FluentD, but 27% CPU and 26MB Memory for FluentBit.
However, when comparing FluentD to other log shippers in this guide, it performs quite well.
FluentD Pros & Cons
Here is a list of pros & cons for FluentD (in comparison to other log shippers in this guide):
| Pros | Cons |
|---|---|
| Resource-usage: Low | Configuration difficulty: High |
| Input plugins: Many | Community activity: Low (6 PRs over the last month) |
| Output plugins: Many | Documentation: Complex |
| Configuration capabilities: High |
Promtail
For those that know Loki, you have probably heard of Promtail. Its use case is specifically tailored to Loki, and it can collect logs both locally and for Kubernetes pods.
As for comparison, you could refer to an article by CrashLaker, which compares Loki vs ELK vs Splunk. On top of it, there is also an insightful Medium article by Ronen Schaffer, which analyses Promtail including write path performance, read path performance, and much more.
Promtail Pros & Cons
Here is a list of pros & cons for Promtail (in comparison to other log shippers in this guide):
| Pros | Cons |
|---|---|
| Resource-usage: Low | Input plugins: Fewer |
| Configuration difficulty: Low | Output plugins: Fewer |
| Documentation: Simple | Configuration capabilities: Low |
| Community activity: Low (people wait for months to receive a reply and only 8 PRs over the last month) |
Conclusion
Let’s conclude with a comparison table that includes all of the above mentioned log shippers:
| Logstash | Vector | Filebeat | FluentD | Promtail | |
|---|---|---|---|---|---|
| Resource-usage | High | Low | Low | Medium | Low |
| Input plugins | High | Fewer | Fewer | Fewer | Little |
| Output plugins | Many | Fewer | Fewer | Many | Fewer |
| Configuration capabilities | High | High | Low | High | Low |
| Configuration difficulty | High | Low | Low | High | Low |
| Documentation | Complex | Simple | Simple | Complex | Simple |
| Community activity | High | High | High | Low | Low |
I hope this guide helped you to analyse just the surface of features for top log shippers, as to see which solution you should explore further and add to your stack.
Let me know if you have any questions about any of the tools, I will see the comment and reply asap! Thanks for reading.
Top comments (0)