Hi Jordan, thanks a lot for your feedback! There are definitely other alternatives for pushing HTTP traffic logs to Kafka: the tutorial shown here was meant to be as lightweight as possible so it only uses Node.js and Kafka. I agree that a more solid, production-ready pipeline could be built with, for example, Logstash (which I love, unfortunately I've never used fluentd so I can comment on that). If you're already writing server traffic logs somewhere in your environment, you may want to build your pipeline from those. But if you are, for example, still prototyping analytics pipelines based on traffic logs, a simple and straightforward solution as shown here could be in place as it only requires adding one middleware in code. That's what I think, I'm curious to hear your thoughts!
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Interesting concept adding express middleware to handle that use-case.
Had you explored alternatives like Logstash or fluentd's file-parsing + http enrichment + Kafka output capabilities?
Hi Jordan, thanks a lot for your feedback! There are definitely other alternatives for pushing HTTP traffic logs to Kafka: the tutorial shown here was meant to be as lightweight as possible so it only uses Node.js and Kafka. I agree that a more solid, production-ready pipeline could be built with, for example, Logstash (which I love, unfortunately I've never used fluentd so I can comment on that). If you're already writing server traffic logs somewhere in your environment, you may want to build your pipeline from those. But if you are, for example, still prototyping analytics pipelines based on traffic logs, a simple and straightforward solution as shown here could be in place as it only requires adding one middleware in code. That's what I think, I'm curious to hear your thoughts!