DEV Community

loading...

Migrating Apache Flume Flows to Apache NiFi: SYSLOG to KAFKA

Timothy Spann
I am a Developer Advocate at Streamnative. I work with Apache Pulsar, Apache Flink, Apache NiFi, Apache Spark, IoT, MXNet, DLJ.AI, Deep Learning, Machine Learning, Streaming...
Originally published at datainmotion.dev on ・2 min read

Migrating Apache Flume Flows to Apache NiFi: SYSLOG to KAFKA

Article 5 - This

*Article 4 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html

*Article 3 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html

*Article 2 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html

Article 1 - https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

*Source Code: * https://github.com/tspannhw/flume-to-nifi

This is a simple use case of being a smart gateway/proxy between SYSLOGand Kafka. We can do a lot more than that in NiFi. We can be a Kafka Consumer and Producer as well as read and parse all types of logs including SYSLOG. We have a GrokReader for converting semistructured logs into manageable tabular style data with schemas. Log->JSON/CSV/AVRO/PARQUET. All with No Code.

It is extremely easy to do this in NiFi.

See: Log Parsing: https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

Detailed Tutorials

https://blog.davidvassallo.me/2018/09/19/apache-nifi-from-syslog-to-elasticsearch/

https://community.cloudera.com/t5/Community-Articles/NiFi-Send-to-syslog/ta-p/248638

http://www.youritgoeslinux.com/impl/bigdata/nifi/syslog

Processors

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ListenSyslog/

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ParseSyslog/index.html

References

Discussion (0)