Migrating Apache Flume Flows to Apache NiFi: SYSLOG to KAFKA

Migrating Apache Flume Flows to Apache NiFi: SYSLOG to KAFKA

Article 5 - This

*Article 4 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html

*Article 3 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html

*Article 2 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html

Article 1 - https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

*Source Code: * https://github.com/tspannhw/flume-to-nifi

This is a simple use case of being a smart gateway/proxy between SYSLOGand Kafka. We can do a lot more than that in NiFi. We can be a Kafka Consumer and Producer as well as read and parse all types of logs including SYSLOG. We have a GrokReader for converting semistructured logs into manageable tabular style data with schemas. Log->JSON/CSV/AVRO/PARQUET. All with No Code.

It is extremely easy to do this in NiFi.

See: Log Parsing: https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

Detailed Tutorials

https://blog.davidvassallo.me/2018/09/19/apache-nifi-from-syslog-to-elasticsearch/

https://community.cloudera.com/t5/Community-Articles/NiFi-Send-to-syslog/ta-p/248638

http://www.youritgoeslinux.com/impl/bigdata/nifi/syslog

Processors

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ListenSyslog/

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ParseSyslog/index.html

References

• https://community.cloudera.com/t5/Community-Articles/Su-Su-Sussudio-Sudoers-Log-Parsing-with-Apache-NiFi/ta-p/249461