Migrating Apache Flume Flows to Apache NiFi: Twitter Source to Kafka Sink

Migrating Apache Flume Flows to Apache NiFi: Twitter Source to Kafka Sink

Article 4 - This.

*Article 3 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html

*Article 2 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html

Article 1 - https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

*Source Code: * https://github.com/tspannhw/flume-to-nifi

This is a simple use case of pushing Tweets to Kafka. We can do a lot more than that in NiFi. If you see the referenced article I can do Deep Learning on Tweet Images, Run Sentiment Analysis, Query the Tweets in Stream, Send messages to email / slack based on certain criteria and retweet automagically. All with No Code.

It is extremely easy to do this in NiFi.

Create a Kafka Topic To Send Tweets To

Example Tweet

Configure Your Connection to Twitter

Configure Kafka Producer

NiFi Flow to Send Two Twitter Sources To Kafka

Additional Apache NiFi Flow Details For Other Processing

Kafka Messages in Topic Sent From NiFi Producer

Source Code

https://github.com/tspannhw/flume-to-nifi

References

• https://community.cloudera.com/t5/Community-Articles/Automating-Social-Media-Sending-Tweets-with-Apache-NiFi/ta-p/248274