loading...

Migrating Apache Flume Flows to Apache NiFi: Twitter Source to Kafka Sink

tspannhw profile image Timothy Spann Originally published at datainmotion.dev on ・3 min read

Migrating Apache Flume Flows to Apache NiFi: Twitter Source to Kafka Sink

Article 4 - This.

*Article 3 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html

*Article 2 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html

Article 1 - https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

*Source Code: * https://github.com/tspannhw/flume-to-nifi

This is a simple use case of pushing Tweets to Kafka. We can do a lot more than that in NiFi. If you see the referenced article I can do Deep Learning on Tweet Images, Run Sentiment Analysis, Query the Tweets in Stream, Send messages to email / slack based on certain criteria and retweet automagically. All with No Code.

It is extremely easy to do this in NiFi.

Create a Kafka Topic To Send Tweets To

Example Tweet

Configure Your Connection to Twitter

Configure Kafka Producer

NiFi Flow to Send Two Twitter Sources To Kafka

Additional Apache NiFi Flow Details For Other Processing

Kafka Messages in Topic Sent From NiFi Producer

Source Code

https://github.com/tspannhw/flume-to-nifi

References

Posted on by:

tspannhw profile

Timothy Spann

@tspannhw

I am a Principal Field Engineer for Data in Motion at Cloudera. I work with Apache NiFi, Apache Kafka, Apache Spark, Apache Flink, IoT, MXNet, DLJ.AI, Deep Learning, Machine Learning, Streaming...

Discussion

markdown guide