DEV Community

Timothy Spann.   ๐Ÿ‡บ๐Ÿ‡ฆ
Timothy Spann. ๐Ÿ‡บ๐Ÿ‡ฆ

Posted on • Originally published at datainmotion.dev on

Migrating Apache Flume Flows to Apache NiFi: Kafka Source to HTTP REST Sink and HTTP REST Source to Kafka Sink

Migrating Apache Flume Flows to Apache NiFi: Kafka Source to HTTP REST Sink and HTTP REST Source to Kafka Sink

Article 6 - This.

*Article 5 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_35.html

*Article 4 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_8.html

*Article 3 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_7.html

*Article 2 - * https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html

Article 1 - https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html

*Source Code: * https://github.com/tspannhw/flume-to-nifi

This is a simple use case of being a gateway between REST API and Kafka. We can do a lot more than that in NiFi. We can be a Kafka Consumer and Producer as well as POST REST calls and receive any REST calls on configurable ports. All with No Code.

NiFi can act as a listener for HTTP Requests and provide HTTP Responses in a scriptable full Web Server mechanism with JETTY. Or it can listen for HTTP REST calls on a port and route those files anywhere. https://community.cloudera.com/t5/Community-Articles/Parsing-Web-Pages-for-Images-with-Apache-NiFi/ta-p/248415 . We can also do websockets https://community.cloudera.com/t5/Community-Articles/An-Example-WebSocket-Application-in-Apache-NiFi-1-1/ta-p/248598. https://community.cloudera.com/t5/Community-Articles/Accessing-Feeds-from-EtherDelta-on-Trades-Funds-Buys-and/ta-p/248316

It is extremely easy to do this in NiFi.

Kafka Consumer to REST POST

HTTP REST to Kafka Producer

Full Monitoring on Apache NiFi

A Very Common Use Case: Ingesting Stock Feeds From REST to Kafka

https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221

References

Top comments (0)