Building Search Indexes with Apache NiFi Streams
I can go to my Apache Solr dashboard hosted on Cloudera Data Platform - Data Center via port :8983/solr/#/
For those who have done the CFM or CDF Hands on Labs, we extend that lab to store to SOLR.
Login to Solr
Create a Collection for Your Data (sensors)
NiFi Flow (Data in, Output Records to Solr)
We need to configure the processor. Just set Solr Type to Cloud, set Solr Location to URL:2181/solr and then pick your record reader ours is JSON. You also need to select Fields to Index in the data. In our JSON file I pick the timestamp and Id.
View The Collection To See Data Sent From NiFi
** To Query Via REST API**
http://YOURURL:8983/solr/sensors/select?q=*%3A*
Example Collection Data
{ "responseHeader":{ "zkConnected":true, "status":0, "QTime":2, "params":{ "q":"*:*", "_":"1587566383627"}}, "response":{"numFound":153,"start":0,"docs":[{ "sensor_id":[73], "sensor_ts":[1587566202642550], "id":"e8652da3-dddf-4735-a187-799f95d2e362", "_version_":1664683988343586816}, { "sensor_id":[22], "sensor_ts":[1587566203806855], "id":"9bae2dff-e5ec-4751-b20e-f4d3a9c6e894", "_version_":1664683988399161344}, { "sensor_id":[10], "sensor_ts":[1587566204972747], "id":"4cc9dd26-e559-4368-b0c2-47095a0f78ab", "_version_":1664683988407549952}, { "sensor_id":[48], "sensor_ts":[1587566206142079], "id":"ba73b5d7-4c06-4202-8145-0f3437f37759", "_version_":1664683988415938560}, { "sensor_id":[9], "sensor_ts":[1587566207304793], "id":"be85793a-1c3c-46ab-96d8-14057e6a2e27", "_version_":1664683988422230016}, { "sensor_id":[77], "sensor_ts":[1587566208471159], "id":"94a1a150-2836-4b60-aafd-9b09fe2ac62a", "_version_":1664683988428521472}, { "sensor_id":[75], "sensor_ts":[1587566209635840], "id":"729b488c-7a75-49a8-9121-3e192bd84a81", "_version_":1664683988434812928}, { "sensor_id":[64], "sensor_ts":[1587566210807450], "id":"e6f76296-f70a-430d-a532-8040fbf4139e", "_version_":1664683988440055808}, { "sensor_id":[1], "sensor_ts":[1587566211973630], "id":"a3cb1028-76d1-4dba-a2ef-03b267d6714b", "_version_":1664683988447395840}, { "sensor_id":[33], "sensor_ts":[1587566213135150], "id":"dea6ea44-28ab-4d6e-9fad-b0ffaef69dcb", "_version_":1664683988454735872}] }}
Example Code
https://github.com/tspannhw/nifi-solr-example
References
https://lucene.apache.org/solr/
https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/search_introducing.html
https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/cm_mc_solr_service.html
https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/search_tutorial.html
Top comments (0)