This demonstration simulates real time set top box data capture with simple event processing and real time search & discovery.
Products used: Java, Kafka, SOLR/Banana, NiFi
-
Get Hortonworks Sandbox: http://hortonworks.com/products/hortonworks-sandbox/
-
Install NiFi (https://github.com/abajwa-hw/ambari-nifi-service). If you're using VirtualBox, make sure you port forward 9090 when your install is complete.
-
Start SOLR from /root
./start_solr.sh
-
Clone this repository in /root
-
Change directory into the cloned repo
cd SetTopBox
-
If not using /root/SetTopBox as primary directory; update DemoData.properties file. Namely, these settings...
settopboxdemo.data.input.program_guide=/root/SetTopBox/DemoData_ProgramGuide.csv settopboxdemo.kafka.input_file=/root/SetTopBox/input_test.txt settopboxdemo.data.output.events=/root/SetTopBox/input_test.txt
-
Load NiFi settopbox template into NiFi. File is located in /root/SetTopBox/nifi/NiFi_SetTopBox.xml. Load the template from the NiFi UI and then drag the SetTopBox template onto the canvas.
-
Once loaded, start NiFi processors.
-
Create settopbox SOLR index:
Make sure a response status of 0 is returned.
-
Copy the Set Top Box dashboard into the Solr Banana web app.
mv /opt/lucidworks-hdpsearch/solr/server/solr-webapp/webapp/banana/app/dashboards/default.json /opt/lucidworks-hdpsearch/solr/server/solr-webapp/webapp/banana/app/dashboards/default.json.bkp cp /root/SetTopBox/banana/SetTopBox_Dashboard.json /opt/lucidworks-hdpsearch/solr/server/solr-webapp/webapp/banana/app/dashboards/default.json
-
Validate the dashboard renders: http://127.0.0.1:8983/solr/banana/index.html#/dashboard
-
Create settopbox topic
sh /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --topic settopbox --zookeeper 127.0.0.1:2181 --partitions 1 --replication-factor 1
-
From /root/SetTopBox, run the following. This should create an input_text.txt file in /root/settopbox with some sample set top box data/
java -cp SetTopBox.jar com.hortonworks.settopboxdemo.DataGenerator DemoData.properties
To run the demo, the MessageSender class in the SetTopBox jar will incrementally push records into Kafka, thru Nifi and ultimately land in SOLR. MessageSender simulates a set top box end point, Kafka is obviously the message broker, NiFi is the router of these data streams into SOLR were real time event analytics can be visualized and searched upon.
java -cp SetTopBox.jar com.hortonworks.settopboxdemo.MessageSender DemoData.properties
--- EXTRAS ---
If you need to delete the SOLR index, run the following commands:
curl "http://localhost:8983/solr/admin/cores?action=UNLOAD&core=settopbox&deleteIndex=true"