Real time streaming using Spark streaming, Kafka and Pyspark

Now a day real time streaming is a one of the most challenging task in our data industries. It might be sensor data, fleet tacking, traffic, cellular, website real time streaming or web crawling. You might need to analyze the data on your real time for your application or end user needs.

Spark stream and storm are kind of technologies which will help you a lot for your real time streaming purposes. My choice is spark streaming for real time streaming.

Here I have implemented a basic streaming application which will cover the Spark streaming, kafka and a basic crawling using python.

will be continued …