According to Forrester Research, adoption interest for streaming data analytics has grown 62%, during 2012 to 2014. This momentum is the need for today’s data-driven culture. It is truly said that you can not cross a road with heavy traffic just by looking at the snapshot taken few minutes back!
Today our sensors, IoTs, mobile devices, clickstreams and transaction data are flooding our logs and data warehouses at overwhelming speed. Traditional approach lands all the data into data lakes and analyses it after weeks or months to get the insight. Sooner the insights you can get from historic data, better decisions you can make using those insights. Such insights let you identify risks or opportunities on time and take action accordingly. Forrester refers to such insights as ‘perishable insights’, as it has its own shelf life. There is a close gap between identifying risks or opportunities and taking action.
For example, an application that monitors temperature sensors or pressure sensors activities. You need to take action in that moment when things my go wrong with the temperature or pressure sensor activities are unusual. As an another use case of it, think of an application that is able to identify potential fraud credit card transaction in real-time. To be able to map such activity you will have to manage huge amount of transaction data for hundreds of customers and take action quickly as soon as any fraud activity is discovered. You must take action within a very short time-window to minimize the loss.
So, to be able to identify and leverage such perishable insights you must be able to build a data model and integrate it with your existing systems. Hadoop is becoming de-facto due to its scalable distributed processing powers and other technologies like Apache Storm, Apache Kafka and Apache Spark. These technologies have opened up many more opportunities for enterprises to adopt stream analytics quickly. However, for each business use case time-window for perishable insights may vary from a few minutes to microseconds.
Usual insights by mining historical data may help you take better long-term business decisions, whereas short-term business decisions depends on your ability to identify perishable insights. Today’s technologies like Hadoop implemented with thoughtful architecture can offer you the best of both the world. Apart from that, adoption of streaming data analytics depends on architectural aspects like performance, scalability and availability that can be achieved by leveraging existing systems.
