I have been around Big Data for my entire career. Having worked with over 500 end client engagements, a constant has been that every company, big or small, is awash with data. Yet, when speaking with the executive team, very few will admit to harnessing this data into constructive insights. This dilemma may very well be the twin events of rapidly less costly storage, and accelerated data capture of unstructured data such as web log files. The analogy I make is everyday a company refills their pens with cattle, sheep and pigs; we as marketers, need to render this mass of meat into something more useful, such as ground sirloin, lamb chops or pork ribs.
Big Data as a concept is now in its first decade. While IT has been enamored with container like tools, the business owners have needed a smart suite of tools that provide predictive analytics to empower campaigns, data mining tools that uncover relevancy amongst customer segments, and visualization tools to present results back up the chain. Container strategies work best when the data can be staged in an efficient manner. Sadly, most Big Data efforts fail because the data scientists spend 90 percent of their time rebuilding, and 10% of their time actually working on business insights. It is like walking toward a mountain peak, in snow shoes, in deep snow.
I have recently met up with a “next wave” of toolset companies that work in the “in memory” space. Simply put, these companies can rapidly review billions of records, and provide true speed of results. The companies are AuriQ, Data Torrent and Speedtrack. I am very excited by these technologies, especially that companies can now focus on their end results and missions. Looking down from the mountaintop is always sweeter than the climb up.