InformationWeek reports that Big Data was the big news in New York last week at the sold out Strata Conference. There's a high degree of hype around big data, but there's also a high degree of innovation, tangible benefits, and venture capital backing. Coming from the perspective of the established business intelligence world, here's the skinny on where big data meets BI.
First, big data is more than Hadoop, the open source distributed file system capable of scaling to handle petabytes of data. Scalability is not the only appeal of Hadoop; it can also handle multi-structured data such as clickstreams, tweets, video, Facebook comments, sensor data, and so on. Such data is a challenge to model and store in a traditional relational database and data warehouse schema.
In the traditional BI world, technologies such as analytic appliances, columnar databases, and in-memory engines can also handle big data. It all depends on whether the challenge is volume and performance, variety and complexity, or combinations thereof.
Survival of the smartest has been prevalent theme in the recession. Some companies are still struggling to analyze sales and who's buying what.
In the last year, many BI vendors have announced support for Hadoop. Access to data in Hadoop has been through Hive, a virtual data warehouse for Hadoop that has its own query language, HiveQL. The thing is, HiveQL generates MapReduce jobs to get to the data in Hadoop. MapReduce is batch-oriented and slow, in contrast to BI which is supposed to be fast. But hey, we might be talking petabytes here, so maybe a little patience is reasonable.
So what's an impatient BI user to do? This is where the rest of the big data architecture comes into play. Querying vast volumes of granular data in Hadoop via Hive may be slow, so once an initial exploration is done, BI vendors will cache the results in their technology to ensure speed-of-thought analysis. This is where solutions like SAP Hana, MicroStrategy OLAP Services (for in-memory), SAS LASR server, the Tableau Data Engine, or the QlikView in-memory engine all come into play.
With Hadoop gaining traction, there are a number of new data visualization and exploration vendors that offer access to and exploration of data in Hadoop. Isn't that what BI vendors with Hadoop connectors do? Yes, but those BI vendors can also access data in a data warehouse, analytic appliance, or spreadsheet. Hadoop may or may not be part of the picture.
Survival of the smartest has been prevalent theme in the recession. Some companies are still struggling to analyze sales and who's buying what. In the big data economy, the analysis extends to who's interested in your products, who's influencing buying decisions, and who's not even engaged, but should be!