InformationWeek reports that only one number mattered to the data analysis aficionados watching the presidential election results on Tuesday night. That number, 538, is Nate Silver's blog (now under the New York Times auspices). Silver, by using predictive analytics applied against a range of polling and related data, hit a perfect 50 for 50 in his state-by-state predictions.
Remember, this was in a race where the pundits for the losing side were confident in their landslide-win predictions, and pundits on the winning side were predicting a razor-thin victory margin where vote counts and recounts could stretch for weeks. While Silver was spot-on, the pundit stars were overwhelmingly wrong.
Silver is now a media star with sales of his recent book, The Signal and the Noise: Why So Many Predictions Fail -- But Some Don't, up 850%. Does Silver's success mean that comments about aggregated analytics, gamma distributions and sum-of-squares formulas will now become de rigueur on the Washington cocktail circuit? That all those math teachers telling toiling high school students that statistics really will be useful in real life will finally be vindicated? That all the talk about "big data as the next big thing" will actually prove to be more than passing buzz?
Silver's success is less about big data (which is quickly being overused into meaninglessness) and is more about rigor, innovation and looking outside your business confines to find inspiration. One of the best explanations of Silver's methods to be in the Wikipedia entry about his role as a "practical statistician." "The practical statistician first needs a sound understanding of how baseball, poker, elections or other uncertain processes work, what measures are reliable and which not, what scales of aggregation are useful, and then to utilize the statistical tool kit as well as possible."
The goal of all the discussion around big data and data analysis is not to make the wrong decision faster, but to develop the best decision at the right time and deliver the information to the people that most need the information.
So what does any of this have to do with your business? The goal of all the discussion around big data and data analysis is not to make the wrong decision faster, but to develop the best decision at the right time and deliver the information to the people that most need the information.
Call it business intelligence, data analysis or predictive analytics, IT's role here is to provide a foundation for your company to make the right decisions. Those decisions might be what to charge passengers for seats on a flight, how much to charge to for a season ticket or how many widgets to create to strike the right balance among manufacturing costs, inventory and availability. These decisions are fundamental to business success.
There is no magic to Silver's methods. There is hard work, a willingness to make mistakes and adjust, and a realization that the common wisdom is sometimes not wisdom at all. Innovation can happen in strange places and Silver has shown that the buttoned-down world of statistics doesn't have to be that buttoned down at all.
Pay attention to Silver's process, but be equally assertive in looking at how your company operates. You will probably find lots of silos of activity where each group tends to use the same measures and methods year after year. Your job is to think outside the box, think like a customer and consider all the influences that would go into a purchasing decision. Understand the influences and you will be on your way to developing a prediction model that actually works for your business.
