Celebrity statistician Nate Silver, famous for his 2012 presidential election predictions, is also author of The Signal and the Noise -- a phrase that was repeated ad infinitum by panelists during this year's MIT Sloan Sports Analytics Conference to illustrate the challenges of sorting through and drawing conclusions from a maelstrom of data.
"One thing analytics can do is filter out the really stupid decisions," Silver said during an MIT panel on true performance and the science of randomness. He and his fellow panelists discussed what to do when data doesn't -- or can't -- be gathered to the point of being statistically significant, or when it doesn't back up the eye test.
In sports, as in the enterprise, there is sometimes simply no way to obtain statistically significant data in the area you're seeking to investigate. "There are no double-blind trials on operating on Yao Ming's foot," said Houston Rockets GM Daryl Morey, referring to his NBA team's former star, whose career came to an end after foot surgery and a failed comeback attempt.
And while statistically significant data is a boon, don't over-rely on the numbers.
"I think statistics make general managers better when they embrace them," said Steve Pagliuca, co-owner of the NBA's Boston Celtics. "It's a great tool, but it's not everything."
As a CIO, if your data analytics contradict the eye test, take a deep breath before proceeding with large-scale changes. You know your IT team, your business model, and the applicability of analytics to particular scenarios. Don't let the data dissuade you.