From data gathering to competitive strategy: The evolution of big data
A comprehensive collection of articles, videos and more, hand-picked by our editors
Setting out to write about data presentation, I remembered an expression first heard many years ago: Chance favors the prepared mind. I might have picked it up in my required science course but more likely read it in a novel. Googling, I learned the quote belongs to Louis Pasteur and comes prefaced with the words "dans les champs de l'observation," in the fields of observation.
But now, decades later, in a world awash with data, it's dawning on me that a prepared mind might be a liability of sorts -- the enemy of insight. I am thinking the prepared mind can become so prepared, so pickled, as to be impervious to fresh ideas. You can't teach an old dog new tricks. Worse, that perilous point might pass us by unnoticed, so set did we become in our ways.
We have big data on the brain these days at SearchCIO. In recent weeks, SearchCIO's Nicole Laskowski has written stories on using Bayesian probability in analyzing big data and Ancestry.com's use of Hadoop to expand its business model. Karen Goulart has pondered the wisdom of using analytics to plot employee value on an age curve and interviewed Sears Holding CTO Phil Shelley on tapping value from big data with open source tools.
If you want to see what the data hold in an unbiased way … there may be no choice but to literally see it.
As CIOs work at how to collect, store, analyze and present big data, they are left with the ever more difficult challenge of using data as a means of driving business decisions. After a year or so of sounding out experts and CIOs about how IT organizations can capitalize on big data, I'm beginning to think this effort will require CIOs to go where few have gone before: to the movies.
3D approach to data presentation "the way of the future"
The most recent jolt to my thinking came from Tom Davenport, whose forthcoming book on analytics drives home the point that the job of the CIO and the BI organization is not simply to present data, but to show it in a way that influences business decisions. In his survey of the best ways to present data, Davenport was most impressed by a group that turned the results from "a pretty straightforward financial reporting application" into a video. That's probably the way of the future, he said. "If you look at how we are persuaded to do other things, to buy products, to amuse ourselves -- think of all the clever cat videos people watch on YouTube these days -- it can't be every long until we get into the approach of communicating the results of analyses through video.
The case is particularly compelling for presenting big data. A simple graph can plot one thing against another -- that much I remember from algebra class. Add color and maybe it's possible to get three variables in one graph. But big data, as I've been told by chief scientists from PayPal to State Street, can contain hundreds of variables that are changing over time and space. Holy cat video! What kind of data presentation can make sense of this?
Recent CIO Matters columns by Linda Tucci
Two tangled tales of workplace privacy
Are big data and personal privacy mutually exclusive?
Well, if you already know what you're looking for, thanks to your prepared mind, then you just filter out all but the relevant few variables. But what if you don't know? What if you are hoping that the petabytes you've stored will reveal new relationships, as opposed to simply providing evidence to support or refute old ideas? If you want to see what the data hold in an unbiased way -- the holy grail, I'm told, of big data analytics -- there may be no choice but to literally see it. But how?
This, I think, is where the moving pictures come in and why that video of financial results made such an impression, even on an analytics veteran like Davenport. Movies render three dimensions of space; they let us see time pass. The application of color, texture and even sound depicts still more variables. The world we all understand is 3D, and moving pictures with sound pass for real experience.
Now CIOs have to make big data look like that world. Do all the rendering tools already exist for this? Of course not; Davenport's view is that they are still quite immature. But every computer generated animation we see shows that there is a path forward. Software tools of the kind James Cameron used in Avatar, like Autodesk Maya and Zbrush, could find their way into the CIO suite. New hires for data visualization could be the next big trend. Lights, camera, action! I, for one, am looking forward to big data in the movies. And so, I surmise, is the business. As another sage once said, you can observe a lot by watching.
Let us know what you think about the story; email Linda Tucci, Executive Editor.