From data gathering to competitive strategy: The evolution of big data
A comprehensive collection of articles, videos and more, hand-picked by our editors
In the data wars that are sure to come, most companies will be flying blind and will depend on others for the information they need. In an illuminating report out this week from the nonprofit industry association CompTIA, 63% of IT and business executives said they don't have a firm grasp of the concept of "big data." But they're worried about it. If they don't take steps now to seize big data, they said, their organizations could lose...
out -- on productivity and on business agility. The business could get bogged down by "internal confusion over priorities" and suffer "reduced margins due to operational inefficiencies."
The 44-page "Big Data Insights and Opportunities" report, based on two online surveys of some 500 IT and business executives in July, shows executives on both sides of the IT-business aisle are aware of what data -- big, little or medium -- means to getting business just right. Two-thirds of respondents agreed with the statement, "If we could harness all of our data, we would be a much a stronger business." Corralling that currency will only get harder. The market research firm IDC pegs the amount of world data in 2012 at a cool 2.7 zettabytes and expects that volume to double approximately every two years.
"As expected for an emerging technology with an evolving definition, many executives are still moving along the big data learning curve," said CompTIA's Tim Herbert, a research vice president, in a note accompanying the report.
Mok Ohchief scientist, PayPal
Of course. But it's not just the learning curve for the big and even bigger data to come that's steep. Most companies are still fighting data wars on their own turf, according to CompTIA research. Nearly three in four companies surveyed reported a moderate to high degree of data silos in their organizations, so they lack a single view of their data. Respondents said they fall short on analyzing Web patterns (80%), measuring email campaigns (85%) and monitoring social media (88%). Unstructured data -- the audio and video files and social streams of data that don't easily fit into the traditional databases that companies know how to manipulate -- is passing them by.
Efforts are afoot to gain ground. Over the next two years, 41% of large companies plan on hiring new employees to meet the data analysis and business intelligence requirement to leverage big data, while 39% plan to use outside consultants or vendors. That's assuming they have the data to leverage. Just one in three companies reported having a comprehensive business continuity and disaster recovery plan, meaning many companies will lose the data wars just by losing data.
In data wars, fortune favors the few
Only 6% of those surveyed said they "are exactly where they want to be" with managing and using data. "Exactly" is a tall order. (And, indeed, nearly a third of the survey's respondents said they were "very close," and another 47% said they were "moderately close" to where they need to be with managing and using their data.) But my hunch is that those 6% -- or whatever the small percentage is of big-data-savvy companies out there -- will wield tremendous influence over their industries and over us, their customers. The report points to a few examples: Wal-Mart's and Target's well-publicized ability to sift through mountains of social data points to make shopping predictions. New Jersey and other states are harnessing traffic, weather, driver and mobile phone data and other streams to get real-time snapshots of their transportation networks to predict delays and respond faster to accidents.
More CIO Matters columns by Linda Tucci
The gap between companies who harness big data and those that don't is going to widen. I'm sure of it. I recently interviewed the chief scientist at PayPal, Mok Oh, who's in charge of "anything and everything science-y" at the global payment company. "Currently what I am really focusing on is big data and data science," he said. Big data, the way most people use it, is a misnomer, he said. CIOs at Fortune 500 companies are going to spend tens of millions of dollars on infrastructure -- or as he put it, "these big-ass tools to connect to big data." But that's just "setting up the pipes." At PayPal, the pipes are being laid and the data is being mined to connect PayPal merchants and customers, not just on a generic level, but on a taco-truck-by-taco-truck level -- on every use case imaginable in the PayPal universe of 100 million active users and millions and millions of merchants. Oh yeah, and what Oh is oh-so-excited about is not pipes or even what the customer is buying now, but what we will buy next. PayPal wants to know more about our shopping habits than we do. Oh's big data quest is nothing less than the subconscious.
But here's the unfortunate part. When I asked Oh whether companies that don't have millions of customers or that are more specialized (or both) will be able to get value out of these kinds of big data analytics, his answer was, "It depends." "As time goes by, more and more people are realizing that data is an asset and something they should own and capitalize on. With that said, it will get tougher and tougher to get data," he said.
Some companies won't have the resources to tap into the big data pools that might yield business insights, so they will need to purchase data. Big companies will own and sell their big data -- whatever part of it they want to sell -- to the little companies. For evidence of how precious a commodity data has become, look no further than the new iPhone 5. Google Maps isn't on it because, well, why would Apple want to give that information away for free to a competitor?
Let us know what you think about the story; email Linda Tucci, News Director.