Essential Guide

Browse Sections


This content is part of the Essential Guide: Big data tutorial: Everything you need to know
News Stay informed about the latest enterprise technology news and product updates.

Data hoarding and bias among big challenges in big data and analytics

Companies will rise and fall on their ability to use big data and analytics, research suggests, but human foibles make success anything but certain.

Hype aside, exploiting big data and analytics will matter hugely to companies' future performance, remaking whole industries and spawning new ones. The list of challenges is long, however. They range from the well-documented paucity of data scientists available to crunch that big data, to more intractable but less-mentioned problems rooted in human nature. 

One of the latter is humans' tendency to hoard data. Another is their tendency to hold on to preconceived beliefs even when the data screams otherwise. That was the consensus of a panel of data experts speaking on big data and analytics at the recent MIT Sloan CIO Symposium in Cambridge, Mass. Another landmine? False hope. There is no final truth in big data and analytics, as the enterprises that do big data well already know. Iteration is all, the panel agreed.

Moreover, except for the value of iteration, CIOs can forget about best practices. Emerging so-called next practices are about the best companies can lean on as they dive into big data, said computer scientist Michael Chui, San Francisco-based senior fellow at the McKinsey Global Institute, the research arm of New York-based McKinsey & Co. Inc.

"The one thing we know that doesn't work: Wait five years until the perfect data warehouse is ready," said Chui, who's an author of last year's massive McKinsey report on the value of big data.

Seeing data quality in relative terms

In fact, obsessing over data quality is one of the first hurdles many companies have to overcome if they hope to use big data effectively, Chui said. Data accuracy is of paramount importance in banks' financial statements. Messy data, however, contains patterns that can highlight business problems or provide insights that generate significant value, as laid out in a related story about the symposium panel, "Seize big data and analytics or fall behind, MIT panel says."

Organizations that are "up-front about the quality of data" -- even using meta tags or color coding to indicate its quality -- get to work on big data faster than their counterparts can that are "spinning their wheels" over getting the best-quality data, said panelist Shvetank Shah, executive director of The Corporate Executive Board Co. (CEB), a Washington, D.C-based consulting firm.

Big data's messiness, however, also makes business acumen all the more important -- putting a premium on a manager's ability to know when the data is worth pursuing, Shah cautioned. "This is why you pay a manager: to analyze, to make some calls and iterate," he said.

In scientific research, it's impossible to understand everything about every variant, so "iteration is important," said James Noga, CIO at Partners HealthCare System, a Boston-based health care nonprofit. The type of people who excel at working with big data must be able to cull out salient points and to "make a best guess at the time," he said.

That can be a stretch, however, not only for companies that over-specify data quality, but also for IT shops accustomed to structured IT processes.

Hard to 'unstick' old ways, deep beliefs

People with pattern-recognition skills, curiosity and an understanding of the value of experimentation are the key to using big data and analytics effectively, the panelists said. Making the scientific method part of company culture, however, is extremely difficult, CEB's Shah has found. "You can have all the smart quants [quantitative analysts] sitting in the center of the enterprise making lots of smart decisions, but equally if not more important are decisions made by customer service reps, by managers and so on at the periphery," he said. Because most companies cannot hire enough data scientists to exploit big data, finding people with coaching skills is another challenge.

The CEB has found that few companies use big data and analytics to drive business decisions. Its recent study of roughly 500 companies showed that about 20% of respondents go by gut instinct when they make decisions; 49% want more data; and 38% are what CEB calls "informed skeptics," employees who can blend judgment with data and drive the business forward. Moreover, people tend to hold on to accepted beliefs, "even when the data is telling you something completely contradictory," Shah said. "It is very hard to unstick biases."

Hoarding data is another barrier to exploiting big data. One of the surprises of McKinsey's study, Chui said, was that financial services -- with its long history of collecting and analyzing data -- lagged in using big data. "What we found in a lot of Western banks is that the lines of business silos had become so strong that the idea of sharing data is just very, very weak," he said.

Let us know what you think about the story; email Linda Tucci, Senior News Writer.

Dig Deeper on Enterprise data storage management

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.