Big data applications: Real-world strategies for managing big data
A comprehensive collection of articles, videos and more, hand-picked by our editors
How can CIOs get tripped up by big data? Lack of leadership -- on the business and IT side -- is one of biggest inhibitors of big data success, according to Gartner Inc. The Stamford, Conn.-based consultancy has found that big data leadership failures often can be traced to two root causes: not understanding the business benefits that can be derived from collecting and mining large data sets, and underestimating the changes needed in...
IT and the business to realize those possibilities.
For most businesses, leveraging big data will require making cultural changes to become more data-driven, as well as making investments in new technology, such as Hadoop, and in new talents, such as data scientists. And doing all that, according to Gartner, rests on that age-old first priority for CIOs -- a close working relationship between IT and the business.
At the recent Gartner Business Intelligence Summit, analyst Doug Laney laid out the steps CIOs and their business peers need to take to develop a successful big data analytics strategy. The first of this two-part tip lists Gartner's big data rules for IT, from the need to embrace new platforms and recruit new skills to a renewed commitment to governance and, yes, IT/business alignment. Part two lays out the basic building blocks for the business: identifying value, inventorying data and adapting use cases from other industries.
CIO rules for developing an effective big data strategy
Embrace the cloud. The data warehouse, built to encapsulate that single version of the truth, will likely always remain the foundation for traditional business intelligence (BI) efforts. When it comes to big data analytics, however, businesses will also need to embrace new kinds of technologies.
"The monolithic data warehouse probably doesn't have a place in the big data world," Laney said. While still incredibly efficient when it comes to structured data, relational databases get tripped up on unstructured data such as video and text that don't fit neatly into rows and columns. Of the three V's of big data -- volume, velocity and variety -- variety is the biggest issue for organizations by a margin of 2:1, according to Laney.
When it comes to developing a big data strategy, IT departments have options, which include NoSQL databases. They also have the option of cloud computing. Laney believes that the industry is at the beginning of a 20-year span between no-cloud and all-cloud computing, a future CIOs should start grappling with right now.
Doug Laneyanalyst,�Gartner Inc.
"The day is coming when it's no longer going to make sense for organizations to be storing their own data under their own mattresses and burying it their own backyards," he said. "We're going to see the rise of the information service organization that's going to do that stuff for you in much the way the banking industry arose centuries or millennia ago to manage and handle our financial assets."
Invest in skills. There's a lot of talk about the relationship between data scientists and big data. While some still believe the term "data scientist" is a glorified title for a BI analyst or a statistician, Laney said that's not quite the case.
"We see more propensities for a data scientist to have a PhD, to be skilled in machine learning, to have better skills in data preparation, perhaps better skills in overall business modeling, communication [skills] and a passion for data," he said. "That came up in job postings -- a passion for data and a passion for the truth."
With more data and more potential relationships between data points, businesses will need experts to sift through and pinpoint the signal from the noise. But the strategy for big data skills investments shouldn't begin and end with data scientists. Laney believes IT departments need to continue building up a data-driven mindset. That includes investing in the back end of data by improving governance policies and data quality.
"Leadership isn't going to really get behind any kind of big data initiative if there's still trust and consistency and quality and accuracy and incompleteness issues in the data," he said. Leveraging internal data with external sources of information adds yet another challenge. CIOs must be ready to help form company policy on integrating external data with the organization's own data sources.
Pay attention to the role for data governance. Is a chief data officer (CDO) in your company's future? For Laney, the bigger question than hiring or not hiring a CDO is how businesses can build a strong data governance program for big data. Good data governance, which ensures data is cleaned, integrated, organized and labeled consistently, is vital to the success of big data programs. This is especially the case as businesses move to a more real-time environment, look to automate more processes or build architectures beyond the monolithic data warehouse, he said.
How big data is pushing the IT envelope
- Information is now seen as an asset rather than a resource.
- Formal data governance policies are a must-have with big data.
- The division between unstructured and structured data will eventually transition into hybrid-oriented data.
- Businesses are moving from hindsight to foresight analytics.
- Businesses are sharing, publishing and even monetizing data rather than locking it away.
Because good data governance is foundational to trustworthy analytics, Laney believes it should be included in the overall corporate governance program. "A strong data governance organization is going to not just set these policies, but also have a series of penalties and rewards or carrots and sticks associated with them." And while data governance tends to get typecast as the gatekeeper, the role is really much broader and more important than that: namely, "improving the realized value of information throughout the organization," he said.
Work hand in hand with the business. Bridging the divide between the IT department and the business has been a top priority for CIOs since forever, but as the businesses become more data-driven, closing the gap will become even more important.
"Most business people don't understand the art of the possible with big data," Laney said. "They don't understand what data sources are available or the range of the analytical capability that's available, so [IT] really needs to move much more to a consultative role."
On the surface, this means giving up the more traditional workflow of the business asking for something and the IT department delivering. For companies that would balk at this culture change, Laney recommends keeping the traditional BI approach for conventional BI projects and carving out a separate big data initiative that operates under different rules. The big data initiative would function more like an investigative team, with business and IT people brainstorming ideas together.
"The ideal data science 'tiger team,' in this regard, is going to follow a more scientific method," Laney said. "They'll come up with a hypothesis with the business, gather data, test it, build something in a sandbox and then go back to the business and say, 'Is this something you can use or will use if we institutionalize it?'"
That last question is the key to success, Laney stressed: IT departments must produce analytics that are useful to the business. Without it, big data remains a pile of stuff with no actionable insights.
Go to part two for Gartner's gloss on the basic building blocks for the business: identifying value, inventorying data and adapting use cases from other industries.