Essential Guide

From data gathering to competitive strategy: The evolution of big data

In this CIO Essential Guide, trace the evolution of big data and learn how to harness data-gathering activities as part of your competitive strategy.

Introduction

The potential of big data -- or "the new oil," as some CIOs and industry experts have dubbed it -- seems as endless as it is elusive. Big data campaigns are in their infancy, with enterprises of all stripes figuring out how to use new, old, unstructured and external data to create a competitive strategy.

Even though the ground rules for gathering data and scrutinizing its worth are still taking shape, companies know they need to get in the game. They are collecting and mining data on customers, employees, market dynamics, the weather, you name it, with tools ranging from traditional business intelligence (BI) systems to more experimental ones, such as geospatial and real-time mobile tracking technologies, social media analytics and NoSQL databases.

SearchCIO isn't standing on the sidelines, either. Our Essential Guide on big data includes a primer for getting started with data gathering and analysis, real-world case studies from the CIO and business perspectives, tips on how to overcome obstacles encountered by the big data pioneers, and predictions on the next big data frontier and what it means for competitive strategy.

This guide on the evolution of big data is part of SearchCIO's CIO Briefings series, which is designed to give IT leaders strategic management and decision-making advice on timely topics.

1Initial considerations-

Big data: The promise, and a primer

Before big data got its name, IT executives such as CIO Christopher Perretta of State Street Corp. and Warren Ritchie, former CIO at Volkswagen Group of America Inc., understood that CIOs had to stake a claim in this field. Back in 2011, they spoke about the potential of using large data sets to create business value and entirely new business models.

Whether your company is in a fight for its life or simply fighting to stay on top, its ability to manage and mine large data sets will be critical to its success. CIOs such as Ritchie and Perretta knew this. More to the point, business leaders knew this and took action in their organizations to promote data gathering and analysis as part of a larger competitive strategy.

News

Big data challenges CIOs while boosting careers

Learn how Ritchie, Perretta and their respective IT departments used information management to craft a completive strategy before the big data term was coined. Continue Reading

Tip

Five questions to answer before confronting big data

Uncovering the secrets lurking in large-scale data sets might be the next best hope for outdoing the competition, but first, IT needs to crack the code. Before you throw your hat into the big data ring, here are five questions to consider, including whether the tools you're using now for BI will cut it for big data. Continue Reading

News

Employee analysis skills make or break a big data strategy

It takes a village to mine data. That's analyst firm CEB's take on developing a big data plan of attack: Get a broad base of employees on board, particularly "informed skeptics." Here, read a sampling of questions that First Data Corp. uses to measure employees' "analytical maturity." Continue Reading

News

Gartner's IT essentials for a competitive strategy around big data

Leadership -- on both the IT and business sides -- is key for getting big data projects off the ground. In this two-part SearchCIO tip, Gartner analyst Doug Laney lays out the steps CIOs must take to make IT departments big-data enabled and the basic business rules for capitalizing on big data. Continue Reading

News

Experts predict disruptions around big data analysis

The promise of big data spawning new products and industries has eluded many companies. That's about to change, as industries ranging from health care to education to women's apparel use big data to invent whole new business models. Continue Reading

2Videos-

Watch big data evolve before your eyes

Are you on the front lines of harnessing large data sets? These videos offer advice for crafting a competitive strategy around big data.

Video

Next-generation BI and big data trends

IT once thought there was too much information and not enough data, but that mind-set has shifted: CIOs now have too much data and not enough information.

Video

Generating business value from data tools

Sears Holding Corp. chief technology officer Phil Shelley is working toward making Sears a real-time digital enterprise. That includes the migration of workloads off mainframes, implementing large-scale private open source cloud computing, and using Hadoop.

Video

Data-sharing initiatives drive innovation in health care

Cathy Bruno, vice president and CIO at Eastern Maine Healthcare Systems, is improving public health through innovations in data sharing, mobility and analytics.

3Making progress-

Big data in traction: The challenges

Statistician Nate Silver, crowned the big data expert by his political colleagues following the 2012 election, believes that while bigger, richer and more granular data sets offer deeper insights and greater business value, they also come with challenges that make uncovering those insights more difficult than before.

Silver unlocked the potential of large data sets when he accurately predicted nearly all of the results of the 2012 presidential campaign. Following on his success, IT leaders are spending more time focusing on competitive strategies for utilizing big data.

Tip

Big data challenges explained by Nate Silver

You've worked with your colleagues to identify a business opportunity using big data. You've done the impossible and assembled a team of data scientists. In other words, you've followed the best practices, yet your big data program is still going nowhere. You are not alone. Big data management is hard to do. Continue Reading

News

Data silos plague BI efforts

"Out damned silo, out!" has been IT's anguished plea since BI emerged on the scene decades ago. Yet data silos are still an albatross for CIOs, especially in the case of big data. But vanquishing data silos is a risk, as David Gallaher, IT service manager of the National Snow and Ice Data Center for NASA, puts it: "We want to let people look at the data, but we have to make damn sure they can't change it." Continue Reading

News

Data hoarding and bias slow big data progress

Many a hard-driver has pegged his or her career on keeping knowledge close to the vest. But for a successful big data strategy, sharing is king. Changing this hoarding mentality is one of many behavioral challenges that need to be addressed, along with the belief that the data has to be of perfect quality before it is analyzed. Continue Reading

News

Big data vet Martin Leach talks infrastructure

Analysts will tell you that reaping benefits from big data is all about changing the IT and business culture. That's true, but there are still plenty of nuts-and-bolts problems to solve, such as infrastructure, as a CIO veteran from the genome project explains. Where are the good plumbers when you need them? Continue Reading

News

Using data to predict the future poses problems for IT

Now that the volume, velocity and variety of data -- or big data -- are becoming the norm for many businesses, the value of a good predictive analytics practice has appreciated. This is easier said than done, as data scientist from the world's leading universities can attest. Continue Reading

News

Data wars on the horizon

Data -- gathering, processing and mining it -- could mean the difference between the haves and the have-nots in terms of productivity and business agility, not to mention profit. To avoid falling into the latter category, CIOs must get their own data house in order before entering the data-gathering battlefield. Continue Reading

News

Is personal data privacy at risk with big data?

In the rush to capitalize on big data, privacy issues are often overlooked. They won't be for long. Make sure, as you build a competitive strategy around big data, that you don't expose information unduly. Continue Reading

4Glossary-

Big data: Some terms to know

Brush up on some of the basics of a big data strategy with the terminology below.

5Real-world examples-

Big data in action: The case studies

For companies such as Land O'Lakes Inc. and its CIO, Barry Libenson, keeping up with an increase in population, and therefore increase in customers, is all about productivity. Using big data and predictive analytics, agricultural companies can utilize valuable information to keep up with changing business environments.

The new initiative not only involves integrating diverse data from disparate sources but also putting sophisticated and easy-to-consume analytics right into the hands of the company's internal sales and marketing team, as well as into the hands of the cooperatives and farmers it distributes to.

Feature

'Dirty data' gives Land O'Lakes an advantage

Talk about hitting pay dirt with big data! The IT team at Land O'Lakes used "dirty data," or data that could contain errors, to help its farm customers and leave competitors in the dust. By combining internal customer data with reams of external data, the business can better predict what crops will yield the highest value -- down to the acre. Continue Reading

News

Ancestory.com reinvents legacy systems with Hadoop

If Ancestry.com's data store (currently 4 petabytes) doesn't meet the threshold for big data, its newest venture will. Genealogy by autosomal DNA test will rely on data from a person's whole genome. Specifically, Ancestry.com will analyze the 700,000 different points along the genome. Meeting that need required reinventing Ancestry.com's traditional systems for processing and storing data. Continue Reading

News

Chief scientist's role in big data analytics

Mok Oh talks about his use of big data analytics during his tenure as chief data scientist at PayPal. Hint: It goes way beyond tracking transactional data. The ultimate goal is to fathom the human subconscious by looking at who buys what, when and why. Continue Reading

News

Miami Dolphins use mobile to get fans to the stadium

Are you ready for some football? How about a service that can map the best route to the stadium based on current traffic patterns, send mobile alerts on which gate to enter for the shortest wait time, and offer real-time discounts on food and drink based on consumption habits? With analytics technology, this Bud really is for you. It will all become a reality under Miami Dolphins CIO Tery Howard's big data plan. Continue Reading

News

Leveraging a semantic database at State Street

State Street Corp. Chief Scientist David Saul talks about the challenges and tradeoffs of getting a semantic database off the ground, and how semantic technology could produce really big benefits for many companies, thanks to his and others' efforts now under way to develop standards. Continue Reading

News

Data discovery in the hands of users

With more than 1 billion active users on its site every month, Facebook decided it was time to build a better data discovery mousetrap. That ultimately meant swapping out a homegrown reporting system and BI vendor tool set for data discovery tools that put the power of big data in the hands of business users. Continue Reading

6Quiz-

Big data storage and technologies: A quiz for CIOs

Changes in the way organizations collect, store and analyze information are transforming the way CIOs view big data. With massive amounts of data to account for, CIOs must equip their team with the tools and resources necessary for gathering data and finding value, and develop a plan for properly maintaining all that information.

Are you following the latest big data trends and staying hip to the infosphere? Where are you storing data? Are you analyzing information thoroughly? In this quiz, brush up on some of our recent stories and test your big data knowledge.

BIG DATA QUIZ FOR CIOS

7The next steps-

Big data on the horizon

According to our CIO expert Harvey Koeppel, we hear a lot these days about the "Big Data Frontier." By now, we all have a reasonable understanding of what we mean by "big" -- terabytes, petabytes, exabytes and so on. We certainly know what we mean by "data" -- fields, records, files, marts, warehouses, rows, columns, tables, blobs, structured data, unstructured data, etc. But what about the "frontier"?

Koeppel explains that big data will not only help us do a better job of anticipating our future -- it will be a significant enabler in creating our future.

Opinion

The future of big data

Every two days, we create as much information as we did from the dawn of civilization up until 2003. Where will big data lead us? As Koeppel lays out in this CIO Matters column: someplace big. He advises CIOs to look to the moon landing as an example of the many downstream breakthroughs that could come from harnessing the power of data. Continue Reading

Opinion

CIOs rely on data presentations to make informed decisions

The work CIOs do to gather, store and analyze big data will be for naught without data presentations that influence business decisions. CIOs might want to page special-effects phenom James Cameron. After all, the world we all understand is 3-D, and moving pictures with sound pass for real experience. Continue Reading

News

Data mining challenges to come

One way to allow data to reveal itself is to tap into the strongest human analytical sense -- our vision. But it won't be easy to build front-end visualization interfaces that present the data in ways we humans can easily apprehend and interact with. Continue Reading

News

Why collect big data?

In his quest for a state he calls Analytics 3.0, former PayPal Chief Scientist Mok Oh ponders the enormity of big data and what it will take to wring value from all this information. "Natural languages, analog-to-digital conversions and making sure there is very little lost in translation -- maybe that is Analytics 3.0!" Eureka? Continue Reading

News

CIOs turn to metadata practices

Metadata has been seen as secondary to analytics and data management programs at many companies, but with the advent of 'big data,' that's changing. Continue Reading

Opinion

Information is the next big company asset

CIOs by definition understand the value of information technology. Now, the rest of the world is on the case, marrying the field of information and economics. Welcome to infonomics. Continue Reading