The potential of big data -- or "the new oil," as some CIOs and industry experts have dubbed it -- seems as endless as it is elusive. Big data campaigns are in their infancy, with enterprises of all stripes figuring out how to use new, old, unstructured and external data to create a competitive strategy.
Even though the ground rules for gathering data and scrutinizing its worth are still taking shape, companies know they need to get in the game. They are collecting and mining data on customers, employees, market dynamics, the weather, you name it, with tools ranging from traditional business intelligence (BI) systems to more experimental ones, such as geospatial and real-time mobile tracking technologies, social media analytics and NoSQL databases.
SearchCIO isn't standing on the sidelines, either. Our Essential Guide on big data includes a primer for getting started with data gathering and analysis, real-world case studies from the CIO and business perspectives, tips on how to overcome obstacles encountered by the big data pioneers, and predictions on the next big data frontier and what it means for competitive strategy.
This guide on the evolution of big data is part of SearchCIO's CIO Briefings series, which is designed to give IT leaders strategic management and decision-making advice on timely topics.
Big data: The promise, and a primer
Before big data got its name, IT executives such as CIO Christopher Perretta of State Street Corp. and Warren Ritchie, former CIO at Volkswagen Group of America Inc., understood that CIOs had to stake a claim in this field. Back in 2011, they spoke about the potential of using large data sets to create business value and entirely new business models.
Whether your company is in a fight for its life or simply fighting to stay on top, its ability to manage and mine large data sets will be critical to its success. CIOs such as Ritchie and Perretta knew this. More to the point, business leaders knew this and took action in their organizations to promote data gathering and analysis as part of a larger competitive strategy.
Uncovering the secrets lurking in large-scale data sets might be the next best hope for outdoing the competition, but first, IT needs to crack the code. Before you throw your hat into the big data ring, here are five questions to consider, including whether the tools you're using now for BI will cut it for big data. Continue Reading
It takes a village to mine data. That's analyst firm CEB's take on developing a big data plan of attack: Get a broad base of employees on board, particularly "informed skeptics." Here, read a sampling of questions that First Data Corp. uses to measure employees' "analytical maturity." Continue Reading
Leadership -- on both the IT and business sides -- is key for getting big data projects off the ground. In this two-part SearchCIO tip, Gartner analyst Doug Laney lays out the steps CIOs must take to make IT departments big-data enabled and the basic business rules for capitalizing on big data. Continue Reading
The promise of big data spawning new products and industries has eluded many companies. That's about to change, as industries ranging from health care to education to women's apparel use big data to invent whole new business models. Continue Reading
Watch big data evolve before your eyes
Are you on the front lines of harnessing large data sets? These videos offer advice for crafting a competitive strategy around big data.
IT once thought there was too much information and not enough data, but that mind-set has shifted: CIOs now have too much data and not enough information.
Sears Holding Corp. chief technology officer Phil Shelley is working toward making Sears a real-time digital enterprise. That includes the migration of workloads off mainframes, implementing large-scale private open source cloud computing, and using Hadoop.
Cathy Bruno, vice president and CIO at Eastern Maine Healthcare Systems, is improving public health through innovations in data sharing, mobility and analytics.
Big data in traction: The challenges
Statistician Nate Silver, crowned the big data expert by his political colleagues following the 2012 election, believes that while bigger, richer and more granular data sets offer deeper insights and greater business value, they also come with challenges that make uncovering those insights more difficult than before.
Silver unlocked the potential of large data sets when he accurately predicted nearly all of the results of the 2012 presidential campaign. Following on his success, IT leaders are spending more time focusing on competitive strategies for utilizing big data.
You've worked with your colleagues to identify a business opportunity using big data. You've done the impossible and assembled a team of data scientists. In other words, you've followed the best practices, yet your big data program is still going nowhere. You are not alone. Big data management is hard to do. Continue Reading
"Out damned silo, out!" has been IT's anguished plea since BI emerged on the scene decades ago. Yet data silos are still an albatross for CIOs, especially in the case of big data. But vanquishing data silos is a risk, as David Gallaher, IT service manager of the National Snow and Ice Data Center for NASA, puts it: "We want to let people look at the data, but we have to make damn sure they can't change it." Continue Reading
Many a hard-driver has pegged his or her career on keeping knowledge close to the vest. But for a successful big data strategy, sharing is king. Changing this hoarding mentality is one of many behavioral challenges that need to be addressed, along with the belief that the data has to be of perfect quality before it is analyzed. Continue Reading
Analysts will tell you that reaping benefits from big data is all about changing the IT and business culture. That's true, but there are still plenty of nuts-and-bolts problems to solve, such as infrastructure, as a CIO veteran from the genome project explains. Where are the good plumbers when you need them? Continue Reading
Now that the volume, velocity and variety of data -- or big data -- are becoming the norm for many businesses, the value of a good predictive analytics practice has appreciated. This is easier said than done, as data scientist from the world's leading universities can attest. Continue Reading
Data -- gathering, processing and mining it -- could mean the difference between the haves and the have-nots in terms of productivity and business agility, not to mention profit. To avoid falling into the latter category, CIOs must get their own data house in order before entering the data-gathering battlefield. Continue Reading
Big data: Some terms to know
Brush up on some of the basics of a big data strategy with the terminology below.
Big data in action: The case studies
For companies such as Land O'Lakes Inc. and its CIO, Barry Libenson, keeping up with an increase in population, and therefore increase in customers, is all about productivity. Using big data and predictive analytics, agricultural companies can utilize valuable information to keep up with changing business environments.
The new initiative not only involves integrating diverse data from disparate sources but also putting sophisticated and easy-to-consume analytics right into the hands of the company's internal sales and marketing team, as well as into the hands of the cooperatives and farmers it distributes to.
Talk about hitting pay dirt with big data! The IT team at Land O'Lakes used "dirty data," or data that could contain errors, to help its farm customers and leave competitors in the dust. By combining internal customer data with reams of external data, the business can better predict what crops will yield the highest value -- down to the acre. Continue Reading
If Ancestry.com's data store (currently 4 petabytes) doesn't meet the threshold for big data, its newest venture will. Genealogy by autosomal DNA test will rely on data from a person's whole genome. Specifically, Ancestry.com will analyze the 700,000 different points along the genome. Meeting that need required reinventing Ancestry.com's traditional systems for processing and storing data. Continue Reading
Mok Oh talks about his use of big data analytics during his tenure as chief data scientist at PayPal. Hint: It goes way beyond tracking transactional data. The ultimate goal is to fathom the human subconscious by looking at who buys what, when and why. Continue Reading
Are you ready for some football? How about a service that can map the best route to the stadium based on current traffic patterns, send mobile alerts on which gate to enter for the shortest wait time, and offer real-time discounts on food and drink based on consumption habits? With analytics technology, this Bud really is for you. It will all become a reality under Miami Dolphins CIO Tery Howard's big data plan. Continue Reading
State Street Corp. Chief Scientist David Saul talks about the challenges and tradeoffs of getting a semantic database off the ground, and how semantic technology could produce really big benefits for many companies, thanks to his and others' efforts now under way to develop standards. Continue Reading
With more than 1 billion active users on its site every month, Facebook decided it was time to build a better data discovery mousetrap. That ultimately meant swapping out a homegrown reporting system and BI vendor tool set for data discovery tools that put the power of big data in the hands of business users. Continue Reading
Big data storage and technologies: A quiz for CIOs
Changes in the way organizations collect, store and analyze information are transforming the way CIOs view big data. With massive amounts of data to account for, CIOs must equip their team with the tools and resources necessary for gathering data and finding value, and develop a plan for properly maintaining all that information.
Are you following the latest big data trends and staying hip to the infosphere? Where are you storing data? Are you analyzing information thoroughly? In this quiz, brush up on some of our recent stories and test your big data knowledge.BIG DATA QUIZ FOR CIOS
7The next steps-
Big data on the horizon
According to our CIO expert Harvey Koeppel, we hear a lot these days about the "Big Data Frontier." By now, we all have a reasonable understanding of what we mean by "big" -- terabytes, petabytes, exabytes and so on. We certainly know what we mean by "data" -- fields, records, files, marts, warehouses, rows, columns, tables, blobs, structured data, unstructured data, etc. But what about the "frontier"?
Koeppel explains that big data will not only help us do a better job of anticipating our future -- it will be a significant enabler in creating our future.
Every two days, we create as much information as we did from the dawn of civilization up until 2003. Where will big data lead us? As Koeppel lays out in this CIO Matters column: someplace big. He advises CIOs to look to the moon landing as an example of the many downstream breakthroughs that could come from harnessing the power of data. Continue Reading
The work CIOs do to gather, store and analyze big data will be for naught without data presentations that influence business decisions. CIOs might want to page special-effects phenom James Cameron. After all, the world we all understand is 3-D, and moving pictures with sound pass for real experience. Continue Reading
In his quest for a state he calls Analytics 3.0, former PayPal Chief Scientist Mok Oh ponders the enormity of big data and what it will take to wring value from all this information. "Natural languages, analog-to-digital conversions and making sure there is very little lost in translation -- maybe that is Analytics 3.0!" Eureka? Continue Reading