SAN DIEGO – Best practices for big data analytics projects aren't the easiest things to come by. That's why Karen...
Liu, IT engineer at Cisco Systems, found her way to Gartner research director Svetlana Sicular's "Lessons Learned from Seven Big Data Failures" at this week's Gartner Catalyst conference.
"We're in the beginning stages, and we want to see what other people are doing -- what are the best practices," Liu said. "All of the failures, these are things we need to watch out for."
Liu wasn't alone. David Kropman, director of enterprise architecture at Family Dollar Stores Inc. in Charlotte, N.C., recently acquired by Dollar Tree Inc., attended for the same reason. "We're just getting started on the journey of big data," he said. "We're trying to find the right use case, and we're trying to make sure not to make the same mistakes other companies have made."
Sicular's session gave Kropman and Liu tangible information to take back to their teams so that they can plan for -- and hopefully avoid -- the missteps of others. There were lessons like this nugget: "Big data projects don't fail because of a single reason," Sicular said. "They fail because of a combination of reasons."
Sicular divided the seven lessons on big data failures into the following three categories: strategy, skills and analysis. They are as follows:
1. Organizational inertia. A successful travel logistics company dug into weblog data to shine a light on customer behavior. As it turned out, customers navigated the site and made purchases in a way that was contradictory to management assumptions. When team members explained this, management told them to do something else. But the team didn't give up. "They had an aggressive individual who figured out how to run A/B testing and, as a result, they managed to fire their management," Sicular said.
The end result isn't a realistic goal for every CIO, but the lesson holds: Get ready to work with management, and help them understand big data analysis and its value. One piece of advice for CIOs? Rather than focus efforts on a project that's failed, start fresh, Sicular said.
2. Selecting the wrong use cases. An insurance company wanted to investigate the relationship between good or bad habits and the propensity for buying life insurance. When the company realized "habits" was too general, it focused solely on smokers versus non-smokers, but even that didn't work. "In half a year, they closed this project, because they didn't find anything," Sicular said.
The failure, in this case, was due to the complexity of the problem. There's a big gray area the insurance company didn't account for: People who smoked and quit, a nuance likely overlooked because, to put it simply, "they're not healthcare professionals," Sicular said. She cautioned attendees to prioritize use cases and gradually increase the complexity of the problems they're trying to solve.
3. Inability to address unanticipated difficulties. A global company had a big data team that identified insights so profound, it wanted to make them available to the company as a whole. "As a result, they decided to implement the project in the cloud," Sicular said.
That, she said, can be problematic because a project performed in a controlled, proof-of-concept environment doesn't necessarily transfer to a production environment. "What they miscalculated was that they literally would be failing slow, because the network congestion didn't allow people in various locations across the globe to access this valuable analysis," she said.
The company needed to think about how to support big data and big data analytics, which requires a multitude of skills and cross-functional IT support to get off the ground. "I can't overestimate how many times a big data project failed because of the network, or because of security, or because of the facility," Sicular said. "You need to figure out who should participate on your team and who can tell you how to validate your results."
4. Lack of big data analytics skills. The CEO of a retail company didn't want to be "Amazoned," so he asked his CIO to build a custom recommendation engine. Executives promised the CEO he'd have the engine in six months, but the team soon realized concepts such as collaborative filtering were out of reach, prompting one team member to suggest building a "fake recommendation engine" using bed sheets as the sole recommended product, Sicular said.
The engine worked something like this: "People who bought blenders bought bed sheets; people who bought hiking books bought bed sheets; people who bought books also bought bed sheets," said Sicular, with bed sheets acting as a default recommendation for every purchase.
It wasn't a bad idea; the default recommendation gave the company a lift in sales, but the failure to build a real engine was the result of a lack of big data skills. This takes a long time to build up, Sicular said. The good news is big data adoption moves in stages, and knowing what those stages look like can help CIOs prepare. The stages are as follows:
- Top executives give their blessing.
- A big data strategy is planned.
- Experimentation, which includes trial and error, begins.
- A process is implemented and perfected. (Sicular referred to this as the tactical stage.)
- The ROI is realized; the company understands the value of big data. Sicular referred to this as the strategic stage.
- Companies ahead of technology begin building custom data products. Sicular referred to this as the transformative stage.
5. Not questioning the data. When Google began predicting flu trends in 2008, the Internet search giant started strong, predicting a flu epidemic two weeks ahead of the Centers for Disease Control and Prevention. But a couple of years later, Google overestimated doctor visits by 50%. "Media was talking so much about Google's success, people started looking for Google Flu Trend success instead of googling flu," Sicular said. "That skewed the data."
Once CIOs have the data, they need to insist it's inspected from various perspectives -- that "you understand where the data originated; you understand how you validated this data, whether you want dirty data or whether you want to have some control upfront," Sicular said.
6. Failure to ask the right questions. A car manufacturer with dealerships around the globe decided to pursue a sentiment analysis project, one that took six months and cost $10 million. When the results were in, the manufacturer contacted its dealers to share what they thought were new insights that would change how cars are sold. They turned out to be wrong.
"The team didn't take the time to go and ask the [dealers] what their problems are or what they might benefit from," Sicular said, making the analysis worthless.
She referred to this type of a situation as "satisficing," a decision management term that means settling for good enough. "Take your time," Sicular said. "Understand what question you're answering and what is the business benefit of this question."
7. Applying the wrong models. A Ph.D. at a bank turned to other industries to look at big data successes and see if the bank could co-opt ideas. He found one in the telecommunications industry: models built to predict and prevent customer churn.
The bank hired an expert from the telecommunications industry to help, and this expert soon spotted clear patterns of customers who looked to be on the verge of churning. "By the end of the proof of concept, they were ready for a pilot," Sicular said. Before getting started, the bank was supposed to print and send letters to customers asking them not to leave. But first, it asked its own business experts to take a look at the data to confirm the pattern.
The business experts discovered something surprising: Yes, people were planning to leave the bank, but not because they were unhappy with the bank's service, Sicular said. These were customers planning to divorce their spouses, so they were moving assets, sometimes quietly, in preparation. "It was about love, not about money," Sicular said.
Understanding the right models to use, the right level of data abstraction and the model's nuances "is very challenging," she said. "This is one of the keys of big data analytics." When it comes to big data projects, it's also important for CIOs to think ethics.
Read how two retailers are profiting from big data.
Test your degree of big data in the cloud expertise.