From data gathering to competitive strategy: The evolution of big data
A comprehensive collection of articles, videos and more, hand-picked by our editors
We hear a lot these days about the "Big Data Frontier." The relentless proliferation of columns (like this one), articles, blogs, videos and tweets provides more input on the subject than any one of us is capable of reading, watching, tracking and absorbing. Finding and exploiting value from the plethora of such content is in itself a big data challenge, and one that is seemingly self-sustaining (not necessarily the "green" kind).
By now, we all have a reasonable understanding of what we mean by "big" -- terabytes, petabytes, exabytes and so on. We certainly know what we mean by "data" -- fields, records, files, marts, warehouses, rows, columns, tables, blobs, structured data, unstructured data, etc. Frankly, we have been told ad nauseam how many times all of the information contained in the Library of Congress can fit on the head of a pin.
But what about "frontier"? Most of us gloss over that part of the term, casually accepting it as the softer marketing part of the story -- the word that authors, editors or publishers insert to grab your attention and motivate you to read their story about big data instead of the hundreds, maybe thousands, of other stories about big data that ubiquitously appear across the channels that day.
That is not, by the way, why I included the F-word in the title of this column; I included it in the title because I believe that it embodies the essence of the meaning and value of what big data is really all about. While we have meticulously honed the art and science of leveraging "little data" to increase the efficiency of processes and operations and to build better enterprises that are more well-informed about the past and present to more accurately predict the future, we are just at the beginning of understanding how to collect, store, manage, retrieve and analyze big data.
Big data will not only help us do a better job of anticipating our future, it will be a significant enabler of creating our future. That is the meaning of "frontier" in the Big Data Frontier.
The Big Data Frontier: A historical perspective
While accepting his party's nomination as their candidate for the office of the presidency of the United States on July 15, 1960, John F. Kennedy shared the following vision with those in attendance that day in the Memorial Coliseum in Los Angeles:
Just like JFK, Mashey knew that pioneers are imaginative and innovative and inventive out of necessity, and that new inventions generate incredible value when frontiers are reached, limits are expanded and new frontiers are created.
For I stand here tonight facing west on what was once the last frontier. From the lands that stretch three thousand miles behind us, the pioneers gave up their safety, their comfort and sometimes their lives to build our new West. … Some would say that those struggles are all over, that all the horizons have been explored, that all the battles have been won, that there is no longer an American frontier. But I trust that no one in this assemblage would agree with that sentiment; for the problems are not all solved and the battles are not all won; and we stand today on the edge of a New Frontier -- the frontier of the 1960's, the frontier of unknown opportunities and perils, the frontier of unfilled hopes and unfilled threats. … But I believe the times demand new invention, innovation, imagination, decision. I am asking each of you to be pioneers on that New Frontier.
About a year later, on May 25, 1961, by then President Kennedy posed an audacious challenge to a joint session of Congress and to our nation: "I believe that this nation should commit itself to achieving the goal, before this decade is out, of landing a man on the moon and returning him safely to the Earth."
Eight years later, on July 20, 1969, NASA's Apollo 11 mission would land the first humans on the moon, returning them home safely four days later.
At the time that JFK acknowledged the New Frontier and subsequently posed the challenge of landing men on the moon in that decade, he could not possibly have known what stepping up to and eventually meeting the inherent challenges would mean. He knew that his country's future as a world political leader would be built upon imagination and innovation, and that his country's future as an economic and military leader would be built upon invention and productivity.
NASA's participation in the space program has been directly credited with the invention and commercialization of hundreds of products including the following:
- Anti-icing systems for aircraft
- Artificial limbs
- Cochlear implants
- Enriched baby food
- Firefighting equipment
- Freeze-dried food
- LEDs used in medical treatments
- Memory foam
- Solar energy
NASA additionally lists more than 1,650 spin-offs in the fields of computer technology, environment and agriculture, health and medicine, public safety, transportation, recreation, and industrial productivity. Contrary to common belief and urban legend, NASA did not invent Tang, Velcro, or Teflon (source: "NASA spin-off technologies," Wikipedia).
JFK clearly did not set out to invent DustBusters or freeze-dried foods. When we look at the breadth, depth and ultimate value of these technologies, either individually or collectively, it is likely that no one at the time envisioned what moving forward into and beyond the New Frontier of the 1960s would bring to our society, our nation or our planet. That is the meaning of "frontier" in the Big Data Frontier.
A contemporary perspective on the big data frontier
On April 25, 1998, John R. Mashey, then chief scientist at SGI, described the Big Data Frontier as "the next wave of InfraStress."
Source: John R. Mashey, "Big data ... and the next wave of InfraStress."
More on big data strategies
Nate Silver on the power of big data done right
Big data and personal data privacy
Gartner on big data strategy building essentials
What Mashey told us was that our ability and propensity to generate continuously increasing amounts of data far exceeded the limits of our ability to store, process, analyze and transmit it, and that this situation would cause stress on all of the underlying infrastructure used to generate and process the data. Mashey postulated that dealing with such InfraStress (stepping up to and meeting the challenges of the Big Data Frontier) would, of necessity, spawn significant advances in storage technologies, communications technologies, operating systems, analytics capabilities and others.
Just as JFK could not possibly have predicted that the space program would give rise to thousands of commercial businesses inevitably worth billions of dollars that would be seeded from the more than 1,650 technology innovations that came out of NASA, Mashey could not have predicted the details describing the millions of businesses that would generate trillions in revenues as small local storefronts grew to become global competitors based upon Internet, cloud, mobile technologies, in-memory analytics or geofencing. What he understood was that these technologies would be created as the natural outgrowth of pioneers pushing boundaries. Just like JFK, Mashey knew that pioneers are imaginative, innovative and inventive out of necessity, and that new inventions generate incredible value when frontiers are reached, limits are expanded and new frontiers are created. This is a classic example of the whole being worth considerably more than the sum of its parts.
Potential future value of big data
In a recent research study, McKinsey Global Institute explored the implications of big data. Researchers looked at the phenomenon from social, economic and political points of view. A main objective of the study was to quantify and articulate the magnitude of important issues being created by the Big Data Frontier that governments and organizations need to understand and address. Here are a few key findings from the research:
Potential annual value of big data to U.S. health care
Potential annual value of big data to Europe's public sector administration
Potential annual consumer surplus from using personal location data globally
Potential increase in retailers' operating margins attributable to leveraging big data
Additional number of people needed with deep analytical skills to support big data projects
|1.5 million||Number of data-savvy managers needed to take full advantage of big data in the U.S.|
Source: McKinsey Global Institute, "Big data: The next frontier for innovation, competition and productivity," June 2011
Let me know what you think. Post a comment or drop me a note at firstname.lastname@example.org. Discuss, debate or even argue -- let's continue the conversation.