All too often, I see companies jumping headfirst into Teradata and other seductive big data analytics technologies like Hadoop, with no clear purpose or strategy.
I can somewhat excuse the overzealous department heads who want to dip their toe into the scorching-hot domain of big data analytics. For CIOs, however, such a leap is inexcusable because CIOs are much more accountable to their executive team and board, and have a greater responsibility to make smart decisions about any big data strategy. So, you should keep a few principles in mind before you leap into this trendy technology.
Have strong reasons for doing big data analytics
If you decide on an emerging technology to process your data, make sure you have very good reasons for doing so. Constantly challenge the assumptions that are driving that decision. There are plenty of reasons to build out a Hadoop implementation that have absolutely nothing to do with your corporate strategy -- it's a cutting-edge, exciting technology to experiment with, and the bragging rights will do wonders for your ego.
However, if you are using Hadoop for something that any relational database handles, you are making the wrong decision. The Hadoop platform is still an emerging technology. It should not be used for mission-critical applications. For instance, the Hadoop file system, or HDFS, is not ACID (atomicity, consistency, isolation, and durability) compliant: You can't guarantee transaction integrity -- and not having transaction integrity will foul things up severely when the system goes down unexpectedly.
Don't rely on advice from your directors and department heads. They will always opt for exciting, new technology. Instead, clearly understand your business metrics and processing requirements; insist on benchmarks from your current technology; and force your team to explore other options, such as hardware upgrades or more intelligent software.
I'm not advocating that you dispel big data technology altogether; in fact, I encourage you to understand and embrace its capabilities -- as long as it's used in the right context. Innovation should be a key enabler for one of your most critical contributions as CIO. Technologies like Hadoop can help you spearhead information products and services for your company, and distinguish you in a very competitive and unforgiving marketplace. Consequently, driving such efforts speaks volumes about your value with the rest of the executive suite.
Leave business metrics to the analysts
I've seen far too many IT professionals try to be data analysts in the race for big data. Because these efforts tend to be driven from the technology department, some organizations take the architects and developers who installed business intelligence (BI) technology and coerce them into analyzing the data so they don't lose control of the domain -- absolutely the wrong approach. Although they seem somewhat related, BI data analysis and technical development are two, completely separate disciplines.
If domain control is your issue, it makes perfect sense to staff your organization with a few data analysts. By doing so, you can build a super team that provides turnkey business metrics services for the other areas of your company. You could also cross-train your IT professionals with data analysis skills.
These suggestions come with a cost and won't happen overnight. If you need business metrics quickly, partnering properly with data analysts in other functions of the company is a completely viable option. You'll demonstrate a spirit of collaboration that everyone in the company will appreciate.
Architecture is everything
These days, knowing when and where to use big data technology is an art rather than a science. This can be very tricky: Trusting an architect's advice on exciting technologies like Hadoop is not very wise. Architects have a habit of inventing reasons to use emerging technology, hiding behind a cloak of specialized knowledge that nobody else seems to understand.
More on the big data debate
The architecture should be strongly guided by the reasons you adopt from the first principle. If your goal is to empower the product analytics team -- a goal that requires processing massive amounts of data -- build out your Hadoop platform to provide intelligent cubes that can plug into a front-end tool like SAP or Tableau.
Keep these principles in mind when building your strategy for big data analytics and you will avoid 90% of the pitfalls I've been seeing recently. Why not start today by launching a big data pilot for an innovative product or service? It just might be the next big thing for your company's business metrics, not to mention for your CIO career.
This was first published in February 2012