Emerging: DataOps and three tips for getting there

A look at DataOps, Agile analytics and an IT leader who is striving to make Boston a data-driven city: The Data Mill reports.

Chris Bergh, founder and "head chef" at DataKitchen Inc., a consulting company in Cambridge, Mass., didn't give his Agile analytics presentation at the Boston Data Festival with CIOs in mind, but IT leaders might want to pay attention anyway.

Bergh's list of best practices provided a look at what analysts want -- or need -- to support an Agile analytics program, and much of it relies on culture and technology. To wit, Bergh called attention to an emerging concept sometimes referred to as DataOps.

Like DevOps, the impetus for DataOps is to blend teams together to increase collaboration and agility. While DevOps combines the development and operations teams, DataOps is "the set of best practices that improve coordination between data science and operations," according to consultant Lenny Liebmann, founding partner at Morgan Armstrong in Teaneck, N.J., and contributing editor to technology publications.

Andy Palmer, CEO and co-founder at Tamr Inc., based in Cambridge, Mass., provided more detail in a recent blog post: "DataOps is a data management method that emphasizes communication, collaboration, integration, automation and measurement of cooperation between data engineers, data scientists and other data professionals."

As with any new approach, the pioneers haven't sorted out the language just yet: While Palmer refers to it as a "data management method," Bergh calls it an "analytic development method" that should be overseen by a chief data officer or a chief analytics officer. (The Bergh team refers to DataOps as AnalyticOps.) In either case, the ultimate goal is to accelerate analytics. And, regardless of how businesses decide to practice DataOps, successful programs will require IT expertise in the form of data integration, data quality, data security and data governance, according to Palmer and Bergh.

Agile analytics: Three tips

Here's a look at three of the seven tips Bergh provided to attendees on how to build an Agile analytics program:

  1. Set up a testing environment. Bergh recommended analysts test new features or processes they're tinkering with before unveiling them to the company or the customer. But, to do so, the analysts will need a separate environment or laboratory where they can experiment without disrupting day-to-day operations. Bergh takes this piece of advice right from the engineering world. 
  2. Give your analysts "knobs." Analysts often want to have their own databases, but they aren't "full-stack database engineers," nor do they have to be, Bergh said. Providing tools such as a user interface, scripts and so on can help analysts over the hurdle, he said.
  3. Use simple storage. IT's relational data warehouse just isn't cutting it these days, and Bergh suggested businesses build a data lake, where raw data can be stored and easily accessed. "If your analysts are going to have their own environment, sometimes they will want to pull the lowest level of data into their own environment" to research on and experiment with, Bergh said. Tools such as the Hadoop Distributed File System and Amazon Simple Storage Service (S3) can help with this. For those who decide to go the S3 route, Bergh suggested taking a look at Amazon Glacier, a cheap data archiving option. 

The data-driven city

Ushering in change is one of the toughest tasks facing any CIO. But in the government sector, where tape recorders and typewriters are still common tools of the trade, it's arguably the toughest.

When Kelly Jin, who leads data visualization efforts for the city of Boston, started her position in January, she encountered her share of skeptics. And she addresses that skepticism by bringing it back to a basic "business" problem. "Whenever I get asked what our team does, I say we are trying to improve the quality of life for Boston citizens," Jin said during her presentation at the Boston Data Fest.

To quantify her statement, she'll tick off data points like these: Boston proper has 650,000 residents, 48 square miles of land, 850 miles of streets and 128 schools. "We produce a ton of data," she said. Still, building a data-driven culture isn't like flipping a switch. It takes time, leadership and, as cheesy as it sounds, passion.

CIOs know the typical wave of adoption -- technology or otherwise -- starts with Early adopters. But even before the early adopters, CIOs will need to find their innovators -- employees who are, essentially, change agents. "In order to build a culture, we needed to identify not only the people who have technical skills or the business skills, but those who also are fearless. They want to go out to an organization and actually change things -- they want to change the way government works," she said.

For the city of Boston, a 25-member analytics team, which includes Jin, are those change agents. Together, they're set on making city government more efficient and user-friendly for city workers and constituents alike. Last winter, when more than 100 inches of snow fell on the city, Jin's team rolled out a new webpage that gave residents a near real-time window into what streets were plowed when. Similar efforts have been applied to public safety initiatives, to the permitting process and even to how data is delivered to the mayor.

Before Jin arrived, a basic dashboard was designed for Mayor Martin Walsh, the first of its kind for the city. A year later, the mayor's dashboard has not only become more a sophisticated administration window into Boston doings, it also acts as a constituent-facing information portal.

Welcome to The Data Mill, a weekly column devoted to all things data. Heard something newsy (or gossipy)? Email me or find me on Twitter at @TT_Nicole.

Next Steps

Read up on why CIOs should be thinking DevSecOps instead of DevOps.

Dig Deeper on Enterprise business intelligence software and big data