News Stay informed about the latest enterprise technology news and product updates.

Companies want explainable AI, vendors respond

Dispatch from the Strata Data Conference: The push for explainable AI is on, and companies like and Microsoft are looking to deliver.

Fed up with the bribery, insider trading, embezzlement and money laundering committed by white-collar criminals?...

What if there was an app that could help nab these crooks by using the same machine learning tools and geospatial data increasingly relied upon by police to predict where the next burglary, drug deal or assault might go down?

Sam Lavigne, co-creator of the White Collar Crime Risk Zones app, was onstage at the recent Strata Data Conference in New York, claiming to be able to do just that.

"We used instances of financial malfeasance; density of nonprofit organizations, liquor stores, bars and clubs; and density of investment advisers," a straight-faced Lavigne said to an audience of data experts who immediately got the dark humor.

For although the White Collar Crime Risk Zones app was indeed built -- using historical data from the Financial Industry Regulatory Authority -- its purpose is not to track white-collar crime, but to draw attention to the danger these kinds of applications, and the data they rely upon, present.   

Machine learning models are trained on data collected by humans, and so the models can produce results that are biased and unfair.

"This is particularly dangerous in the case of predictive policing because that data comes from police departments that, at times, could be accused of being systemically racist," Lavigne told the audience. "And the risk here, of course, is that it produces a kind of feedback loop for over-policing communities of color."

AI is only human

Machine models also rely on a definition for success and a penalty for failure, according to data scientist and author Cathy O'Neil, who gave a talk at the event on the shortcomings of mathematical models.

These, too, are written by humans -- often data scientists -- and the problem here is that, "not everyone agrees on what success looks like outside the confines of a game of chess or a baseball game," O'Neil said.

Plus, machine learning algorithms operate in a so-called black box -- so the inputs and outputs are known, but how or why an algorithm makes the recommendation it does is not clear.

Yet, despite all of this, machine learning recommendations are often presented and seen as objective truth, O'Neil said. And blind faith in their accuracy is already damaging  lives, as ProPublica uncovered in a recidivism algorithm that favored white criminals over black criminals and Gary Rubinstein discovered in a flawed teacher evaluation algorithm used to dismiss educators who scored poorly.

O'Neil has a name for algorithms like these: She calls them weapons of math destruction (see sidebar). "When we build algorithms -- and I'm a data scientist, so I build algorithms -- I am making a bunch of subjective decisions," she said. "And when I present the results of those subjective decisions as objective, unbiased truth, I am lying."

'Weapons of Math Destruction'

Cathy O'Neil labels algorithms as "weapons of math destruction" if they meet these three criteria:

  1. They are widespread. "They're making important decisions about a lot of people," she said. "I'm going to give you an example: getting a job, getting a credit card, getting insurance, going to college, going to prison."
  2. They are secret. "People don't understand how they're being scored, they can't appeal, and they often don't understand that they're being scored."
  3. They are destructive. "They are unfair to the individual -- and often tons of individuals, hundreds of thousands of individuals who are unfairly being denied something they deserve."

Source: Cathy O'Neil, author of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. To hear more from O'Neil, watch her recent TED Talk.

A call for explainable AI

The two presentations, part of the opening keynote series on the first day at Strata, were a warning cry to the data scientists who are helping build a more algorithmic society. The keynotes deviated in tone from the zealousness and passion often heard at artificial intelligence and machine learning events today, but O'Neil and Lavigne weren't seen as wet blankets.

Despite the AI media frenzy, companies and governments are concerned about the machine learning black box. The EU's General Data Protection Regulation, which takes effect next year, includes a right to explanation clause, or an explanation of how a model made a decision. Plus, heavily regulated industries are required to show that the models they're using aren't biased, according to Doug Henschen, analyst at Constellation Research Inc., based in Cupertino, Calif., and a Strata conference attendee.

Some in the industry are referring to this push to pry open the black box of machine learning as explainable AI -- often truncated as XAI. Some are calling it FAT ML, or fairness, accountability and transparency in machine learning. Regardless of the acronym, the opaqueness associated with machine learning is a real issue for companies, which are making their reservations known to vendors, Henschen said. "That's why you're seeing a lot of [vendor] announcements around this idea of transparency," he said.

He pointed to two product launches that were made the same week as the Strata conference:'s machine learning interpretability with its Driverless AI product, an enterprise platform that's powered by Nvidia DGX Systems; and Microsoft's next generation of Azure Machine Learning announcement, which includes an Azure Machine Learning Workbench.

Pulling back the covers

The two products provide different functionality, but both are directed at making machine learning models more transparent. One of the new features in the Azure Machine Learning Workbench is a model management service, which claims to give developers a view into the development lifecycle of a machine learning model from creator to source code to training data.

Model management is a significant pain point for companies, according to Matt Winkler, group program manager for machine learning at Microsoft. "And while we've got a lot of background in the software space for managing components, we don't really have a lot of those capabilities in the data science, ML/AI space," he said.

Microsoft's model management service doesn't pull back the covers on how machine learning models make decisions, but Winkler said it helps to lay the groundwork for doing so. "You can actually go in and debug why a decision got made," he said. The service can be helpful to companies that have an obligation to explain how a decision gets made because it helps to expose "the methods, tools and frameworks being used," he said.

H2O's new features include techniques such as "surrogate models" that can help interpret the outcomes of machine learning models. "These are simple models of complex models," said Patrick Hall, senior director of product at, based in Mountain View, Calif.

Many of the techniques are based on the work done in credit scoring, which uses models to automatically approve customers for credit cards or small loans. Credit lenders are required to provide explanations, often called reason codes, to customers if they're turned down. H2O's machine learning interpretability features aim to provide the same kind of insight into machine learning model predictions.

"They are ways to understand for one person, one customer, one patient, one row in the data set why the model made the decision it made," Hall said.

But H2O's new features don't solve the transparency problem completely. Hall, for example, doesn't recommend the product be used by regulated industries right now. "In machine learning interpretability, we provide approximate explanations for more exact models. And all of the regulation is based on linear models, so it's based on exact explanations for approximate models," he said. "I don't want to say it wouldn't work in regulated industries, but if I'm being honest, that's a very high bar to hit."

Hall, who is in favor of the regulations themselves, said H2O will continue to chip away at the transparency issue in machine learning, looking for and developing new techniques that can provide a level of interpretability capable of standing up to regulatory compliance.

"Personally, I don't think it's a good idea to turn black box models onto decisions that are going to affect people's lives anyway," he said. "It's a recipe for letting people hide behind machine learning to make biased decisions."

Next Steps

A machine learning platform comparison

The promise of deep learning is still only a promise

Machine learning is an opportunity bonanza for CIOs

Dig Deeper on Enterprise artificial intelligence (AI)