News Stay informed about the latest enterprise technology news and product updates.

Machine learning's training data is a security vulnerability

Microsoft's Danah Boyd has a sobering message for CIOs: The data used to train machine learning algorithms is at risk.

For as long as there has been an internet, there has been data manipulation. In 1998, Sergey Brin and Larry Page...

launched Google as a search engine that was designed to get rid of the junk in search results. But the more sophisticated system soon gave rise to more sophisticated data manipulation methods, including Google bombing -- the use of crowdsourcing to bias search results.

History shows that the data manipulation methods used by adversarial actors evolve in lockstep with the technology (see sidebar). That's a fact that should get CIOs' attention, according to data expert Danah Boyd, especially as companies increasingly use machine learning to power predictive features in their applications. Machine learning requires training data -- a lot of it -- to get the algorithms working correctly, and one popular data resource used by developers is the internet.

"I'm watching countless actors trying to develop new, strategic ways to purposefully mess with systems with an eye on messing with the training data that all of you use," said Boyd, a researcher at Microsoft and a presenter at the recent Strata Data Conference in New York. "They are trying to fly below the radar. And if you don't have a structure in place for strategically grappling with how those with an agenda might try to route your best laid plans, you're vulnerable."

To stand up to quickly evolving changes in the threat landscape, Boyd proposed companies return to the old-school practice of rigorous testing and find ways to inject what's known as adversarial thinking into the design and development process. That includes hiring what Boyd called "white hat trolls."

Training data is at risk

The manipulation of data by bad actors is nothing new, Boyd said, citing the relatively benign example that became known as "Rickrolling." In 2007, pranksters disguised hyperlinks to trick people into watching English singer and songwriter Rick Astley's 1978 hit music video, "Never Gonna Give You Up."

While Rickrolling was entertaining, Boyd said the methods behind its success have served as the basis for more nefarious data manipulation. The prank not only taught people how to manipulate systems, but also showed the strategic benefit of going viral and was the antecedent to the disinformation campaigns of the 2016 presidential election.

Pizzagate, the debunked conspiracy theory that linked Hillary Clinton's presidential campaign to human trafficking and child pornography, for example, required a distributed network of dummy accounts, known as sock puppets, to bait journalists into reporting on it, Boyd said.

But the data manipulation methods are about to get a lot more sophisticated. Up until now, the manipulation of algorithmic systems relied on manual methods. With the onset of machine learning, that's about to change, according to Boyd.

To tune machine learning algorithms, developers often turn to the internet for training data -- it is, after all, a virtual treasure trove of the stuff. Open APIs from Twitter and Reddit, for example, are popular training data resources. Developers scrub them of problematic content and language, but the data-cleansing techniques are no match for the methods used by adversarial actors, according to Boyd.

Nicolas Papernot, a computer science and engineering grad student at Penn State University, published a paper last year on his experiments with computer vision. He altered images of stop signs so the neural nets saw them as yield signs. Here's the detail that should make CIOs nervous: The changes could not be detected by human eyes.

The most successful injection attacks on machine learning models are happening in research, but the methods, which are evolving, will no doubt be aimed at mainstream models soon, according to Boyd. "It's time we started building technical antibodies," she said.

Data manipulation through the years

2003: Google bomb. Former U.S. senator Rick Santorum was Google-bombed when LGBT activist Dan Savage, in protest of Santorum's negative views on homosexuality and opposition to same-sex marriage, used crowdsourcing to redefine "Santorum," when searched for in Google, as a word associated with a sexually explicit term.

2007: Rickrolling. Pranksters used the bait-and-switch tactic of a disguised hyperlink to trick people into watching videos that were interrupted by Rick Astley's 1978 hit "Never Gonna Give You Up." The prank was a sensation; it reintroduced the single to the charts, and Astley even made an appearance in the 2008 Macy's Thanksgiving Day Parade.

2015: Radical weev. Andrew Auernheimer, who goes by the name weev, manipulated a Twitter advertisement system to promote white supremacist ads to users who would be most offended.

2016: Pizzagate. The conspiracy theory linked Hillary Clinton's campaign to human trafficking and child pornography. In an environment where people don't trust the media, the media's refutation of the conspiracy prompts those people to believe there's something to it and "self-investigate," according to Danah Boyd, researcher at Microsoft. Pizzagate resulted in the shooting at a D.C. pizzeria by one such vigilante. 

Needed: Adversarial thinking

One antibody is to return to what Boyd called "a culture of test." Today, the technology industry often relies on A/B testing, or what Boyd described as "the perpetual beta," essentially turning customers into a quality-assurance (QA) department. But when members of the QA department are also looking to expose bugs in the system for their own gain, CIOs might need to rethink the process.

"QA wasn't simply about finding bugs," Boyd said. "It was also about integrating adversarial thinking into the design and development process. That's a lot of what we lost."

She pointed to cutting-edge researchers who are doing just that -- building adversarial thinking into the development of machine learning systems. Generative adversarial networks (GANs) are one example. GANs use two unsupervised neural nets. One generates data that looks like the data it was trained on; the other discriminates between generated data and real data. The two neural nets operate a zero-sum game until the generated data is indistinguishable from the real data.

But GANs don't go far enough, according to Boyd. "We need to actively and intentionally build a culture of adversarial testing, auditing and learning into our development practice," she said. 'We need to build analytic approaches to accept the biases of any data set we use. And we need to build tools to monitor how the systems evolve with as much effort as we build the models in the first place."

And CIOs might want to go one step further and take a page out of Matt Goerzen's book: Invite white hat trolls to mess with systems and help companies understand system vulnerabilities -- just like companies do with white hat hackers.

"We no longer have the luxury of only thinking about the world we want to build," Boyd said. "We must also start thinking about how others might start to manipulate our systems [and] undermine our technologies with an eye to doing harm and causing chaos."

Next Steps

AI giants are paving the way

Digital transformation comes before enterprise AI

The marketing department's next colleague: a bot

Dig Deeper on Enterprise artificial intelligence (AI)

PRO+

Content

Find more PRO+ content and other member only offers, here.

Join the conversation

1 comment

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

Would the concept of "white hat trolls" be an asset to your company? Why or why not?
Cancel

-ADS BY GOOGLE

SearchCompliance

SearchHealthIT

SearchCloudComputing

SearchMobileComputing

SearchDataCenter

Close