Keeping a Human In-The-Loop: The Limits of AI in Security Systems

June 10, 2021

In the spring of 2020, Croatian chess player Antonio Radic’s YouTube chess channel, which had more than one-million subscribers at the time, was blocked during a chess show. Although YouTube restored access within 24-hours, Radic was never provided with an explanation for the block. He suspected his account was blocked because he referred to the chess match as “Black against White.”

After this incident, Ashiqur KhudaBukhsh, a project scientist at the Carnegie Mellon University’s Language Technologies Institute, ran a test using one of the best AI speech classifiers available to screen several hundred thousand comments on five popular YouTube chess channels. He found 82-percent of the comments had been miscategorized by the AI as engaging in hate speech due to the high use of terms like “black,” “white,” “attack,” and “threat.”

In 2016, Microsoft launched an AI effort to improve their understanding of normal conversations, particularly the slang used by teens and young adults. The project was called “Tay” and intended to use machine learning: as more people talked with Tay, the chatbot would learn how to write more naturally and engage in better conversations. Less than 24-hours after Tay launched, internet users had thoroughly (and apparently purposefully) corrupted the chatbot’s personality. By overwhelming the bot with anti-Semitic, racist, and misogynistic tweets, social media users had managed to turn Tay into a proponent of extremist ideologies.

These are just two examples of the pitfalls that can be faced by the over-reliance on AI for decision-making. While there has been significant progress made in improving AI learning, even the best AI tools still struggle with nuances that come naturally to most people. As a technology company, we realized early-on that the best technologies are those that augment human capabilities, not replace them entirely. It is for this reason that we continue to advocate for a “human in the loop” approach when it comes to AI and the security industry. Sound human instincts and judgement are still, for the moment, irreplaceable assets.

Leave a Comment

Your email address will not be published. Required fields are marked *

Contact Us



[email protected]

Monday–Friday | 9-5 pm EST


Contact Us