Updated Turing Test Proves How Dumb AI Chatbots Are
Toggle Dark Mode
Artificial intelligence has been on humanity’s radar for quite some time. But recent developments such as AlphaGo’s triumph over Lee Sedol and Watson winning the quiz show Jeopardy have given us an even greater and discomfiting awareness of the vast strides that have been made in this field.
On the other hand, Microsoft’s famously bungled attempt to create an AI Twitter account, which started spouting racist and genocidal vitriol within a day, reminds us that an AI-orchestrated apocalypse is still far off. Indeed, the simple fact of the matter that all iPhone users realize is that Siri hasn’t even quite mastered the nuances of human colloquialisms and syntax.
This is precisely the point that the Winograd Schema Challenge seeks to prove. In the contest, AI-enabled computers are tested with ambiguously worded sentences that humans can easily understand. As Engadget reports, the participating AI programs were unable to parse the following sentence: “The city councilmen refused the demonstrators a permit because they feared violence.” The problem this sentence presents for AI is that it’s unclear who “they” precisely refers to. People, on the other hand, are able to use their social knowledge and context to figure out the sentence’s meaning
The contest awards $25,000 to participants that are able to correctly interpret such sentences with a success rate of 90% or more. So far, none have prevailed. The two best entrants were able to choose correctly only 48% of the time, which is only slightly better than the 45% success rate that you get from guessing randomly.
The contest proves that computers are still atrocious at understanding and acquiring common sense knowledge. Facebook and Google are proponents of developing the natural language capabilities of computers to improve their personal assistants and chatbots and help aid this exact weakness. The two companies, unfortunately, did not participate in the challenge.
The proliferation of voice assistants, image-recognition technology, and chatbots may give the average consumer the false perception that machines are becoming great at learning human languages. And while films like Ex Machina raise the disconcerting possibility that humans may not be the only sentient entities inhabiting this planet by this century’s end, this contest has proven that there is still a ways to go.
