Chatbot developed at Carnegie Mellon uses humans to answer questions AIs can't
A digital assistant that taps real people to answer questions just got smarter thanks to artificial intelligence.
The team at Carnegie Mellon University that developed Chorus, a chatbot that uses the power of the crowd to answer questions, added AI-powered systems similar to Siri and Alexa to the mix to create Evorus.
“It's sort of the ever-learning, evolution of Chorus,” said Jeff Bigham, an associate professor in CMU's Human-Computer Interaction Institute. “The reason we made this is that we were kind of underwhelmed by the existing digital assistants out there. Not because they aren't amazing technological achievements but because they are rather narrow.”
Bigham said devices like Amazon's Alexa, Microsoft's Cortana, Google's Hey Google and Apple's Siri are good for answering simple single questions. Complex questions or even follow-up question often stump the devices. Bigham's team created Chorus to demonstrate how a crowd of people could answer more complex questions with surprising speed.
Bigham doesn't look at it as humans versus computers or see Chorus and Evorus as pleas for the survival of people-powered intelligence. Artificial intelligence is coming and the best approach is to figure out what it does best, what humans do best and how they can work together.
“Leveraging the best of what AI and humans have to bring,” Bigham said. “For the first time we're able to hit that middle spot where it is leveraging both automated systems and human systems.”
Chorus makes use of Amazon's Mechanical Turk, a network of people who do on-demand work. When someone asks Chorus a question, it submits the question to the network and people begin submitting answers that are voted on until the best answer is selected and sent. Answers take only a few minutes and the people who worked on the query are paid. The average chorus session costs about $2.48.
Evorus adds AI and machine learning to the process. The system will now remember previous questions and answers and start to suggest answers before it asks people for help. Evorus will also help select the best answers with less human involvement.
During a five-month test with 80 people asking 181 questions, automated answers were chosen 12 percent of the time and human voting on answers dropped by 14 percent. The cost to reply to each message dropped by 33 percent.
“You can really start to see how we could achieve both the reliability of the human workers and the low-latency and the low cost of the chatbots,” Bigham said.
Brigham said most digital assistants don't know how to answer a question like, “What sort of pet should I get?” Siri pulls up a list of websites about the question. Evorus responded with questions asking if the person lives in the city, has a lot of room, is gone often on vacation or travels for work. It asked if the person has a backyard and if the person has ever owned a pet. Through these questions and more answers, the chatbot identifies the appropriate pet. In my case, it was a dog.
I also asked Evorus what I should wear Wednesday as the weather in Pittsburgh shifted from rain to snow to rain. Siri told me the weather when I asked her. Evorus asked what the weather was like and started suggesting warm clothing options.
“Can you wear jeans to work? How about a sweater and some warm pants?” Evorus responded. “A leather jacket would be suitable for snowy conditions. A shirt layered with a long-sleeved shirt and casual slacks would work nicely.”
Evorus uses Google Hangouts to communicate. People can try it for free by signing up at http://talkingtothecrowd.org/ . The chatbot is far from perfect, so expect bugs and hiccups.
Aaron Aupperlee is a Tribune-Review staff writer. Reach him at firstname.lastname@example.org, 412-336-8448 or via Twitter @tinynotebook.