Emotional AI: Teaching Computers Empathy with Pascale Fung - #9

Key Takeaways

[Music] hello and welcome to another episode of twiml talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam charington once again the recording you're about to hear is part of a series of interviews are recorded live from the O'Reilly Ai and strata conferences in New York City my guest this time is Pascal fun professor of electrical and computer engineering at Hong Kong University of Science and Technology Pascal gave a really interesting presentation at the AI conference focused on how we teach computers and robots to understand human emotion and be empathetic she also had some really interesting things to say about the theoretical foundations of the various modern approaches to speech understanding and we dig into all of this in our conversation as always I'll be linking to Pascal and her research in the show notes which you'll be able to find at Twi ai.com talk9 as is unfortunately the case with my other field recordings there's a bit of unavoidable background noise but it's not too bad and now on to the show [Music] all right hey everyone I'm here at the O'Reilly AI conference and um with Pascal Fung who is a professor of electrical engineering yes at Hong Kong University of Science and Technology right uh and I sat in on her talk earlier on uh emotions and Ai and how do we enable computers to recognize emotions and she graciously agreed to spend a few minutes with us to tell us a little bit about what she's working on so welcome Pascal thank you uh how about we start with talking a little bit about your background and the the kinds of things you work on sure so my background I am a electrical engineer and computer scientist and I've been working on speech recognition since 1988 and then move on to statistical machine translation in the '90s and after I became a professor at HK us I lead a team working on speech language and more recently on emotion and mood recognition or sort of using um statistical modeling and machine learning methods um that's my you know technical background I worked at um different places before I was a student while and while working on my PhD thesis at Bell Labs I was very lucky to work um with some of the best people uh in the area and uh in the early '90s when I started my thesis it was when um the uh the field of natural language processing was transforming from a heavily knowledge-based Linguistics based field to statistical modeling and at the time statistical modeling for language was very controversial people actually didn't believe you could learn language or study language or understand language with just statistics but um you know 20 20 years later now that is the mainstream approach almost every everything we do is with um machine learning sta phys schal modeling in in natural language processing one of the slides you you put up had a quote from uh a professor who I think you mentioned you work with that said for every linguist I fire J yeah he's uh he's sort of uh uh the father of statistical speech recognition so we owe the field owes a lot to him him he passed away a few years ago and then this quote of his was yeah was controversial but uh what was the quote the quote was that every time I fire linguist the speech recognition accuracy goes up so um it came from the at the time at IBM he was leading the IBM group uh with a uh group of mathematicians and information theoreticians to work on the problems of uh speech recognition Okay which was previously um worked on using knowledge based approach in Ai and um actually he his group wasn't allowed to use that approach they somehow they did it anyways so there was always a little bit of conflict between the knowledge-based AI community and the um at the time the statistical minded um people so there were Papers written about the impr prives versus rationalists mhm um in the '90s and uh you know there was uh I remember my first conference presentation on a statistical based natural language processing paper I was yelled at by some of the senior people in the field wow um yeah those were the days but now it's totally noncontroversial at all because you can see you know machine learning is everywhere right and uh yeah right so um he said that because um the approach he post his group proposed was very radical at the time which is not looking at how to imitate human at all but just looking at the input and output where of the task so for example from for speech recognition the input is um speech speech wave found uh wave forms and the output should be words right and for machine translation the input could be French and output should be English right and they basically treated um all these problems as a kind of information theoretic problems to solve MH so a different mathematical approach from the traditional knowledge-based AI Approach at the time can you maybe give a a 30,000 foot background of information Theory and how it plays into all this okay I'll try so information theory was invented by uh the entire field came from a paper written by claw Shannon in 1948 called the mathematical theory of um communication so um it basically looks at you know the information coding uh for example if you want to transmit um telephone signal through telephone cable there's only that much information you can transmit and uh how many simultaneous calls can you transmit in one cable it's limited by by by by physics actually right and uh so the information Theory really is talking about you know how do you encode um transmit and decode information mhm so the earliest application was of course uh telephone systems um so no coin so it was no accident that claw shanon was also at baps M and then later on um information theory was then applied actually in the at the time when CLA shanon came up with this information theory he already had the paper on the um information of language oh really yes yes he actually wrote a paper uh very early about how language can be encoded and learned okay and uh so um he was one of the earlier pioneers of AI even though people don't think of him always that way I had no idea yes and he actually also had a Scientific American paper on the the first chess playing game uh chess playing algorithm okay 60s so yeah so information Theory became an entire field of research and it's apply widely in many many different areas um uh most importantly telecommunications Communications and then of course in all the statistical learning uh field we also use information theory for example we look at how how we can learn uh how we can learn to uh um model a language from an information theoretic point of view so it's more like a uh um you can you think of as message en coding kind of way yeah rather than linguistically motivated way so it's a different way of looking at uh mathematical approach of looking at problems right you mentioned that in your talk and I found that fascinating it it never occurred to me I think you you pose uh the idea of thinking of the machine translation problem as as you've got this uh you've got this message that is you're trying to translate French to English you've got this message that's a noisy English how do you clean up the noise and get to the true English ex so for example exactly that so for example the word orders are different in different languages so it can be you can think of that word reordering as a kind of a distortion right and then uh some of the words uh actually in French and English some of the words are the same like 40% overl right uh yeah but other words are different so you can again think of a word that's different in a different language being a kind of noise I mean a kind of distortion I'm sorry and if you can learn that Distortion then you learn the translation so and that is some information theoretic approach that's what Google Translates still based on mhm so interesting so how starting from what are some of the um you know the algorithms or approaches that kind of come out of the information Theory background like how is it applied more concretely to that problem okay so for example all modern speech recognition software is based on the uh noisy Channel model okay so the whole idea of you can train a speech recognizer uh with lots of data so uh let's say voice search and all that it's all based on uh what other people have said and it's based on these days millions of hours of speech data like Siri for example y it's trained on this data and then um it uses uh um different kind of machine learning method mhm and uh but all these speech recognition uh methodologies based on one big Paradigm is still the noisy Channel model which is speech in words out yeah right through the through this channel now the latest approaches um uh has turned part of this um these methods into using deep learning to replace some modules for example replacing um how phone names can be modeled or replacing part of the uh predicting what word will come after which word so some of this is now enabled by Deep learning but the whole Paradigm is still a a information Theory approach similarly for for things like Google translate it's still based on the um what I just talked about the NY Channel model okay and recently there's some research work on using neuronet but um um we haven't seen any um commercial application that's that claims to be using neuronet for so end to end neuronet for for Mission translation yet okay mhm okay um so how did you how did you um kind of get in not to mention all the encryption software all that is based on information Theory right yeah okay that is not my field but that's actually a main application for for this kind of work yeah okay awesome awesome how did you get into the study of emotion how did I get into the study of emotion basically we noticed that um um for so I I've been working on spoken language understanding for for a long time yeah and uh I've participated on uh different efforts from different generations of um virtual agents what we call virtual assistants today okay so the earliest system um in the 80s was uh funded by DARPA which was a ticket booking system where you call and say I want to go from here to there and then it's trying to um uh uh to to book a ticket for you okay and from that time on we have seen different generations of Dial Systems up until today we have Siri and cotana right and working on these Dial Systems I've noticed that we've always sort of just look at literal meaning um of a uh user query mhm so the machine just pays attention to you know what is the destination the origin City uh how many Texs do you want um you know what kind of restaurant you're looking for so very uh sort of um literal interpretation of your query which means that your cury has to be very clear these days people always complain oh Siri doesn't work well and all that um to us um researchers we can see why it doesn't work well because the system assumes uh users to be very clear and say explicitly what they really uh want to get right and you unless the are lawyers I mean very few people talk that way you know we talk naturally we expect you to understand what I mean right and for example you just laughed because you know I was trying to be funny and that kind of information is completely lost in our dialogue systems of previous generations and but it is very very important for True communication if we want to go go past uh the current sort of um um sort of a plateau of understanding we have have to also incorporate the understanding of emotion intent and all that right uh in addition to understanding the meaning of the words so um that's when I started working on incorporating so I do not recogniz I don't work on recognizing emotion for emotion sake it's really recognizing emotion for communications um so it's what I that's what I call empathy module MH um and then when I look at um the future applications princi we do uh some of the most immediate applications that immediate in the sense that in the next 20 to 30 years we see wipr application will be um health care and elderly care because um by the year 20 um 50 there will be more old people than young children in the world so and uh so Elderly Care is a it's a big area and the governments will be running out of resources human resources to the take care of these elderly so they we're going to need a lot of machine assistance now my mother lives um spends a lot of time alone she's very independent she's uh in her 70s she doesn't want to live with me um and she wants to be left alone to do her own thing and I'm always worried and uh in her independence you know I I worry about her health uh she seems healthy when I see her but um at her age I want to make sure that she's fine uh right now for example I want to know she's fine right right so this kind of uh um I I want to know her her health conditions but also her mental conditions so if she doesn't want to live with with me now how about if I have if we build a home robot who can be there uh at you know at her home or be around around her and uh converse with her sometimes just to get a sense of how she's doing simple things right and then sends me a message so I know how she I mean or sends me a curve of her vital signs in addition to her emotional state then um that will help the guilty children busy guilty children but also help the elderly because you know a lot of people have um emergency problems like a heart attack or something that could have been they could have been saved if someone knew and then there are others who can get lonely and depressed and um um and then they can also be helped by machine to some extent not completely by Machin you know living with just machines is also very um sad but right but when there's not enough humans around people around then the machines can help MH so this is why I want to work on empathetic uh robots people in the case of a crisis situation like a heart attack where does empathy and um where does that come in I think heart attack um it's it's basically it's an emergency then the robot has to basically alert called ambulance right that's the first thing but what what empathy comes into play is that U so daily reminder to take medicine right uh in uh some of the Aging studies they people have found that a lot of elderly they don't want to take medicine to some unless somebody talks to them okay like um sometimes you have to um sort of entice them I mean some elderlies are like children mhm so in that case so the machine doesn't just say take your medicine and repeatedly insisting that you take your medicine like with the same command that will be extremely annoying and it will have the opposite effect right so the machine needs to know that the elderly is hesitant or resisting and how is the person's patient's um um emotional state and to know whether now uh the machine can insist or it's time to call a doctor or a nurse or whether you know telling the patient a a bedtime story will soothe them um so that requires empathy if you think about all the nurses you know in the hospital a lot of their job a lot of their work and their tasks are very repetitive um and uh sometimes the better nurses are the ones who really have a very good bedside manner right and what is a bedside manner other than being empathetic just being empathetic you know for both doctors and nurses so if you want doctors and nurses to be empathetic obviously you want the healthcare robots to be empathetic right right your talk I interpreted its focus on uh empathy recognition but your description is also talking about what you might call generative empathy right right so empathy has two sides it's the emotion recognition right and then the the appropriate emotional response okay so empathy I only in my talk I focus mostly on the emotional recognition part because that is hard until we can recognize emotion um we don't know how to react right right so I focus on that and uh but the response part um I I talked a little bit at the end that we're we're trying to learn the appropriate response mhm as well from uh data so also using uh deep learning mhm you mentioned Healthcare as a a use case I think there's often for a while now uh people have talked about a customer service use case where right sure sure you know the the the hold line will recognize when you're getting frustrated sure sure in fact uh at AT&T Bell apps in the 90s already there um um they uh a group uh worked um came up with a system called how may I help you MH so when you call the AT&T line uh it's a it's a virtual assistant uh virtual operator that talks to you first and says how may I help you and you basically say whatever you want and then it goes to different categories yeah and that is the intention classifier okay and also at the time at AT&T had internal programs to analyze all this call center data to see whether people upset they're happy and all that so that is already the beginning of emotional recognition that was my first um contact with emotional recognition and it was for for customer service indeed it is a big area um but I don't get the sense that it's widely deployed or at least not in a way that um I would see as an everyday person maybe it's more back and so this is the thing because it's not consumer facing it's really for it's really in the in in the area the realm of data analytics yeah so it's more of a a tool used by corporations to um improve the efficiency of their call centers kind know Performance Management they do do that there there are companies that um provide technology to to customer service for this yeah yeah cuz that my my place in line never seems to Accel because I'm angry right right um so you also talk Al about um the use of convolutional neuron Nets in recognizing emotion and kind of Drew some interesting correlations across um you basically that the CNN were able to functionally approximate clear functionality yeah the human some perception system indeed we so um so I think my talk I started out by saying okay traditionally speech recognition look at these uh human human uh sound perception system and try to imitate that but we sort of Hit the bottom neck and we had to move away from that and uh go with the information theoretic approach where we actually try we don't try to imitate human model at all and what's interesting with CNN is that uh a lot of times CNN or other deep neuronet are being used as a black box so we know it works we don't know why right and uh humans being humans are always interesting in knowing why and in fact there is a a a practical reason is that if you ever want to commercialize a technology like that right to provide to your customers they want to know why what it's learning right so um so we then look take we then took a look at the CNN at different layers of CNN and saw that it was actually um as I mentioned in my talk that it's basically a approximating the filter Bank in our CA um that's connected the auditory system and then we also saw that it's picking up on the uh amplitude the peaks in the amplitude that correspond to different emotions so we thought that was very interesting that we can actually see uh what's going on for example it was always um CNN was first applied to image recognition um they use like seven eight nine layers of um uh um CNN to achieve that purpose and in image recognition they were able to see that each layer so for example one layers recognizing the edges of image and the other layers recognizing um um maybe something else some features on your face and all that so it's all very obvious and really nice and we were never able to explain how deep neuron net works on speech and language so it was interesting to see why it works on emotion we still are not able to figure out what it's doing on languages so what each layer is recognizing uh we were hoping that each layer will correspond to some linguistic um function such as syntax or some we haven't seen anything that that that neat yet so interesting interesting so it's not blinding learning learning something it's learning something uh which has a physical meaning mhm so mhm but you made the point that it's it's also an error to correlate it too tightly to brain function because that doesn't really it doesn't it's not because no so even though I say it approximates human perception system it's really the hearing system right it's not the understanding part it's the hearing system and we know exactly how our hearing system function we know very very well we don't guess but how how our minds functioning understanding the meaning we don't know we we're activating our as you know I mentioned 100 billion neurons and 100 trillion synopsis to get that um until we have neon network of that size it's hard to um come up with something similar to human cognitive um ability MH so it's not we don't have that so that's why I say it's no coincidence that um we can use we can explain what CNN is doing for perception so speech recognition and emotional recognition are both perception um problems perception is actually uh easy because we understand human perception how how our skin feels the temperature we actually understand the physics of that very very well but once you go into understanding which is language understanding which is the trans you know if I want to use a noisy Channel model it would be from words to meaning right once we get into the realm of that uh we are kind of Clueless is to how humans so there are a lot of linguistic theories about how humans think how humans understand but you know for every linguis Theory there are other people who will say no right so there's no no um there's no scientific truth that we all share right now about how human mind understands language or understands anything else how do you know a video of a cat is funny or not how does our mind interpret humor we actually don't know right a lot of marketers would like to be able to I know so yeah how do we predict for example one thing uh we were asked since we could classify music we were asked by a company and say hey can you predict whether a song is going to be a hit or not mhm well if we could predict that wouldn't that be amazing uh no you know cuz you know even we look at Big Data of all the past um hit songs yeah it can't learn it cannot predict what the next one not yet U maybe so all we can do right now is use engineering models to approximate input and output you know what I mean it's really a mimicry like just looking at this input can we come up with output that's similar to to the truth mhm so we're not no we're not in any way near to um imitating human minds and that would be even though so even the term deep learning is a new term for something that has been around for a while so it's a particular kind of machine learning right it is uh it is no deeper let's say than other kind of machine learning methods it's it's a terminology and also um new network um it's a very very rudimentary kind of neuron Network you know for speech recognition then may be tens of thousands of neurons MH and for emotional recognition much much fewer MH so that cannot compare to the human brain so yeah one of the things that I noticed in your presentation is the um when you're doing the emotional emotion recognition you're mapping it to kind of the you know these names common names we have for emotion angry sad whatever is that model even too simple or is there an underlying more nuanced model for emotion I'm I'm sure there yeah well so um there's a lot of research done on the underlying model for emotions such as uh veillance arousal you know and uh there are models that try to predict that first before they predict the final label and to interrupt you cuz I saw the slide but these folks haven't veilance arousal is veilance is like the strength of the emotion and no arousal is the strength veilance is the positive and negative of the emotion and so the various you know angry sad cool it's a combination of right they are a combination of different values of veence and arousal this is one emotion Theory so these are models that psychologist came up with uh to try to organize what we know about emotion and um so it's just human Minds we're symbolic animals you know we need to have names we give names to everything yeah so it's just easier for us to give a name to emotions so we know what we're talking about yeah rather than give this vague U you know number right veence arousal if I tell you oh this is veence arousal this and this number you wouldn't know what I'm talking about if I say he's you know he's showing happiness on his face you kind of kind of um know so it's kind of um emotional recognition is kind of like speech recognition when it was only recognizing um ISO related words MH it's oversimplified for sure yeah uh we're not good at all with emotional recognition you saw my slides the performance is nowhere near the ability uh the performance of recognizing words recognizing emotion is much harder right right now one reason as you pointed out is that it's hard to Define what emotion is you know for example maybe it's easy to see if somebody's happy but is he smiling because he's happy or is he smiling because he trying to be polite right and uh also what about emotions like frustration how can you tell um sure some some some things are obvious you know when someone's frustrated if they're rolling their eyes or something but there other times you know from The Voice from the tone of your voice we can tell a lot of things right but we cannot tell uh everything um all the time so it is a long ways to go you saw the accuracy the accuracy these days even the best commercial systems is just like 60% you know uh and most 70% and speech recognition we're talking about above 90% right right so there's a long way to go for emotional recognition and especially some more complex emotions like humor there sarcasm I think sarcasm yes sarcasm and humor and um and uh deceit you know is this person lying right and there are colleagues in the field who have come up with systems that that has 70 some per accuracy in detecting deceit and actually performs better than humans turns out we're not very good at detecting deceit we're not good at all wow and humans are not very good in recognizing emotions yeah uh you know what we found with uh when we did some human subject studies is that we found that um women are better women are more empathetic than men uhuh um that was actually um quantifiable and then women can detect emotion across languages language in language they don't know interesting also better than men in the same language um so you can talk about what the reason you know we're programmed to be mothers we must recognize the emotion of a baby early on and I think there's some Merit to that so although I I'm not a anthropologist I cannot prove that but yeah so interesting um and then the reasons you know you see there are a lot of women who are nurses doesn't mean men cannot be but just happen that way right a lot of the uh caregivers are women you know kindergarten teachers um you know just nannies you know so if we want to build robots we want them to be more like that more empathetic yeah right great well thanks so much for taking the time to sit down with me it was a great discussion I really appreciate it thank you um would you like to share how folks can find your research or are you on any other the social media networks um well it's easy to Google my name okay which is Pascal Fong p a s c a l e Fong is f u n g if you Google my name you come to my uh website which lists all my our Publications our projects and all that and if they're interested they can they can can email me uh via that website as well great great great well thanks so much thank you all right everyone that's it for today's show if you enjoyed this show or have something to add to the discussion please leave a comment on the show notes page at twim ai.com talk9 or tweet to me at Sam cherington or at twiml aai to let me know what you think thanks so much for listening and catch you next time [Music] time