Enhancing Customer Experiences with Emotional AI, w/ Rana el Kaliouby - #35

Key Takeaways

[Music] hello and welcome to another episode of twiml talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam charington the show you're about to hear is part five of our five-part O'Reilly AI New York series sponsored by Intel Nirvana now if you've listened to the entire series you've heard me say this again and again but I am super grateful to them for helping make this series possible and I'm also excited about the products and projects that they launch at the O'Reilly AI conference including version 2.0 of their nurana framework and their new Nirvana graph project be sure to check them out at Intel nirvana.com and if you haven't already listened to the first show in this series where I interviewed navine raal the head of Intel's AI products group and hanlin Tang an algorithms engineer on that team it's twiml Talk number 31 and you definitely want to check it out my guest for this show is RA Al CBI ra is co-founder and CEO of affectiva if you remember my conversation about emotional AI with Pascal fun from last year's O'Reilly AI conference you're going to love this one my conversation with ra kind of picks up where that one left off with a focus on how her company is bringing emotion AI services to Market it's a great conversation and I know you'll love the show let's get to [Music] it all right everyone I am on the line with ra L calubi who is the co-founder and CEO of affectiva Ra was here at the O'Reilly AI conference and delivered a presentation on emotional AI this topic may sound familiar to you because last year at the O'Reilly AI conference I interviewed one of her colleagues in the field Pascal fun and so I'm looking forward to speaking with you ra to learn a little bit about affectiva and how you're applying emotional Ai and making it useful for Enterprises welcome to the show ra thanks Sam thanks for having me so you know when we look at human intelligence there's many aspects to that and one important aspect is emotional intelligence and that references the ability that we each have to read and understand other people's emotional and social and mental States and then adapt our Behavior accordingly and So based on years of research we know that people who have higher emotional intelligence tend to do better in life they're more likable they're more persuasive they are more successful in their professional personal lives and they even live longer so our thesis is that advanced AI systems and autonomous vehicles and you know all the technologies that are surrounding us today they all have high cognitive intelligence but they're really lacking in this emotional intelligence area and we are you know my team and I are on a mission to humanize technology by bringing in artificial emotional intelligence we see that there are two sides this coin one is to build Technologies and digital experiences and even devices that can sense emotional you know consumer's emotional states in real time and adapt to that accordingly so that's kind of one stream of work that we do the other is more around Gathering data about how people feel towards things it could be online video ads or product Concepts or new experiences and we collect all that data and aggregate it and as it turns out this information is very very important and meaningful for organizations as well as consumers interesting interesting how did you get started in this area what led you to study emotional AI so my own background uh as an undergraduate I studied computer science I was fascinated by the role technology plays in actually changing how we connect with one another as human beings and then I got my first you know living abroad opportunity I got the opportunity to do my PhD at Cambridge University and at the time I used to live in Egypt so I moved to Cambridge UK and I realized I was spending more time with my laptop than I was with any other human being kind of sad and of course it was frustrating because even though we were intimate this laptop and I it had absolutely no idea how I was feeling so there were often times when I would be stressed because I had you know a project deadline or a paper deadline and it was just completely oblivious to all of that but I also realized that it was my main portal of communication back to my family back home and this was before we had smartphones and I think it's still true with smartphones right like it's you think of these devices and they are becoming the main portal of communication absolutely exactly but yet I I felt like even though you know this kind of you know social platforms have allowed us and messaging apps and all that have allowed us to connect with more people the quality of the connection is quite poor because a lot of the nuances and the nonverbals are kind of lost in cyers space so I you know one example that really kind of stuck in my brain and and has motivated a lot of my research I remember feeling very lonely in Cambridge and you know I'd be chatting with my family back home and I'd be sometimes in tears like I'd be really upset and they would would have absolutely no clue until I send them that little Emoji with tears you know and then be like oh so you're kind of sad and it just it just it was very explicit and it and it took away from from the Gen you know what do you call it the sincerity of the emotion and that made me realize that we could do better and so I started imagining what would it take to build a computer that had some emotional intelligence skills the way my friend would or my mom does you know and just got very interested in the psychology aspect of human emotional intelligence and then how to replicate that in a machine it is really interesting the degree to which you know something as simple as an emoji expands the bandwidth we have for communicating emotion and so I'm curious how you you know with that as kind of a backdrop tell us a little bit about affectiva and the specific things that you guys are offering to to allow companies to better you know harness emotion yeah so like all great things I came to affectiva you know by accident so my plan my career plan was to do my PhD and then apply for a faculty pos position and take an academic and research career I ended ended up joining Professor rosalin Picard's group at MIT media lab so she wrote the book affective Computing in the late 90s and that's kind of what inspired me to get into this research space okay and I met with her and ended up joining her lab at MIT in 2006 and she and I collaborated on the application of emotion sensing to specifically mental health and being at MIT media lab twice a year we would invite all these sponsor companies and they would say you know what we're very interested in this work you're doing like we want to apply to advertising research or you know monitoring drivers in cars there was just like a a whole slew of applications and after about like four years of doing this at MIT we decided that there was a commercial opportunity and that the market was ready for you know this kind of technology so we spun affectiva out of the media lab okay with the vision of bringing emotion AI to the market in a commercializable way and it's been a few years we found that you know we still have a very broad Vision we still see that this technology is a core AI capability that's going to play in a lot of areas the ideas to sequence the markets and so the first use case if you like where we've applied this technology is an advertising research and market research and the problem we're solving there is you know like think about Coca-Cola or kellogs or Pepsi they want to understand how people are engaging emotionally with their content interesting yeah because emotions Drive Behavior so right right if you feel very positive or inspired by an ad you're going to share it with your network you're going to go out and you know you're you're going to build a positive brand Affinity or loyalty you're going to buy the product so a lot of you know a lot of the marketers and advertisers we work with want to elicit a strong positive emotion or sometimes it's sentimental and and and it's not necessarily positive but they want to drive an emotional Journey but they struggled to measure how effective they are at doing it and before our technology what they did is they just asked people okay how are you feeling about this and we know that this is very unreliable and susceptible to all sorts of bias right it's expensive as well correct so what we do instead is we use our technology which is essentially a machine learning and a computer vision platform that uses a camera on any device and it reads your facial expressions in real time and it Maps it to an emotional state oh interesting so do you position it as kind of a a virtual focus group type of opportunity or is it more I guess passive observation of people engaging in you know real media experiences does that question make sense yes absolutely it's both so okay most of the work we've done to date is is the laders so we send people a URL and we say okay if you click on this do we have your permission to turn the camera on while you're watching this movie trailer and if they say yes all they do is they you know they watch the movie trailer the camera's on it's streaming their responses we do that for a thousand people and then we aggregate their responses and we and that dashboard that moment by moment journey is what we present to you know to the client or the TV studio okay interesting I can't help but think about a conversation that I had yesterday with Raza Zade do you know raaza from matroid I don't maybe I should should connect us what he and his company is doing is they are indexing TV streams and allowing companies to build detectors that can identify you know different brands and objects in the TV screen in the streams not necessarily TV but you know media streams and I wonder if the degree to which uh some of the brands as they have you know the product placements are huge in some of these TV shows and things like that you know if they'd get some additional Insight by having kind of real time you know detection of emotional response to product placement kind of in cidu in context in uh you know a media as media is being consumed random thought we often get asked you know we we'll often get approached by an Advertiser who you you know the ad they are featuring two celebrities and they want to know which which version resonates the most or yeah they often play with different product placements or different music in the background and the idea is to is to really detect because it's it's often hard for people to articulate if they've liked something it's very hard to articulate what is it that resonated and so we can detect your visceral reaction on a frame by frame basis so we can say okay when you know when the product revealed happened this is what we saw and I've got to imagine that for you know obviously emotion is very subtle and expression of emotion particularly if you we're I'm assuming you're using laptop cameras and we're look primarily looking at face you know we're talking about fine motor movements things like that how do you convince the skeptic that you know this technology can really accurately pinpoint an emotional response so when we first started doing this work the very first question we had to answer was you know when you're in front of a laptop do you even emote right because there's no other human right but I'm I'm smiling and laughing as I in front of my laptop now as as I'm engaging with you but yes I can I I definitely get the question right if you're just watching an online video ad do you even show any expressions and so we collected I mean now you know to date we've collected five and a half million face videos from 75 countries in the world of people interacting with their laptops with their phones playing games driving in cars so we have a very substantive data set of people's emotions but when we first started we did not have all this data and so we initially collected data of people watching Super Bowl ads and it was you know we knew that some of them were successful and went viral and garnered a lot of attention and some of them didn't and we found that yes people emote Expressions even if there is no other human being present and we saw the entire gamut of Expressions we saw laughter we saw smirks when people were kind of you know really skeptical we saw confusion we saw boredom and that data set became the data that we use to train our algorithms so we use you know deep learning to train models to detect very very nuanced Expressions you know we're the machine's now able to or the algorithm is now able to identify 20 different facial expressions and it Maps them to eight different emotional state States and just to give you an example some of the Expressions that we're able to identify now include a smile or a brow Furrow which are kind of obvious but then it also is able to read like a lip suck or an ice Quint you know like very very fleeting nuanced expressions and and that that's all because we have the data to drive the machine learning and I recall from pcal that there there's a standardized chart for tions like it's there are two axes and the emotions are mapped out on these axes do you know what I'm referring to and can you remind me what that's called yes so there's a circumplex model of emotions and basically that Maps or it's the dimensional model of emotions so it it posits that you can think of emotions in terms of these main Dimensions like veilance veilance is how positive or negative your experience is and that actually is really you can get a very accurate measure of veilance from the face and then the other axes is arousal which kind of the intensity of the emotion okay and that is best Quantified through your voice and also your physiology like if you're measuring heart rate or heart rate variability or skin conductance okay and so is the implication then that your technology is primarily concerned with veilance or do you attempt to measure both and map everything onto this two-dimensional model so it's interesting because we have a veilance measure and we have a proxy to arousal we we use like you know level of Engagement which is you know how how how much emotions are you showing and also gaze fixation and stuff like that yeah exactly okay are you paying attention or not are you fidgeting too much so so all of these measures could be kind of indirect proxies to arousal but we're also very interested in measuring discrete emotions so that's an alternative model for thinking about emotional states and that is triangulating exactly what the emotion is so is it Joy is it surprise is it confusion is it contempt and the face again is quite good at the specificity of the emotion whereas if you're looking at you know heart rate variability it's often very hard to depict what kind of emotion is it even though you know you know it's it's a high arousal emotion but it's hard to tell what exactly it is we ALS Al provide our partners and our customers with you know a specific emotion measure and we give you probability scores because sometimes they co-occur sure sure have you applied this at all in the political Sphere not in the most recent political cycle in fact the most recent political cycle we have a partner company called higher view they are in the recruitment space so they work with big companies that hire many many people and they have to screen you know thousands of people and instead of instead of requiring that a candidate sends in a word resume you send in a short video interview so oh wow yep and then they use our technology to rank order the video so you know if you're a sales if you're applying for a sales position or a flight attendant having social skills is is really a key component of that job and so we're able to quantify that and so anyway so they their position was that you know in you know the presidential Ates are the ultimate job interview so they you know they used our technology to analyze both Trump and Clinton's expressions and we published that so it's available online and oh interesting interesting I'll get the URL from you on that and we'll include it in the show notes tell me a little bit about the the underlying technology you said it's based on deep learning how has the technology evolved and how would you characterize the approach yeah so it's so we use learning the idea is that we collect you know hundreds of thousands of images of people and videos of people expressing emotion in the wild so it's not your selfies it's not right it's not like your Facebook profile pictures it's actually people you know sitting in front of a TV and just like chilling right or driving around Boston in their car in the typical bus and traffic so it's very spontaneous data which means that it's very hard data like the background is messy sometimes it's quite dark there's multiple people in the video the expressions are not exaggerated in fact they're very nuanced and subtle so it's real data it's what you see in in the real world and we take that data we get a portion of it labeled for training and a portion of it labeled for validation and that becomes the you know the data we inject into the say the convolutional neural Nets we also have temporal models because expressions are very Dynam damic and and they unfold over time and over the years we've continued to accumulate this data so we now have about 2 billion facial frames not all of it is labeled and we've recently decided as a company that it's time to include other modalities so we've started to research your emotions as they are represented in your voice as well not just your face okay okay makes sense how did you collect all of this data do you have have an app that provides some value to the consumer to incent them to just turn it on when they're driving or watching TV or is it all via clients I mean at some point you had to Kickstart this or cold it and accumulate Data before you had clients how did you go about doing that we partnered with the very very first project we did with was with Forbes and we put this thing on their website and you could just go watch Super Bowl ads with the camera turned on then you would see your emotional profile but since then we've done you know we've done this in 75 countries around the world through Partners so we have partners that are collecting this data you know as part of you know testing ads okay oh it sounds a little bit like was it Okay Cupid or the dating site that did the quizzes to get people to like logically profile themselves one of the dating services was known for doing these small personality quizzes to that were kind of entertaining for the people that were participating in them but as a result they were helping the company collect lots of profile data and you're doing something similar it sounds like with your partners around ads and other media experiences so you're using convolutional neuron Nets and you mentioned some time domain neuronet as well like lstms I'm imagining what is the like can you tell me a little bit about the data pipeline how do you manage all of this video and and get it ready to train the the these networks yeah so we have a cloud-based infrastructure that we we've invested a lot into building that over the years so all of our data goes into S3 then we use Amazon web services to to do all the TR so all the training is done in the cloud we also built our own cloud-based data labeling infrast or video labeling infrastructure so and we have a team of certified you know face coders who you know basically their job is to watch these videos and identify when an emotional expression occurs and then we've built again in the cloud we have this infrastructure that takes all this data and says okay there's agreement on this set of videos there's no agreement on this set of videos If there is agreement on a set of videos that becomes you know split it gets split into training and validation data and if there's no agreement it goes back to the labeling pile we also use Active Learning because we have a ton of videos a lot of it has no emotions like if you're driving around Boston for the most part you're just neutral we don't necessarily care about all these frames so we use what we call Active Learning which is we use the machine learning models to bootstrap the labeling so the machine takes a first stab at the labels and then based on what the machine says we either Fork it to the lab to actual human labelers for more labeling and more fine-tuning or we kind of you know not throw that data away but we park the data if it's not interesting so it's a human in the loop labeling infrastructure and then once we have that we say for instance you know so let me take a step back we do have a cloud-based set of apis where you would send a set of frames or videos and we would send you back the emotion probabilities in that video and a lot of our partners use the technology that way we call it emotion as a service but we also have a number of partners that are developing real-time experiences like think of a social robot that needs to respond to you in real time or Amazon Alexa kind of device right and so we can't be streaming you know tons of videos to the cloud and waiting for a response and in that case we have developed on device software development kits so that you could integrate the core engine into your device or your app and have it run have everything run on the device you don't need to talk to the cloud at all and so in that case there's a trade-off between accuracy and speed and computational weight you know how lightweight is it and so we run experiments in the cloud you know where we say okay we're going to train with I don't know 200,000 samples and we are going to evaluate this model on how accurate it is across all these different emotions but also how fast it is and how heavy it is and that all happens in the cloud and our scientists you know will evaluate you know we we come in the morning and there's a spreadsheet of all the different experiments that we're run overnight and and our team will kind of evaluate okay which ones if we want one for the mobile SDK then we're going to really care about how fast it is and how lightweight it is does that make sense oh it does it does so this mobile SDK what are the platforms that that runs on we have it running on iOS Android Unity 3D Raspberry Pi which was quite challenging Windows Linux Mac OSX and was Raspberry Pi challenging because of RAM and CPU limitations yeah exactly okay okay and is the idea with a Raspberry Pi type of implementation that it might be the backend compute module for a a camera that's doing you know some kind of set toop box or surveillance type of application we stay away from surveillance applications but but setop box or you know a connected home device that is yeah that is powering a social robot or you know a home monitor MH and why do you stay away from surveillance is that an ideological decision or a technological decision it's ideological so when we spun out my co-founder Rosine peard and I basically made a decision that we would you know in respecting how important and how person personal this data is that we would only work in use cases or Industries where you knew that the camera was on and you knew that you were being analyzed in this way and we would Veer away or steer away from applications where that was not the case we have stayed away from surveillance and security applications yeah well I certainly respect and appreciate that I've seen some very scary surveillance demos recently and to imagine layering on this emotional context to which I'm you know it's one of those things it it's always one of those things where like someone's going to do it at some point but I can appreciate your decision to not be that someone that does it right our thesis is if we're going to spend a lot of our mind share and our passion and our energy solving a problem you know there's a lot of other problems we can solve yeah right right where do you see the space going yeah very exciting so the ultimate vision is that our devices and our Technologies are going to have emotion AI embedded in them we see a world where you know this is going to run passively in your phone it's going to track your mood and on the path to that we're seeing a lot more interest in the automotive space for instance where our kind of Technology can help improve safety especially in a world where we have semi-autonomous vehicles so you can imagine you know your car is on autop pilot but it may need to relinquish control back to you as a driver and so the question becomes you know in a matter of seconds it needs to know are you awake or not are you paying attention or not and that's where our technology comes into play yeah I can also Imagine a world where you've got digital Billboards and you have some driver opt in some driver opt into this system maybe they get you know some benefit or compensation but you're reading their emotion as they're engaging quote unquote with these Billboards lots of interesting applications and I can you know certainly as uh well I was going to talk about you know being a New York City born and bred driver and road rage but I probably won't go there and producers will cut this segment out of the conversation road rage is definitely one of the emotions that we get asked about but there's also is really oh yes yes Road raage drowsiness distraction and you know what too like just general infotainment so a lot of these cars are now rethinking the brand experience the incab brand experience and they want to you know get a sense of the sentiment inside the vehicle not just the driver or the co-pilot but also the passengers and adapt The Experience adapt the lighting adapt the music adapt the general environment and personalize it so we get a lot lot of interest around that as well okay so that's where the business and use cases are going where do you see the technology going to are there things that technology needs to go to enable these you know new and more ubiquitous use cases yeah definitely multimodal so combining both you know computer vision with speech to get a better understanding of of how you feel or you know your social and emotional signals and that's especially important in conversation contexts so it's okay to just focus on your facial expression if you're just watching a movie that's fine but if you are in a conversation with a chatbot or a personal assistant or you know a social robot or your car then combining all these different modalities is going to be key so that's definitely one class of of product development and and kind of you know core science development that we're focused on but just more generally there's a lot of applications in say Healthcare where there has been academic research that's showed that you know your facial expressions and your voice can can be biomarkers for pain can be biomarkers for depression even Sude assessment oh wow so I would like to really see the technology move into that area and that would require you know substantial data collection of a healthy population versus you know a depressed population but it's yeah I I think that's where the technology can be really transformative fantastic fantastic well thank you so much for taking the time to to speak with us these applications this whole emotional space is one that I find really fascinating and I'll definitely be keeping my eye on what you guys are doing and yeah I wish you the best of luck thanks so much thank you thanks for having me all right thanks r all right everyone that is our show thanks so much for listening and for your continued support comments and feedback a special Thanks goes out to our series sponsor Intel Nirvana if you didn't catch the first show in this series where I talked to navine raal the head of Intel's AI product group about how they plan to leverage their leading position and proven history in Silicon Innovation to transform the world of AI you're going to want to check that out next for more information about Intel Nirvana's AI platform visit Intel nirvana.com remember that with this series we've kicked off our giveaway for tickets to the AI conference to enter just let us know what you think about any of the podcasts in this series or post your favorite quote from any of them on the show notes page on Twitter or via any of our social media channels make sure to mention at twiml AI at Intel Ai and at the AI con so that we know you want to enter the contest full details can be found on the series page and of course all entrance get one of our slick twiml laptop stickers speaking of the series page you can find links to all of the individual show notes Pages by visiting twiml ai.com oilly AI thanks so much for listening and catch you next time oh