Explaining the Predictions of Machine Learning Models with Carlos Guestrin - #7

Key Takeaways

[Music] hello everybody and welcome to another episode of twiml talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam charington the recording you're about to hear is the first of a handful of of interviews I was fortunate enough to be able to record live in New York City from the O'Reilly Ai and strata conferences that I attended last week I'll be sharing these interviews on the podcast over the next several weeks and I think you'll really really enjoy them I'm especially excited to lead off this series with an interview with Carlos gestrin now if that name sounds familiar it's because I've discussed Carlos's work on the show a number of times most recently when I discussed Apple's acquisition of his company T back in August in addition to Carlos's new role at Apple he's also the Amazon professor of machine learning at the University of Washington earlier this year Carlos along with one of his PhD students marker rero and postto Samir Singh published some very very interesting Research into the explainability of machine learning algorithms my conversation with Carlos is focused on This research and the paper that the group recently published called why should I trust you explaining the predictions of any classifier this paper has been on my reading list for a while and I encourage you to take a look at it of course you'll find links to Carlos and the paper in the show notes which you can find at twiml ai.com talk7 a quick note about the background noise and this and the other on-site recordings they're not too bad considering the noisy caverns in which they were recorded but some of you might find the murmurs and bumps a bit annoying if you find yourself in this Camp please accept my apologies and now on to the [Music] show so hey everyone I'm here at uh the strata conference in New York City and I happen to find Carlos gestrin uh who we've talked about on the podcast before uh he's the Amazon professor of machine learning at the University of Washington and uh we've known each other for a bit uh so Carlos say say hi hi thanks for having me here and uh it was great running into you it's great event yeah absolutely absolutely uh in fact I think we probably had a briefing like right in this at this very table a year or two ago yeah yeah we we and I think we met at this event in this very place was that room over there yeah yeah yeah uh so I guess I'll I'll say very briefly to the the audience we're not in the most convenient spot for podcasting so if there's the occasional trolley rolling by just try to block that out because if you want some lunch it's right behind us right but I'm sure you'll you'll won't remember that at all because we're going to have a great conversation here uh first of all congratulations um on the the acquisition of t n data n graph lab by Apple I mean that was amazing yeah we're very excited to work of Apple it's great awesome awesome uh so why don't we why don't we just start with introductions like introduce yourself talk a little bit about about your background I think a lot of people kind of know what you've been up to but sure I'm happy to share so well I'm Carlos Carlos guest been working machine learning for a long time so I was a professor car melon for about 8 years and then at University of Washington since about 2012 and being excited about machine learning uh for a long time and worked on many areas of machine learning um most recently couple areas have been exciting to me are really around dealing with uh Big Data uh and the two sides of that so on one side algorithms for machine learning that scale to very large data sets so how can you scale up to deal with tons and tons and tons of data and the second side is what I think about is The Human Side of machine learning so how can a human understand large data sets how can a human understand what machine learning algorithm is do doing and bringing some kind of human perspective into the mix so I think about those two sides the computer perspective and the human perspective of machine learning in large data sets and uh I imagine there's also a fair amount of overlap and intersect between those two and and of course right so uh the bigger your data in a sense the harder it is to figure out how to make it work but it's also hard to figure out what's going wrong with that so debugging a machine learning algorithm that requires you to run in a cluster with tons of machines is just almost an impossible task and and honestly the way I think about it is that there's no machine Learning Without humans in the loop right you know we're trying to build uh this incredibly intelligent applications they're going to be self-sufficient and but we'll always have humans be part of the process at some point and so making that more human is a very important part of machine learning it's been underst studied in my field um but it's something that I'm very excited to engage in as well yeah yeah so there's there's humans in the loop in lots of places actually and one of the places that uh humans are most certainly in the loop is um you know on the the back end of a machine learning recommendation and your group has done uh a lot of interesting work very recently at least um on explainability uh can you talk a little bit about how you've AR how you arrived at that research area yeah yeah so so uh just in kind of in one sentence we're interested in being able to provide more transparency to machine learning be able to explain why a machine learning model makes a particular prediction or why it behaves a certain way now we we we fell into this topic kind of interestingly in various ways so for example uh in Academia we're working with very folks in application domains and we said oh come use machine learning solve this problem it's going to be awesome we're going to change your life and their response was like sounds great but why should I trust this model what is it doing like and I was like uh it's got great accuracy uh so that's one side that's somehow unsatisfying unsatisfy yeah yeah no it's it's not enough it's really not enough we can talk about why it's not enough and then on the other side uh once we we build a company around machine learning you know uh t datto graphlab as you mentioned uh we started working for a lot of companies that brought machine Lear into production and there was always a step that nobody talked about but it was very fundamental you train the model to do something recommendations or whatever predict turn predict fraud right and you want to deploy it into a service that every time you SW up your credit card it makes a prediction about fraud you don't just make that happen out of the box you want to make sure that model is working well and is doing things for the right reason because if it's not you're going to get fired right so you really want to understand why that thing is behaving the way it is so we by talking to folks and not just that I we've talked about on the podcast before in Europe there's legislation that's coming down a that mandates explainability for machine learning and predictive applications uh yes there's legislation in Europe and not just that even in the US for certain application domains in in the financial sector uh they mandate certain models you're allowed to use versus others because they believe those models to be more interpretable or more believable or something and so for certain tasks in that sector you you have to use a particular kind of model okay um it's and and that just uh you know blocks a lot of the high accuracy models you might want to use so it is a real issue and I think that issue gets uh bubbled up in three areas so one is just kind General user model how can they gain trust that service is doing things for the right reason so if I go to uh say movie content recommendation say Netflix I want to know that I got recommended Lord of the Rings because I also like Star Wars that gives me a sense that thing is doing the right things recommending things that make sense to me H and they can begin to gain a relationship of trust with that artificial intelligence system underneath so that's kind of a more personal consumer thing but if you think about uh from see a a decision that's really important really life change like a doctor making decision about the treatment of a patient right there you want transparency so if you have a system that says the patient is going to have cancer with 90% probability most doctors are going to that system because they might not trust it and also because there's a holistic approach to medicine that we want to have where it's not enough to just make that one prediction but if if the system were to say um you know this patient is likely to have cancer because if you look at their MRI results you see this lump and if you look at this related cases they were diagnosed in the same way and if you look at this latest study uh this all corroborates the evidence then I can gain a more holistic View and I can gain trust in the system so that's the second piece it's kind of gaining more deeper insights as to what's happening in that prediction and the Third Way which is more kind of personal for me is as a data scientist I want to be able to make the models always better and I want to understand when it's working when it's not working so that I can improve it and so those are three areas p uh public perception of machine learning making more informed decisions not just a prediction for machine learning and um improving the models through feedback and so transparency and explanation going to be indispensable to make that happen and unfortunately as a field we haven't invested enough in that topic right right uh but you guys have uh started to invest in this and you was it a month or two ago you published a paper that yeah the paper came out uh just yeah couple months ago um and it's a system called lime that my student maku and postdoc Samir sing wrote Samir is now a professor UC Vine okay um and we wrote this paper uh based on the feedback that we're hearing and the need to do something more in this area and there been some other works in kind of explainability machine learning but what was unique about uh the perspective that you know Marco and Samir brought into the world is uh how they approach the problem so a lot of the work in machine learning has been about finding models that are transparent or explainable like we talked about in the financial sector so these models have to be simple m so that somebody can understand it but the problem with that is that a simple model tend to be inaccurate right and so you're compromising accuracy uh for explainability and that's I believe is a wrong compromise to have in fact when you were making the comment uh drawing the analogy with Netflix earlier I was thinking to myself you know I'd actually kind of rather that Netflix recommends movies that I want to see and not tell me how it got that then recommend movies that I kind of don't really like it says it's this is the reason yeah yeah so so so right um so accurac how they do it but but uh for me the way I'm thinking about is accuracy is number one right you want to have high accuracy for the right reasons but that's the main thing otherwise you're not going to be able to solve the image processing tasks that we're seeing solved really well deep learning today if you don't have the most accurate models if I'm only allowed to use a very simple model like a shallow decision tree you're never going to be able to detect objects in an image right so what's the point what's you know we're not going to build that kind of artificial intelligence system so um what we did was say okay let's take accuracy as a requirement right and so that means that we want to be able to give a data scientist the flexibility to choose any model they want yeah and uh the question is can we provide an approach that can explain the prediction for any model right that was like the the question and I I think it's a really beautiful question um and you know the way that the work came together was was really interesting of course it's only scratching the surface of the possibility but uh what basically um Marcus Samir did was come up with a system that says Okay I want to explain a particular prediction why did you like that movie or why does this patient have cancer and the way we're going to explain it is in a simple way that's good just for this prediction and we're going to do it by highlighting the pieces of the UT that we believe the model was most important for the model to make this decision so for the doctor's example it is this particular area of the MRI this particular studies that are going on this related cases so that's kind of small explanation for the recommendation system Netflix that you unhappy with it might be that the underly system is very complex and very accurate but the explanation we have to give you is kind of very simple but somehow has to be faithful he has to say it behaves like the model for this particular prediction it's not like the model only uses you know Lord of the Rings for everything but for this prediction that's what was what M right so that's how we we went about doing it and uh you know we we did a bunch of user studies that really showed that this uh can be very powerful and it can be used in various ways so it's pretty cool what was the nature of the user studies so one of the really cool user studies that um uh Marco design um Let me let me just step back and say it's really hard to Valu explanations because it's a subjective thing right so how do we even figure out that's doing anything interesting and so Marco and maybe if it if it if it's better I I want to get into kind of how it works and what the what the research actually showed if it's better to do that first we can do that first and then Circle back to the user studies um uh it is you're you're the boss so um so let's uh let's talk about how it works let's start from there so the way it works is to say I want to have a particular there a a particular prediction I want to explain why how why it was made why did the model make that prediction same way and the way we're want to explain it is let's look at the behavior of the system the behavior of this complex model around this prediction so not everywhere but around it so for this particular patient I'm going to try to explain why the model thought it had cancer so let's look at Patients around it some that were predicted to have cancer by the model some they were not predicted to have cancer by the model and fit a simple explanation that explains the difference between those similar patients it doesn't explain the difference between every patient but just patients are kind of like you so patients that had most the same characteristics as you that have cancer had this things going on and patient simar characteristics as you that don't have cancer have these other things going on and that gives me a lot of sides for this particular decision and that's how more or less how it works so as a course approximation of what I'm hearing uh take the example of that you use in your course you know predicting real estate prices based on a bunch of different variables it almost sounds like you know the you might have a a model that's a regressive a regression model that's predicting based on house size right and the the the the lime system is almost a reverse regression that's going the other way like predicting what the inputs might be based on the output is that in a sense so so one way to think about about it is um house prices are very complex depends on the neighborhood you live in the the characteristics of the house everything around it there is a simple explanation why your house costs you $5 million right that's how much your house cost so thanks can you give me a raise too so I can pay for my $5 million house I'll double what I pay you right now um so so um you know why did your house cost 5 million why did house cost certain amount who knows really complex and models can be very complex but your specific house can I explain why the model cost 5 million that's very doable I can look at your house and I look at similar houses that like it and I can fit locally a simple model with only a few variables that were most important that's basically let's say linear and it says around your house the variables that were most important were um square footage and zip code and you know number of bathrooms that's really important for $5 million house for a for a you know $500,000 house it might be a different thing is more important MH um and so that's the kind of thing that the model would say MH um right so and so that's that's kind of how how it works it provides uh the key pieces of the input they were most distinctive for a particular prediction mhm and as I said this is one of way to do explanations which scratch in the surface there's all sorts of other ways you can imagine doing it and I think there's a lot opportunity to do even more but one of the challenges is how do you figure out if this makes any sense whatsoever if these kinds of explanations are good if the algorithm is working at all like there's so many dimensions that this can go wrong how can you even figure out if this is at all reasonable right and uh uh Marco Samir and I spent a lot of time banging our head against the wall to figure out out how could I even test this how can I even measure it how do you explain your explan explain my explanations and uh and Marco had a couple of brilliant ideas that uh really surprised me um and so let me let me give you one of them so he wanted to know uh if explanations are good and intuitive that would mean that somebody who is not a machine learning expert a late person could look at it and make good decisions from it okay so that's a that's what uh that's PR thought so how can we test that hypothesis so he did two two tests validated the hypothesis in a really brilliant way so the first one was if I can interrupt you uh it it sounds like the that the aim of the research was not just um to spit out like an ordered list of features in terms of you know waiting or importance in the the output but more generating uh human readable description am I reading too much into this or is that no no no uh it was a human interpretable descrip interpretable not necessarily readable okay that's one way to explanations can be many things right we we explored the human readable explanations we the visualizations we explored different ways to explain things okay um but uh yeah so so here here's the experiment that he did which was pretty brilliant so he took a data set and uh just a just a little background which is kind of a fun EXP story that I I just told uh there's this famous data set called the 20 news groups data set the what 20 news groups data set 20 news groups okay yeah it's a data set that's been around for about 30 years um in the machine learning community and it's uh from news groups which you might not know what they are but they were something later called forums now called Facebook pages right where they have a topic and people talk about the topic and they post things right and the the data set was famous because um the idea was given the text of the posting can you predict whether this was about Christianity or atheism or hockey or computers whatever the topic was and basically any modern machine learning approach gets 94% accuracy so everybody used this data set in their um in their classes like I use in my class and said oh machine learn is so cool get 94% accuracy when Marco run his explanations on that uh it turns out that the main features being used are things like the email address of the poster okay like so Sam atgmailcom always Post in the I don't know what your interest RM but in the let's say podcast News Group recop podcasting yeah recop podcasting clearly and uh and obviously it's a great predictor right but it doesn't generalize to somebody else right and so it's not a good feature it's a good feature for you but not a good feature for the world and so if you remove that those kinds of features the accuracy went down from 94% to only 57% yeah so the statea said that everybody has used for decades uh machine so well actually wasn't doing so well wow uh and you were able to see that from the explanations so the question that he he want to ask going back to the user study was um as an expert he discovered this with explanation can uh somebody who's not a machine learning expert discovered this and improve the performance of machine Learning System so he took this data set and um and he there was only getting 57% accuracy and then clean the data as much as it scrub scrub the scrub the scrub remove all this bad features and retrain the model and he was able to get about 70% accuracy okay by removing all the bad features like Sam gmail.com and now and then uh coming up with a new model or new model train on that train on the clean data set so that's the go stand the clean data and there was the the dirty data Y and the question was using explanations could mechanical turkers who know nothing about machine learning identify bad features we don't we said look at explanation just cross out things that you think should be relevant for this decision oh interesting okay we didn't say anything else just cross out things that you think they're irrelevant and we thought okay could Crossing it out get performance so close to Marco's gold standard M mhm from nonexperts meaning so you ran the you ran the lime system the explanation system against the dirty data set yeah you came up with these explanations that included things like emails that should be irrelevant and you asked if turkers yes turkers if they could figure that out figure out what parts of the explanation right what in a sense what features you thought should not have been used as part of this decision and how did that go so um after just three rounds of mechanical turkers Crossing things out they were able to get better accuracy than maros gold standard di set wow so they were able to clean the data better than uh Marco did w what is round in this Cas round was um showing the so so so I didn't go into details but uh we showed the the explanations to a number of mechanical turkers y they were able to cross it out and retrain the model okay then we showed the new model to Mechanical different set of mechanical T because they cross some things out and we show again so we just did that three times three iterations with nonexperts and but looking at explanations they were able to find all sorts of problems in data clean it and get better performance than Marco did like sitting down and like trying to clean the data himself wow which is surprising that so that means that non-experts this is just an example sogg just non-experts are able to understand explanations of a complex machine Learning System and provide some feedback to that system that can be used to improve the performance of that system which was really surprising wow that's that's very cool it was very cool and then the second user stud so so then you know Marco is bold and then want to come up even more interesting B uh study uh and uh and uh the second one was also really exciting so here's what he wanted to ask um when when you train a machine learning model you usually train on some data and you evaluate it on some data you hold out it's called the test set sure so that you don't um kind of uh get a biased prediction of how well the model do MH so you can imagine some models might do well on the training set but don't don't do well on the test data set so we want to throw out those models Y and some models will do well on the test data set and um and then you want to keep those models so that's what you typically do if you go back to the 20 news groups data set if we had just looked at the 20 news groups with the email addresses in that in the test set you do well in the test set so you think you're doing well but it wouldn't validate very well validate very well but if you had tested on some other data set they didn't have time at gmail.com then he would have done badly then he would be able to throw it out so so here is the experiment that Marco did which I thought was brilliant um he split the data into uh a training set in that set right and he trained a bunch of models a lot of models on different random subsets of that input data of the training data yep and some models did well on the training data mhm and some models did badly on the training data okay he threw out everything that did badly on the training data because we only want to keep high accuracy models okay so he kept only things that were accurate on the training data right and then he looked at the test data set and some models did well on the test data set and some models did badly on the test data set right and then he said oh the model and up till now I mean this is pretty standard what any data scientist would do and pretty standard run a bunch of models see what sticks against the wall and yeah exactly yeah now he he did the following he took mechanical turkers with no nothing about machine learning Y and show them explanations for the models that did they didn't say anything about hit process and it was all randomized and blind and what everything right right so the explanations for models that were doing well on the test set Y and models that did badly on the test set okay and they both look equally good on the training set yeah and um the I forget by test set you mean the validation actually was hidden in this case it was a hidden test set but you can think about validation it was a hidden test set so it was something that you wouldn't do as a data scientist but uh he held off some additional data and he ran both of them both he held some additional data he ran lots of models that did well in the training set and he picked out something that did badly on the test set and did well on the test set M and then show explanations for for all those models to uh make out C turkers and asked which model do you think is going to be better in the real world based on the explanations of why they're making the predictions yeah and they they were asked to pick between one between two yeah between two and you were comparing them with a coin flip you know right so yeah compar to they did better than a coin flip so a coin flip gets 50% accuracy 87% accuracy wow so totally untrained The unwashed Mechanical Turk masses are basically creating you know Fe doing feature engineering on models in their the first part was feature engineering the second part it was like model selection basically right they were able to look at explanations and figure out this model is stupid yeah even though on the training set they look great yeah but in the real world it's going to be bad wow that's pretty amazing and that was amazing to me and the fact that um we can do that as I said we're only scratching the surface here but the fact we can do that to me says um humans will be in the loop in the long run there is there are insights who have having humans in the loop because even kind of the statistical problems that underly this question like if we discussed for a long time we can talk about why this relevant statistically um humans might be able to pick those out and they'll be able to do better future engineering they'll be able to understand problems they're going in the data even untrained folks and now if you imagine doing this to get more insight for doctors or for um systems in the real world I think you could do really amazing things so it's pretty exciting to me to start you know exploring this further and further that's very cool let me ask you this this is kind of in the weeds question but were the features uh what you might think of as natural features or engineered features yeah so so he that's that's the the really interesting or or a really interesting question so uh the underlying models use the engineer engineered feature so for example he also showed this was good for deep learning models for images which use really you know learn complex features of the data but the way that he explained was from pieces of the input so the assumption that he made was the input is interpretable okay and by selecting pieces of images or part of the text a human can look at that and say oh this makes sense if we looked at like the seventh player of a neural network and say oh this is what like a human be like what is that and what do I do I care you know and so that's why we we bias towards this approach doesn't mean that in the long run we want to invent something better that that looks at the features because it problem might be down in the features might be down in the weeds and that's kind of where the research should go but as a first step we looked at uh pieces of the input and it's totally model independent like call modur Network models yeah yeah we've done this with deep neuron networks we've done this with boosted decision trees we've done this with lots of different kinds of models wow wow so I know we need to get you off to your next session where can folks learn more about this so um if you just search for my name and lime you'll find our paper it was a kdd this year here l l um you can also find a GitHub project that Marco has been putting together with open source some of these ideas um but yeah it's been a a pretty exciting uh um work and you just keep tracking know Marco Samir there'll be a lot more in the pipeline it's a really cool thing that's awesome that's great and uh if folks want to reach out to you you're on Twitter or what's the best way to get in touch with you I'm on Twitter gastan it's my last name the um and so reach out give me some feedback we're also as you know on a corsera teaching machine learning and that's another place that uh I interact with folks Y and it's a great course I highly recommend it very case study focused I really enjoyed it okay thanks great all right thanks so much yep I'll do a handshake here on the audio thanks all right everyone that's it for Today's show thank you so much for listening and for your continued support a quick story if you follow me on Twitter you know that I recently called out an iTunes review that I'm actually particularly proud of in this review a user that originally rated the podcast a two out of five based on their disappointment with the switch to the interview format came back and revised that review to a four noting that the interviews were getting better and that the format was really really starting to grow on him now don't get me wrong please I really really really appreciate those of you that left five star reviews on iTunes and I hope the rest of you go run and do that right now but it also felt great to see that in spite of his initial misgivings the shows just kept getting better and the user eventually came around that kind of feedback is great to read thanks to everyone who's stuck with the show through the transition and I hope you're continuing to learn a ton please join Jo the conversation by commenting on the show notes at the twiml ai.com website or by reaching out to me on Twitter where you can find me at@ Sam charington or @ twiml AI all right everyone thanks again for listening and catch you next time [Music]