TWed Talk: Brenda Thomson on "Explanation in Human-AI Systems"

[Applause] please kill that I that was it thank you all right thank you everybody for for showing up today welcome to our I think it's the 5th 2019 season and the enth Wednesday night talk 20x idea talks in our pirates meetings of our spring 2019 term just so a couple of logistical reminders we are streaming and recording live if you're welcome to ask questions Brenda really wants this to be an open discussion tonight and there's lots of people here this is wonderful it's a big topic but so you're welcome to welcome to ask questions just keep in mind that we are being recorded and without further ado Brenda Thompson thank you very much so my presentation tonight is about ok so my presentation tonight is about Martha's explanation or explainable AI February 2019 report it's from there there's two task teams that have been working under DARPA Zechs AI and this one in particular is from their second team which deals with the the meaning of what is explanation so so we're coming at it from that point so basically ok there we go so real quickly the agenda I'm just basically going to kind of cover what the report is how its structured and how we can actually use it what what the X explainable AI program actually is and what are some of the models that they've come up with as well as some of the tools that they've now have available Bruner can I just um in the links that I sent out or in the email I sent out links to the report as well as some other contextual information um are provided so do you want to look at this in detail you can just pull up that email say and at the end for these slides those links are also in the references as well so alright so let's take a look at the report one thing the report is 204 pages so it's kind of a very large long report so it can be a little intimidating but I just kind of want to break it down for everybody because it's not as bad as it sounds so the executive summary just you know as every report will have but what's important to notice is that the actual report itself is essentially a hundred and four pages long and the actual reading part of it is only about 55 pages so it's it's worth your time and effort because and the reason that even though the report builds all of these pages the real thing is is that what they do in each subsection is break down what are the relevant papers reference to the bibliography that are relevant the most important ones for that particular subsection so you'll notice there are 70 pages of bibliography so it really takes the time to go and download this report because the bibliography is so wonderful that you can just go and click on those the links there and get you know the papers themselves for your own research so and the big deal too is there in the appendix that the report ends with metrics that they've come up with to measure the the actual value of explainable AI so how does that actually work so again it's just broken down into some sections the ones that are kind of in that are really standout as on the report itself is section 4 and section 8 section 4 of course covers the key papers and so once you know if you have a particular area that you need to research you can actually just go directly to the section buying the paper buying the reference and then section 8 is brand new in this report there's one prior report for this task team and so section 8 is all brand new and it's actually where they brought in human participants to help evaluate whether or not an explanation was good so the program itself what is explainable AI that's and what is DARPA doing with it and why did they bring it in so artificial intelligence really kind of started back in like 70s 50s okay sorry in the u.s. okay and so there the way dark is looking at is of there's three generations of artificial intelligence and the reason that this is happening now that we've come back around to dart back so thinking this is important and relevant and that it needs to be occurring now has everything to do with those generations and the B onset of deep networks and machine learning so because as those have developed we have wound up losing the explained ability that we had back in the 70s and earlier so so anyway so this is what they you know this is started back in 2016 the explainable AI and so they this was what they were working with at the time and what they've done is taking taken basically from the learn function right we're gonna move through and create an explainable bottle as well as an explanation interface so that the end user actually can interpret what the system is doing we can certainly cover why you know but I think most of us in the room know why an end user would need to know what it what the system is doing and it's a bit a big part of this program of course is where does the explanation come from and what does explanation mean and so as I was saying there's two teams involved here and the team that wrote this particular report is this team here so they talk about you know they look across all of the literature's from computer science to philosophy psychology and try to look at how is explanation defined so that we can actually create some kind of computational theory or how then we how and why we will explain so the model that ultimately they came up with for explanation follows that from you know basically this colors don't show it very well but the first one here is actually the process right the user receives an explanation which revises their mental model and then enables some kind of better performance right because the user then learns to trust or distrust the system hopefully you know once we've got a good explanation the trust model falls into an appropriate range and then we can appropriately use the system as well right so from there all right so the models that they themselves are that they did find to work with let me back at what I meant so here what they're aiming for right is they have this is the accuracy and this is explained ability and this is where we are right now it's the orange and they're hoping you know this is the goal this is where they're assuming we are at this point right we move things forward a little bit with explain ability but we have still have quite a ways to go and so for the deep explanation here basically they're just learning using the deep deep learning models basically so that four ways to make things explainable I'll get a little bit further into this later but the next was the interpretive old models and so this is a goal right is to build a more interpretable model and projects in this category focus on implementing the deep learning right by providing an explanation and making it more making the model more interpretable finally there's the model of induction model induction there we go so this is model agnostic basically at a reference cutting here I quoted me without telling you who he is but gumming is in charge of the explainable AI program for DARPA so he's he's the go-to guy so um let's see anyway so I'm not quite clear on model induction to be honest so let me move forward so they've identified two areas in particular for explainable AI they can influence of those are you're either in data analytics area or in a an autonomous systems so I'm gonna say how can we move on to so ultimately in the end these are they came up with this model for measuring these metrics for measuring exclamation effectiveness and so once you're through all the way through the report the program aims to essentially have these these gray areas here are actually metrics that will help you decide how effective your explained ability actually is they they are quantitative as well as qualitative so they're they're expecting a little bit of human interaction on this it is not strictly within the AI system itself yeah yes so and as I pointed out earlier so this is where you know this is the performance versus explain ability metric that they're working towards you know they're calling it tomorrow in this graphic but it really is you know what what we're aiming for because there's about two and a half years left on this program it's what they're saying right now and so to do that this is their end goals right create a suite of new or modified machine learning techniques to produce explainable models that when combined with effective explanation techniques enables end-users to approach and appropriately trust and effectively manage the emerging generation right so hopefully so this is what that end at the end of the report this is what they came up with as the metrics and what an actual report looks like so for each individual structure so they there's about I want to say there was 70 pages this so different AI systems that they have evaluated both using AI as well as this was the big part of section 8 of the report that's new for this year is you know they used humans to actually evaluate the systems and they measured how much they trusted the system how wonderful they thought the explanation was or not and so this manipulation of explanation business is it's kind of a tough metric to measure so it's very it's quite subjective and out of all of it the the thing that's most clearly defined is what is trust and to do that it's really depends on whether you're talking to experts or are you talking to in the case of medical if you're talking to an end user as opposed to the doctor right so a doctor expects a different set of of explanations as opposed to the patient so different levels of trust for different from the different end-user so do things develop on coding for each of these or they have not yet that is coming that's part of the next two and a half years yeah they have yeah what they've done right now is just create the metrics so these are going to be probably not very readable that's shame okay they're nicely reader ball online okay good so the first model here was is what Johnson & Johnson develops so these are simply explain it but anyway so you've got your these are expert systems right so you've got your I can't even read it on my own screen so you move from the experts right where you have you know the learner already has existing knowledge you cover the gaps and you have just declarative knowledge as well as versus procedural knowledge and essentially moving towards you know is the explainer satisfied this is the big metric that we're looking for right you have to find that satisfaction level and it's easier to satisfy they the expert on an expert systems you know you're so your weight your expert and your expert it's not okay welcome back great explainer interactive with the learner correct thank you the model of the explanation of a procedure that's why next comes the cruel model and this one was can't see it on corner there but it's about events right what's that yeah the event occurs and then you move to the event is noticed then interpreted and you have your initial explanation right and so from the initial explanation you move to the evaluation of that explanation and then on and that explanation originally then sets an expectation of what to expect from the system and it feeds that it through the crowd the process stops over here and it feeds back to the evaluation but before you leave this so this is a generic explanation model generic exponential this is part of the big yeah yeah can't even see it that's what's great oh there it is okay I can't find you actually okay there we go okay so yes so these are the generic models right well I think that might be about the type of model system I was thinking about it in the terms of the explanation provided for there's natural language or what is a part of it it's that sense so we should be careful this is why I asked the question before it's just a generic model if this is specific data explanation it's just is this a generic model of explanation you know are we early these are these are ones yes they were created with for explanation in AI okay cuz it doesn't say much about like the outputs of a idlers is the the model that's generated so the predictions classifications models I don't I'm not seeing any bad so just to be clear what these are is just pictures of previous so all of these so I don't know we drilled it out 2014 yeah yeah don't get so eager just people psychologists study information well and some systems including was that I built sometimes have a separate system for giving the explanation from actually what the reason was doing so for example in Hue complicated procedures like do a separate kind of explanation system that some people might think are more intuitive okay and now you know there's there's a whole body of literature the explanation reasoning the reasoning for explanation versus the reasoning for performance okay Wow all right well I have a lot there's a whole bunch more papers to read in this in this report so okay so this is a data format model of sense-making and this one I didn't even follow through at all so unfortunately so anybody familiar with that one okay perfect and this was all the explanation but you know there's you know they give these models into pay in the report but you know there and there is quite a bit of explanation to them in them I obviously should have read them more firmly so even even if you're not able to kind of go into depth it might be helpful first okay so she didn't so yeah so information search identify the key evidence build an argument and deliver a report Oh Brenda use the example beautiful position versus a patron on the system right well si again that's that human-computer interaction so the doctor isn't going to be using the same interface that you would give to an end user right to a patient right so it really is that human in the user interface yes yes no I was looking at the fidelity model and I was confused as to its but this in this in the bigger picture what why it was important because it's breathing is using evidence identifying every searching for it identifying evidence to build a hypothesis right no well enough seeker was it mentioned like whether it validated widely a possibility based on certain evidence so that explained why that hypothesis was that not the intention of that model again this is this is the psychology mouse of explanation not the AI yes so since making first I couldn't even in the psychological sense right related to what explanation it was offering me making is I'm sitting here and I see Brenda pointed a slide why did the point why she asked me and I say well Brenda specific vote so the explanation is to make sense of the event as opposed to dating in the psychological sees it so so in other words you can be so it's the difference in the literature between explanation of sensation right is sense making is generally considered to be triggered by an event isn't well-defined but it's anything that happens something in the real world and I want to know why it happened right so there's not necessarily a system somewhere and so sense-making is more focusing on that part explanation is more of I've got this black box or program or whatever what's happening inside why did it yes I'm learning in a situation like that what's the difference between explanation of the black box system and rationalization in the sense that humans were rationalize decisions at the moment [Music] that's what I was hoping for that there'd be experts in the room to actually like have this discussion to contextualise it yes so let's move towards the tools that have actually come out as a result of the work that they're they're doing right now there's three I've got two to show I didn't find the rise one that Gunung in particular talked about and that's through Boston University if you want to go take a peek they have this sequence to sequence visual is a sequence to sequence visualization anyway so a visual debugging tool right so and basically it's a visual representation sorry that's so small of different stages of the sequence to sequence translation process so just as an example let's see if this comes up here by google lingo nope wrong one 2007 don't look nope there it is okay hi you don't have anything okay so this is actually the sequence to sequence oh really we open close town anywhere nope alright guys they all go back I can't even see the back hold on back no back button is it working hold on Mike oh no there's no back I really do know how to operate a computer we're pushing our luck right that's because we can alright so this is basically the sequence to sequence visual debugging tool right and what did what it's doing is translating from German to English right so what this one is showing you so this is a tool that it's they're using and it is sorry and it's available on github and you can download this and use it but it what it's doing is it's showing you the how how things are flowing through the network is deep learning well and I have a better visualization of this it's specific to language translation so so there is right so now it's showing you here in this zone like where so you choose which decision point you want to look at and how it got there and you lose her here and it shows you where it has moved the decision points from or made the decision where it made those decisions off of what data points they made that decision from and then it in literally yeah so it'll break down from its text where you know highlight that for you show you all the steps that it moves through so it's a pretty nice tool if you're dealing with text and text translation the other tool let's see it I should know that that was the back over here yeah up at the top there you go go and again this is provided in the link to this on the slide but you and then it is there you go it's out of MIT an idea so the I don't think it's it's due to be published or it has been I don't think really okay April of a team okay reprint area okay let me try the other one see if I can get it to Bob over there the next one that most of you may have heard of this one it's from Google the what if tool right so code free probing of machine learning I think this was something you mentioned shweta that your classmates we're using for being able to move through you know using tensor flow you choose your data points and and then you can actually do the counterfactuals as well to determine what it has you know what what happens if I choose this as opposed to that and causality yeah so so it's literally a what-if as in you know what if what if this works we're here as opposed to there what if you know I said yes instead of no so you just use your you know it's literally four facets it's you cannot see that so what's a pitch yeah let's see there you go help no it's four facets like thinking about like a faceted it's fewer yeah yeah that's um where you already pick your pick box basically yes even you know just data points it's not even you can use it across many different types of data sets you can so if you've got you know like think of your network graphs right your edges right you can move the edges around if you wanted to and generate different and find out what if what if the you know what if the value is this as far as I know it's it's the inputs to what they do is they say okay the credit run it again permuting those and then once it got one one said you would have gotten alone if then they you didn't have enough income that was that really what went on but it was it was actually from UIUC was working on a alternate world using semantic technologies where they had you know if this person died here was born here and died here what if this person didn't die here how would it affect events happening in the world that was and using reasoning for facing back to those explanations seems like a numerical but if you doesn't commute the input you put it you ran the system no there was clearly relationship so so this allows you to do yes you can actually get down inside and ya still flip through everything they still haven't come up with you know just be solution for right I'll show you [Music] yes okay let's take one more okay yeah in the face what does that do so just this one is from one of the it's the loan learning how a I makes decisions and what this is the Raisa this is rice yeah okay I did find it so what they're doing here is they're stopping they're basically saying hold on when you have this decision point what were you focused on what was the data points you were looking at and so in this particular case what it's showing you is that you know so this is a white sheep this is a brat they're calling it brown sheep right and here you see you know so when it made the decision is it a she you know is this a sheep it was focused on this one right on this bottle and then based on training data it had decided that the brown sheep was a cow and so what they're saying is that when you have this when you're using this kind of thing as an explanation what this kind of tells you at this point is that you haven't done enough training on brown sheep so we can't you know so it doesn't identify brown sheep but it's also you know using heat maps to explain what it was focused on is actually as you can see is pretty effective so you know just getting these simple explanations out you know this is you know bird 100% that sees that you know 39 percent chance that there's person here and then it sees that this one is a bird but that one is a person right so again you know it's making those it's showing you where its decision points are using a heat map which is pretty effective I think you could use that in a lot of different ways but again rises out of Boston University so not and it didn't have the time actually to go and find out you know it's more publicly available elsewhere but it is pretty effective and I think was developed before yes I was I used input sampling for explanation of black box models that's what it Bryce stands for it have that somewhere okay so I showed what if model we went through from Google and then okay so David getting I think this is a really effective quote when looking across this report as a whole if you like if you put a long pause here you know wait it's it's the other programs that are really dealing with things outside of explained ability so if if you're you know this this report is not the end all be all right you need to look across other programs it's probably a word of caution more than anything Sam I'll show you real quickly a saying move ahead a little bit and so that's a great question I suppose you could I don't you know but I don't yeah yeah I was gonna say I'm not sure how long that systems gonna stay up if you do that well this raises 18 months I mean I don't think they're all 18 months really using those metrics correct yeah and tonight back to yeah yeah yes say that was the point of this right yeah I said that this this particular report is like an 18-month marker for certainly for the task seem to it obviously for the whole program but yeah this was the 18 months marker for that so oh okay no okay yeah I'm hunting to try to get to 12 over DS oh yeah there was something in PC Mag or an adult says the 12 it mentions issues a few right it mentions a few but that doesn't mention as well yeah no you're not you're not crazy on that one it is actually yeah they they mentioned that there's 12 awardees but no so at this point I was going to show up so basically some of the references but I wanted to real quickly referenced here too so the finally in the what they do is the awardees or so the what they have at the end of the report actually is what each you know the universities here or you know the individuals right there's you know what what each one is working on so it literally is like how they're doing explainable models they explanation interface and the problem or challenge that they're answering right and there is so in the report there is pages and pages and pages explaining so if you need some models to work with if you need you know these sort of things they're in there right really great stuff yeah so the the first probably 70 pages of the report is honestly the literature on what is explanation and trying to honestly come up with a really concise computational explanation of what explains this so anyway so yeah so it's pages and pages and pages that's just yeah so these are what is the focus at these universities or yeah right what but this is the program itself right so define an explainable model find an explanation interface develop it based on autonomy and autonomic autonomous system versus you know data analytic system and what are so this is you know there that's there on the report I highly recommend this this part of it in particular but in the report they don't mention who worries you know they don't either I say it actually means if it might have something where I close it all right I might actually have one graphic that I think I saw that on so you can look across that so an autonomous system is going to be a lot of the Department of Defense stuff or the self-driving cars you know so where those kind of systems actually have like a limited or you know just on-site limited resources right to use a data analytic system could really be connected quite well right so an autonomous system is going to be self-contained within whatever piece of hardware it's in may not have direct access to so the data analytics I think we hopefully most of us know so anything that would be able to be connected to the Internet to you know we analyze those systems analytic power so with that I think we should probably have some more discussion on what else is in the report or what explain explanation means and that's kind of the purpose of the first 70 pages or yeah it's part shows the images and this is the important part of the image and so on is is have they explained why it's satisfactory as an explanation mechanism because I personally feel that you know how forego uses that for showing you know this is why I played a stone here but many people have said that's not enough of an explanation for why you Clara stood on a particular point versus like highlighting this area which is supposed to be an important part so I'm curious to say if they've given any explanation as to why that's a satisfactory explanation for why certain decision was made why that's a cow or a sheep or something I don't find it yeah that's a good question no and I say that's the problem that still it still really remains it has not been there is not one definition that they've settled on four exclamation you know there's not the there's 70 pages of what an explanation is and it is really something that they hope to come to like some concise definition is what they're hoping to come to for what it means in terms of computational theory and they just haven't done that yet yes and they you know they they start to report off by mentioning in the executive summary that you know if you you know really want to know what an explanation is it encompasses all of Western philosophy and Western writings you know so I think pick your spot and that's kind of where it is um I don't mailed you a link to the program announcement 2016 and that shows that provides a timeline oh there's a timeline in there 2017 important and actually I'm there 2017 report I finally if you go on on Xai awardees the guys from humanities one I just top you have to actually look specifically for for this report yeah although a number of years slides were a very okay super Gantt chart Gantt chart especially right now yeah today we're where we're at that they're calling it the third generation and of AI that's explainable yeah actually this is the third generation so first generation expert systems yep second rule and then knowledge base tutors was the second generation and then as what there and then there's defining an explained ability winter that happened basically from 1995 until 2013 that so there was nothing really published on explainable yeah then that's what these publications well I was gonna face it small graphic this might be yours first thing so yeah so in the explain how they explained the explained ability winter is it's the time when there has been the rise of deep learning and and machine learning right so deep networks machine learning generative networks right so all of that has has occurred so it so in building that we kind of that we as no researchers right kind of abandoned the explained ability and we're okay with you know working through right the and that's that's their explanation and then now we're we're this point where it's kind of an inflection point and we have to really get back to looking and being able to explain what our deep learning is doing now that's I'm thinking absolutely yeah semantics yeah I think there might be an expert on maybe some decision logics is room yeah that's gonna be really important yes yes okay so the question was is explainable ai acting more on the how question or the why question and it seems to be that they're really looking towards the end user whether that end user is an expert or you know just a you know one of us so or you know who may not be an expert right so so it's looking across what that means it's not really focused on how but except maybe in an expert system where the expert tells them what it needs to do there and then it's kind of more focused on the why for the for for less I don't know how yeah at the end user yes so it does it does look across all of those questions so this is this timeline is going all the way up to mid 2021 so we're in sort of the Inc a mais results phase right now yeah they're saying years one been starting state phase two right that was my we are prep for email to unless I'm looking at older one but this isn't I hear a box I call I'm okay well this was kind of the one of the things that I kind of looked tried to look across when I was just reven reading through the report with some signs of scope creep and you know just as project management right and you know they really did great job of defining themselves on the first go-round and they haven't you know at least in published reports they haven't had a lot of scope creep they've stayed focused on what they originally intended to do the updated one goes through Nahum 2021 they're saying two and a half years is that that would be correct yeah if these Tim who knows from February of 2019 so here's a question how do they how do they future-proof this program given the amount of change criteria for evaluation that was good you know I'm not sure how easy it is to get those numbers so I know that they so they got several breaks in a site well there's some of these phases in the phases right but I guess when I'm I'm just kind of wondering this how does or they will they be able to respond back strategies is it's kind of relying on the players yeah well you know I think if the world demands those metrics to be satisfied then people who rise to the top are going to start making systems that address those metrics you know we could debate whether those metrics are actually all that good or easy to evaluate but it's a start and my guess is this is near ending his career at DARPA he's got machine common sense and maybe some ways it's the next person who cares well and the people we don't yeah I mean I was looking at the B eyes who so it's seven when we thank Brenda thank you very much thank you make yourselves for showing up this is one of our rare occasions when we have a full room and it's not lightning talks so speaking of quits but those of you are telling this world we're gonna start soliciting for the end of the term it's lasted in term or last Wednesday term is going to be our lightning talks and we're looking forward to that so Wednesday last day of the turn [Music]