Trusting Machine Learning Models with LIME

Key Takeaways

[Music] latest skeptic fetchers interviews with experts on topics related to data science all through the eye of scientific Marco Tulio Rivero is a PhD student at the University of Washington he has a bachelor's and master's degree in computer science and is currently researching methods for making machine learning more interpretable and trustworthy to this end he created lime or locally interprete Belisle agnostic explanations it is this project that I invited him here to discuss today Marco welcome to data skeptic well thank you I'm glad to be here so I thought maybe it would be useful to start out our discussion from some first principles perhaps you could tell me what is a black box and what are some of the negative perceptions of the idea of a black box so black box is basically as I understand it's a system that does something for you gives you predictions or perform some behavior but you don't know how it works the internals of it are completely hidden from you in machine learning specifically we call a lot of models black box even though we could inspect them when inspecting them would take so much time that it basically is infeasible so if you have a really deep neural network making predictions in theory you could go and look at each individual neuron and activations and etc but it's so large that it's functionally a black box for humans so I guess if I'm interacting with a black box I can maybe it's even too complex for me to understand I've opened up a couple small appliances at home and looked at the circuit boards and says wow this is beyond what I was taught but I can at least turn on that device and say well I can see it's working I can test it and I can maybe even check its accuracy so is accuracy enough should I just trust that the model has had good performance on some training set well that's a great question first of all of course everyone who does machine learning knows that just looking at error on a training set is a terrible idea because you can quickly overfit and learn patterns on the training set that will not happen even when a test that earned the real world so you can be really good at one specific data set but usually we want machine learning methods to work on other data sets not the ones we use to train but even if you do the right thing in quotes if you do cross-validation or have a held out test set or something like that there's a lot that still could go wrong there's a lot of problems that can creep in when you overestimate your accuracy and this is very common there's been some research that shows that people consistently overestimate their accuracy in fact anyone who has ever tried to do machine learning algorithm and put it into production knows that this is the case you typically overestimate your accuracy how does lime help us deal with situations where we might have overestimated our the accuracy of our models so the thing is humans have a lot of good intuitions about things if I'm training a model one of the examples we have in the paper we're training a model to distinguish between atheism or Christianity emails I as a human have some knowledge of what atheism and Christianity are so I know what the model should be using and what it shouldn't be used so if I see an example where the model is predicting that emails are eighth is new Christianity using words that are completely irrelevant to it is my Christianity for example stop words or something in the email headers I know that something is wrong similarly humans have a lot of intuitions about a lot of things if I'm training a vision system I know where things are in the image and I know what the model should be picking up on so lime opens the black box as it were it explains individual predictions and from there humans can use their intuition to have insights figure out what is wrong and potentially fix it so would you say lime is unalloted or is it a family of algorithms what's it's sort of formal definition I think the way we said it in the paper is we set up as a framework for generating explanations we did some particular explanations in particular linear models but lime is basically an equation that tries to balance interpretability with faithfulness so I'll say it's a framework you can derive many different kinds of explanations with the line framework we have been working on many different ones in the paper we have a couple of them for images and texts very cool I want to get into some of those specifically but let's maybe take a moment and chat a bit more about that news group example so if I read it correctly you've pulled some posts from an atheist news group and from a Christian news group and then we're going to try and build a classifier that looks at a post it's never seen and decides which group it came from is that right that's right this is an old data set I didn't miss her I didn't put it up but this is a famous data set that's a benchmarking a lot of machine learning throughout the years yeah because of course these groups have very significant difference on one thing but they'll frequently talk about it in the same way so an atheist while they might not believe in God they would use the word God in a discussion of atheism so I can see the interesting challenges of this problem and I've seen smegal yeah I've seen people report really high accurate predictions on it what are some of the problems that you noticed in looking at other solutions where people try to predict where a new post should go usually when people talk about this data set they present it as if they're learning the topics I'm trying to deceive I'm learning how to distinguish between how people talk about Christianity talk about atheism and there's other news groups as well that's how they think that's what they want their models to do this particular data set is not very big and it has lots of extra information like it has the email headers for example the news groups are collected in I think 1997 or something but some news groups have posters some particular users that are that post there a lot so for example in the atheist news group there was an user who's named Keef his name is present in 12% of all atheism emails but that becomes very informative keep if you seek if you know it's atheism and by some coincidence there's no person in Keef posting on the Christianity news group so right there you have a perfect predictor if you see the word Keef it's always atheist in this particular data set likewise there's a lot of email addresses or institutions where it just happens that in this data set you're very predictive but of course you wouldn't expect them to help at all in the real world yeah that's a great example the the Keith case it just so happens that there were no Christian posters of named Keith so how would I learn that in the traditional way if I was going to go inspect a model what would it take for me to uncover this the sort of Keith's anomaly well if you use a model that is interpreted well right away for example if use linear model you may be able to look at the individual weights however the vocabulary is pretty big and a lot of times even if you have patterns like this there are other patterns that are maybe valid so for example the most classifiers will learn that the word Christianity is a good indicator for Christianity so that makes sense so it's easy to miss this sort of pattern even if you have an Interpol model if you have a neural network or a gradient boosted decision trees or something like that like a random forest it's really hard to know that the mod is picking it up what you could do is do some exploratory data analysis to find out the most abhorrent examples so maybe you would find out about Keith but there are subtler examples as well both in this data set and others that you would definitely miss and I would argue that it's much easier to see these patterns if you see the model using it instead of trying to find it with no model if you have a model that's using it extracting it from the model is much easier I've noticed a trend towards more complex approaches especially in text things like word encodings have gotten very popular which make it much more difficult to inspect a model how can we use lime to better get us some insight into what the classifier is actually relying on so that's right more and more people are using features that are themselves completely opaque to humans and I think one of the key ideas that we had in line is separating out what the model is using from what a human understands so we have what we call an interpreter representation that is what we provide explanations in so even if the model is using let's say word embeddings or something the explanations are still in terms of words now of course there's going to be some loss there we can't perfectly explain what a model is doing unless you have the model itself from the features themselves but a lot of times you can have a pretty good approximation especially if it's one individual example so even if the classifier is using word embeddings I'll explain it still using regular words how do you manage to do that let's talk a little bit about the specific use cases then say I have out of my big corpus I have ten particular posts that I'm interested in what are the steps that lime goes through to give me some insight and hopefully build my trust in the classifier it's really simple the idea is that I'm going to learn another model on top of your model so we have an example let's call it X that we want to explain what we do is perturb X in different ways generating a bunch of new data fake data as it were perturb data and then we get predictions for that data so for example if it's a text example like an email in the news group we remove a bunch of different words every time and then we'll make predictions with those predictions we create a data set so we have a data set of perturb examples and predictions and we learn a simple model on top of the data set in such a way that we care more about perturbed examples that are close to the original one so if I completely remove every word I don't care if my explanation captures that case very much because it's so unlike the original example that it's too far but if I only remove one word I want the explanation to get that right meaning if the in the explanation if I say that that word is important it really is important to the underlying model so just summarizing the main idea is we perturbed an example in different ways get predictions and learn a simple model and the reason why we would be able to do that is because even really complicated models are not that complicated in a fixed neighborhood like around a point they may be a simple make sense could you tell me a little bit more about the advantages you see in the perturbation case so I could see two approaches maybe if I want to train this this model on top the model I could grab a hundred posts at random and retrain a classifier or I could train them by this sort of distance metric you guys are using what advantage does that local choice give me one advantage right away is that I don't need access to the training data or anything like that I only need an instance and a classifier and I can generate an explanation for example I explained predictions from Google's inception area network I didn't even look at their training data or anything another thing is that when your data has certain regularities for example in the twenty newsgroups case if you sample from that data you still get that kind of regularity even if you don't want it so you may not notice certain patterns if you just use the original data so you could do potentially a mix of the two but I think perturbing is really useful to get data that you are otherwise wouldn't to avoid biases in the data set for example I love the idea of helping users develop trust in a model which is really what drew me of the paper initially so I was thinking about this news groups case it makes total sense that even if the prediction is correct if it says we determined this is an atheist post because it was by Keith I'm going to be very skeptical of that do you maybe have any examples in the other direction of things that would give me confidence that the model has correctly classified between you know the atheists in the Christian news group if you notice that it's speaking up I have noticed like when even bad classifiers to learn some useful patterns for example if you have God with lowercase G that tends to happen more in eighth ISM news groups and in Christian news groups and model speak up on that yeah that one's interesting I could also see like Christians might talk about scripture or mention you know Paul on the road to Damascus while atheists would talk about Dan Dennett or Christopher Hitchens and eight fans yeah yeah those are things that convinced me it knows it's it's picked up the intuition I guess but that's part of the idea we as humans have intuition we know what a good explanation looks like in a lot of cases and if we see that the model is acting in sensible ways we tend to trust it more yeah absolutely so I've deployed a number of models in my career and sometimes there's a struggle when you're replacing a function that a person used to do and the model is going to be better at and the person is a little bit resistant but I've also had the opposite problem where I deployed a model and I had supporters who were too confident in the results and they were maybe the you know assigning it like strong AI properties that they found a tricky example that the classifier indeed got correctly but didn't necessarily get it correct for the most convincing reasons how might lime have helped me in that situation I don't know if you've heard about the tank example it's a very common example that machine learning professors always used in their lectures let's go into it this story goes I don't know how true it is but that someone was developing some contract for the Defense Department or something where they were trying to identify camouflage tanks and the model had perfect accuracy even on validation that it was perfectly able to identify when tanks were camouflage or when it was just trees as the story goes when they try to apply it in the reward it completely failed and the reason was the way they gathered the data was such that either tanks were only present during the day or at night I don't remember which but basically if photos were taking at one time of the day they're always tanks there and if you're taking another time there are no tanks now I'm sure that if the story is true I'm sure that the developers who first developed the system invented all kinds of explanations for it like they were like oh yeah it's probably detecting the texture in the tank or something like that it's easy to assign reasons like that when you want your model to work if you had something like lime you just see that it's just looking at the background and we try to replicate this experiment in our paper we don't do tanks we do to differentiate between dogs and wolves which is supposedly similar and we ask people why do you think the neural network is doing this and lo and behold they come up with all sorts of reasons oh it's because it's something in the wolf's eyes or the Huskies knows when in fact we train the neural network in this particular case you just attack snow in the background yeah makes sense so in that respect are we really talking about a system for double checking that something hasn't overfit or is that just one special case of what lime can help with I think that's one special case overfit is a problem data leaking is another problem maybe your feature engineering is not capturing what you wanted to capture and lime could help you perceive that so overfitting is one of the problems that can appear but even if you're thinking of systems at work I use the doctor example a lot if you tell a doctor look you have to do surgery probability 90% no doctor is going to do anything based on that but if you say look you have to do surgery look at this symptom look at this others seem to look at this similar case then it's much more likely that the human decision-maker would do something about it so it's not only for when things aren't working but it's also for when things are working so I know a common problem in machine learning is when you have a class imbalance let's maybe pick a classic example like fraud detection you know most of your transactions are good law-abiding people and just a few minority bad apples are in there that you're trying to detect so if I were to randomly sample a bunch of transactions and show someone the the outcomes and also use lime to show them the you know some supporting details of why we think the model arrived its conclusion it may be that we sort of deceive ourselves into thinking the models good because it's covering these average cases but not covering the minority cases is there wait you can pick the more interesting examples that might highlight sort of the rough edges of the model that's a very good point and that is a that's something that I'm working on we did have a initial solution in our paper we had a system that chooses which examples are going to be shown to the user instead of just doing it randomly and that's already a start but for the situation you're mentioning right now where we have big imbalances or there's another case where you have different costs for making mistakes so if you're a police officer and you're going to arrest someone based on a machine learning system you'd better be right so you really care about not getting that wrong or spam detection is another case I haven't seen work and I haven't done the work myself on selecting instances in that particular kind of case where you have high imbalance or different costs but it's certainly a very important problem that needs to be addressed so I really like some of the visuals you have in the paper around the news group example where you're like we keep going back to if it says hey the reason we picked this is because the most prevalent word feature is Keith that gives me some skepticism but if it picked it for some other reason I might grow in in trust of the model so it seems like this is very nice for text how can this work in other settings like in the images you were describing so for images what we did in the paper was we highlight the area where the systems using the most to make a prediction and we make everything else gray and that's already a it's pretty good humans images are easy to interpret because we just look at them and it makes sense there are other representations that were working on that would give us even more power for example if a system is looking at the tone of the image to see like if it's grayscale or something this representation that we chose would not work as explanations it would not be able to explain this kind of behavior so it's always a tricky thing trying to figure out what the representation is you want it to be really interpret abut at the same time you want it to be powerful enough to show things even in the text case for train newsgroups for example if we just show a sequence of bars like a bar plot it's hard to pick up on things such as sequence hmm things like that so we are developing visualizations and things like that that are more powerful but still interpret boom make sense yeah I found the graying out approach you guys develop to be pretty effective in fact you have a great example in the paper of what I would consider an image that's meant to be sort of deliberately misleading could could you describe that image and how he'll I'm ended up breaking it down yeah so we have a person oh I mean we have a dog playing guitar and it's a person's body so it's a really funny YouTube video of this dog on human clothes and there's a human playing the guitar and it looks like the dog is playing it of course the new or not to get that it's very confused by that and it makes a mistake actually it's a it's an acoustic guitar and it predicts that it's an electric guitar but it also has high probability for acoustic guitar in Labrador the particular dog is a Labrador and at first I was a bit puzzled as to why I would think it's an electric guitar but when I when you see the explanation there's a part of the acoustic guitar that looks exactly like the electric guitar so the classroom I could have got it right but I'm not that surprised that it got it wrong it's not like an egregious mistake I understand the acoustic guitar explanation is also pretty good it shows the part that the kusa guitar and the Labrador example just picks up on the dog's face so obviously this is a confusing example but even though the model made a mistake I still trust it because I see that it's doing sensible things so let's say I was a user of a particularly complicated model maybe I'm a doctor looking at some very complex thing like depression and I say it's complex in the sense that you know if I get a cut on my hand you stitch it up or you don't it's pretty straightforward what happens I think depression has numerous causes and sometimes it's chemical versus psychological so it's almost like you need to personalize diagnosis if I were a doctor and I had access to a model and I was going to use lime to help me with that diagnosis what would that user experience be like in a perfect world in a perfect world we would have the doctor see the prediction like this patient has depression for example and then the system would give it all of the evidence in terms of like look this patient has depression because look at this particular exam that you did it would even go as far as saying look I would have even more certainty if you did this other exam it would even mention previous data like I think this patient has depression because of this exam look at all of this other patients that had a similar result on this exam it would be more causal even and that's something we're far from from achieving sure I think everyone in general is yeah so uh when I was I was very excited by you work and I was thinking well what what would be the contrary position and the the first potential for criticism that crossed my mind was to say that every local model you build might essentially just over fit to that local fidelity area and then it's more like you're have this amalgamation of all these local models which is a lot of degrees of freedom could you talk about how you balance between picking those local fidelity models that are faithful to the global classifier yet particular to the local instances it seems like there's sort of a trade-off there there is a trade-off and it's a very clear one if you want something that's interpretive globally you are restricted to using your simple model there will always be some approximation error even when doing this kind of explanation the nice thing I think about Lyme is that you can choose what the trade-off is going to be so if you really care about being really precise then you got to put in the time to look at a really long explanation also we can measure how good or how bad the approximation is even in the local region so you can imagine if you're using lime in the real world sometimes it's saying something like look I have this explanation but I know that it is not a hundred percent this explanation you can't really trust even this explanation so you're going to have to do something else so kind of like letting you know when I'm failing gracefully and I think that's helpful but yeah it is the case that we are explaining the global model by a bunch of little interpretive models the nice thing about that though is that intuitively that makes sense some examples only need a little bit of evidence like let's say I'm trying to predict atheism in Christianity again if you see a post that has a hundred references to Christianity a lot of Bible verses in etc it's very easy so you can find a linear model approximates it well if you see a post that's really hard to tell like if there is some debate going one then it does make sense that the approximation can be a little less faithful if the model really needs to be more complex interesting yes so so perhaps the fact that a post comes from you know Reverend Smith the fact that they have that title is all one needs but if they're debating you know the ontological argument for God then it's a little bit more of a gray area or something like that exactly interested in usually in machine learning we do not want models that are always super complicated because those models tend to be really fickle like if the explanation needs every single word like if you change a little thing that's nice should be completely different that means that the model is too jumpy and we usually don't want that in machine learning anyway so I think it's a fair assumption yeah there are cases when it break but it is a fair assumption I think yeah that's an excellent point so Trust which kind of is is what I think this really comes back to the value that I hope to derive from exploring live in my own work Trust is a concept that's a bit fuzzy for machine learning at least to me you can't calculate it as easily as you can accuracy or f1 score or at least I don't know how to is it measurable at all that's a good question and I don't know the answer to that I think there's a sense when you are assigning a binary measure to it every time you put them water into production if you put the mod inspiration you trust it to be better than what you had there before so in that sense you did trust a model whether you should ever not is a different question oh absolutely agree I actually meant it more from the perspective of a user so I could show user two models and ask them you know after interacting with both which one do you trust well we do an experiment we do not actually we don't measure trusts by itself like we I don't know if the users would be willing to pay for something which would be a way of measuring if they really trusted in the paper what we do is we have saneras web where we know you can't trust a model we try to see if the users can perceive that it's much more complicated though if you want people to actually trust it if you have a model that's trustworthy and you want people to trust it I don't know how I would measure that except by having them use it or have to pay for it somehow it's easy to measure when you can't trust it sure which is what we did in the paper yeah we set up models that we know are untrustworthy and so if people could figure it out well from my point of view Trust is the complement of untrustworthy I suppose could we talk a little bit more about those results and what you were measuring with your users for sure yeah we have three experiments with users two of them are with non-expert users there are people in amazon mechanical trick so we took that classifier that we trained on twenty newsgroups the one we've been talking about that's an untrustworthy classifier we collected a new data set a religion data set that does not have the problems that priya newsgroup has we collected web pages we want to use the model trained on twenties groups to predict crusceni and atheism in web pages and we tell users that we can measure the accuracy of the original classifier and it's really low it's like fifty seven percent what we do then is we train a clean version of the classifier so we clean up training is good to remove the headers remove names like kief and etc and we measure the accuracy of that one on the hidden set and it's higher it's like 69 percent within show users predictions from both classifiers side by side saying look these are the predictions and these are the reasons and most times the prediction is the same they're both predicting the same thing but for different reasons so the user sees a few explanations and chooses which one they think is going to generalize better and this is trying to simulate the scenario that we have very often when you want to put a model into production you have a bunch of competitors they may have different accuracies and they may have similar actresses but you'll know which one to trust and this is an additional thing that you could do look at the explanations in our experiments even non expert users were able to pick the right one 89% of the time which is pretty impressive yeah if you just looked at accuracy you'd pick the wrong one every time because using this kind of untrustworthy features increases your accuracy on the validation set this is the first experimented choosing between classifiers on the second experiment we had the users actually interact with the explanations so when they saw an explanation they could click on words that they thought shouldn't be used so if the user seasoned a prediction for atheism and then the explanation you have Keefe as an explanation these are can click on that and say look the model should not be using this and we do this several times and the interesting thing is even for users who are not experts they do not know about the hidden data set they only have knowledge about Christian and atheist I don't even know how much knowledge they have of that clicking on words clicking a few words with I mean the average time I think it took was 10 minutes they could significantly improve a model the model on average started with 57% accuracy on the hidden set and in the end it was like 72% or something and what's perhaps even more interesting is when I did my round of cleaning with no lime I was just using regular expressions and stuff I got lower accuracy than the Mechanical Turk instead using lime so I guess that says something either about me or about lime I hope about wine yeah I think that's a really fascinating use case I've done some experimenting with active learning where I try and pick good examples to have a user label to help me build up a training set but we're talking about something a little deeper here where you show a a model made and you're soliciting feedback about the innards if you will of the model is that the right way to look at it that is it is and the way that we did is very rudimentary the only thing we're doing is removing features going forward I'm very interested in developing more expressive forms of feedback still not required user should be an expert in machine learning would you consider that sort of a form of regularization that you're showing the user some of the details and asking them to penalize some of the features you can think of it as regularization but it's it's not trying to avoid necessarily overfit you're trying to guide the model to do things that you think are sensible and this stems from the fact that most of the timing machine learning we cannot really train or measure what we really care about we measure accuracy maybe but what we care about in the end is user satisfaction for example and we can't really write that down but humans have a lot of intuitions and they can know that something is wrong where they see it so if you could steer a model towards what you want that would be very helpful so I could see where lime can apply in pretty much any machine learning deployment I was curious if there are any specific areas or industries that you're most interested in not in particular I'm interested in well obviously in the paper we did more stuff related to NLP envision those I think are pretty interesting because it's situations where humans have intuition I don't know how useful it would be for some crazy genome data I don't know how much intuition have about that I'm not a biologist but in some scenarios where humans do not have intuition then I don't know how helpful it would be sure so mainly for cases where humans have a lot of intuitions and texts and images are two examples where that is really the case yeah two of my favorites that you mentioned are the medical one where you know a doctor doing a diagnosis and the policing one as well yeah for sure it would be very concerning to me if police blindly acted on some models prediction but if there's an auxilary system giving them some support for why it made a choice I think that can have a lot of benefit to society so that's a great example of what I was mentioned when I was talking about the mismatch for example we do not want our models to be racist if you using them in policing but how do you define racism it's hard to define even for him is how do you write that down for a model however you know it when you see it you know that if a model is telling you to arrest a suspect because he's african-american you know that that's wrong mm-hmm that's a blatant case but humans are able to detect even more subtle cases for me lime is a tool that's going to help us get some trust that the model is basing its choices on intuitively correct features in the absence of it we can have cases where operators people who that consume the outputs of models maybe have too much trust in their model or too little trust and that we can help close that gap what do you think are maybe some worst-case scenarios if we didn't have lime and people had either too much or too little trust well if you have too much trust you get problems like thinking your model is going to work really well and then it doesn't and the consequences could be catastrophic if we're talking about medicine there is a great example by a guy called rich Cara Juana from Microsoft Research where he was developing a system to detect if a patient is going to I think died from pneumonia and the idea was you'd have the system you'd predict if the patient is high-risk you admit them to the hospital low risk you send them home and give him chicken soup which is the best thing for low risk patients apparently and he had a system that was really good based on a neural network in the end he didn't deploy it because a friend of his was using in an interpreter model which was rules and it learned a very intriguing rule it said that when people have asthma their risk from dying from pneumonia lowers and humans know that that doesn't make any sense asthma increases your risk when they dug into it it was because patients who are admitted with asthma got treated more aggressively so in the way that they gather the data those patients ended up having lower risk because there was an intervention but if they put their system into production that intervention would not happen they'll be sending people home and people will die so it's a really tricky situation to detect and if you don't detect the consequence could be catastrophic in the end they'd apply the model that they could inspect really well so that they could avoid this kind of thing that's a excellent example I think so tell me a little bit more about the open-source components of the project I'm trying to always keep the code open-source I'm also trying to get this project going where we would the idea that you'd add more and more tools to debug machine learning or interact with machine learning right now we have what we have there is line but I am working on a few more components for understanding machine learning and it's really easy to use it's in Python you can get explanations for most models with a line of code like maybe two lines of code and it works it's very general it works blue low scikit-learn cool so people working on the Python data science stack should be very familiar is the best place to learn more to give it a try to visit the github page that's right the the github page and I'm sure you have a link in the when you release the shores yeah yep and the shownotes people can find that link I'll probably tweet it out as well so people can go take a look at that maybe even send a poor request if they get really involved yeah sounds great I have had some poor requests and I hope to get more awesome and so your papers up on the archive I'll link to that as well is it going to be presented anywhere else in the near future yes next week I'm going to k2d in San Francisco and I'm going to be presenting the paper there very cool Marco what's next for you guys in your in your research and in the development of Lyme we are working on multiple fronts in particular one that we're working on is going beyond classification to things like structure prediction where models are even more a peak because the output itself is hard to understand sometimes because it's big on the output space is really big so that's one future direction going to stretch your prediction we are trying to develop the vision part a little little more to go beyond just hiding parts of the image also one area that I think is really important is figuring out which kinds of explanations are more useful than others and depending on the target audience so it's some eight side work that goes into that we are developing different kinds of explanations instead of just linear models it's really hard but we would like to measure how each kind of explanation impacts different tasks so for example if you want to detect overfitting is it better to have a linear model or to have a different kind of explanation mm-hmm so that's another line of work that we're going in the future I hope to tackle the feedback problem as well so I noticed that something is wrong I want the model to do something about it but I don't have to be a machine learning expert and I want to develop that more in the future as well very cool I am what's the best place for people to follow your work I guess my personal webpage I can send you a link as well I'm always going to be posting something I do have a blog that I don't post that frequently in and I hope that I'll be able to put everything on the github project as well also I hope that by the end of my PhD I have a set of tools that people can use to interact with their models all open-source I'll link to your site as well in the show notes so people can follow up there well Marco thank you so much for coming on the show to share the story of Lima and your work I think this is really interesting and I'm glad it's part of the machine learning conversation well thank you so much for inviting me it's great more on this episode visit stay calm if you enjoyed the show please give us a review on iTunes or stitcher [Music]