Experiments with Interpretable Artificial Intelligence | DataHour by Shiladitya Banerjee

Key Takeaways

foreign years of experience of the industry applying machine learning models in the digital advertisement domain and currently in the fintech domain the virtual stage is all yours yeah uh am I Audible uh I think the participants can hear me uh can you confirm if I'm Audible I am unable to share my screen [Music] will you try now please yeah uh is my screen visible yeah it's visible uh do I need to start the video or however hello hello uh I'm unable to hear your voice hello hey can you hear me yeah I can hear you now yeah your audible completely fine okay uh should I start my video this is fine whatever suits you okay uh then I can just start with the session I guess yeah uh so hello everyone uh please uh at least just let me know if I am not audible anytime Midway yeah okay uh so hello everyone uh I am currently I am working as a data scientist for phone pay and in today's session I plan to give you a broad overview of what interpretable AI is so if you look around resources in the internet you'll find different terms such as interpretable ml explainable AI but they all basically constitute a set of methods and uh the plan for this session is to kind of walk you through the basics of each of these methods right so first let's try and understand that why do we require interpretable AI at all so um the way the entire machine learning pipeline works is that we capture certain data and uh to basically predict or to understand patterns in the data we build some kind of model now many a times due to business needs we need to understand the underlying process or at least make other stakeholders understand what the model is exactly doing so the three main reasons I guess um why interpretable AI is so important is one is to understand the underlying process because if we get hold of uh what is exactly happening underneath uh that's a big benefit to any business second is to build trust with the model so uh if say a model is performing very well but we need to make sure that the model is performing well for the right reasons right um there is no basically garbage data going in because of which uh the model just happens to perform well for the time being and Sim on similar lines uh it basically helps us understand the reliability and stability of the model so if we understand which features are important in the prediction uh we can take a guess of how long this model will be stable uh with respect to time uh so that if any feature which may change in the future uh we can change our model appropriately then right so how do we go about doing this uh if you see the chart on the left which is basically explains the trade-off between the predictive power of a model and the explainability uh you see the cluster at the right bottom and those basically constitute methods which are explainable on their own simple to understand but you always have a trade-off with the predictive bar or the accuracy right so one way is to build interpretable models so if you are pretty sure that your um the the process that you're trying to model is not that complex then the easiest way out is to choose one of these models and go ahead and uh the weights of the models or say the splits made by the decision tree that would kind of give us insight as to what the module has learned right but with uh the progress in the the field of deep learning and also uh boosted trees right I mean these days we use xgboost uh just out of the box uh to to solve any kind of problem um we try and use um extremely complex non-linear methods which may not be easily interpretable right from the output right so in order to handle these uh let's have a look at what all methods are currently available uh to explain the Black Box models that we built so before uh we go into the methods uh it's very important to ask the right question right and asking the right question or the question basically will govern what kind of method we choose to interpret our model so one of the methods I'll just skip a slide and go ahead is one of the question rather that we ask is how does model prediction vary with respect to the features of Interest right so any model that is built and if you just get me the model I would want to know how this model varies with each of the features that have been put into the model that would definitely help us gain some understanding of how the model performs right and in order to answer these questions this is basically a global question we are not looking at an individual prediction and trying to dig deep further we are trying to understand the model as a whole right so for these for this particular question uh we resort to Global methods and the model agnostic term is we would like to use methods which do not depend on the type of the model that we build so uh we would be looking at more generic kind of explainable methods which would work for any model that we have built and this is a layer on top that we can apply so that we understand what is going on inside the Black Box model right so coming back to the previous slide that becomes a global model agnostic methods another set of methods can be classified as local model agnostic methods so this basically gives us the significance of instant features for prediction so it means that for each prediction we try and understand which feature is playing an important role in the prediction of only that particular instance and local model agnostic methods if you basically average them out across all instances we do get a global view also right so these are the two broad kind of methods that we will be looking at today and I will come to example based and attribution based later these are just another sub category of the methods that we will look at but the main two methods are the global and the local model agnostic methods right so coming to the global model agnostic methods as I mentioned it basically answers on a whole how the model prediction varies with respect to the features of Interest right so the first method that we will look at uh is a set of plots called a partial dependence plots now as the name says we basically have uh the feature on the x-axis and the predicted value on the y-axis right uh so the partial dependence plot um will basically take for each feature value what is the average predicted value given out given as an output by the model right so let's let's have a look at how to kind of chart these partial dependence plots for um our models that we're building right uh my screen hopefully is visible just let me know if you can't see the Jupiter notebook okay so for the partial dependence plot uh the data set that we'll be looking at so this is a publicly available data set it's the bike rentals data set right and the goal of this data set is to predict how many bikes will be rented depending on features that basically qualify the weather and the day um uh of the of the date basically right uh and this is available from the UCI machine learning repository so this is publicly available so let's let's start running this notebook and see how to chart the partial dependence plots right so here we read the data frame so you can see that for each row uh you basically have different features like holiday weekday temperature humidity wind speed and the variable that we're trying to predict is the count right so you see her 985 bikes so basically for that month for the particular date 985 bikes were rented and just to give a context of what this is like you would possibly rent um a vehicle if you know the temperature is very hot and going outside is kind of a trouble right so that's what the model should basically be seeing so this is so the 731 instances in the data frame right and we build a simple kind of just do the normal train test uh split and let's build a model and see the output right so the model building is not the main focus of this session so I'll just uh run through these cells quickly and uh we are using a package called Alibi uh I will come to this later um but this is like an excellent Library which has kind of collected and still developing uh all these interpretable methods Under One Roof right so we are going to use the partial dependence plot from this package called alibi partial dependence plots are also available as part of scikit-learn uh so we'll see that later in an example that I'll show towards the end right so here we just make a copy and do some let me just yeah so we you can see we have a random Forest regressor that we have built so we are predicting the number of bikes so we are using a regressor so there's some amount of maybe overfitting happening but we are not too bothered about it uh currently the train score is at 0.98 um and the test score at 0.9 uh so let's go ahead and plot the partial dependence um plots so if you see this is the API that we are calling from The Alibi package and what it takes is input is a predictor so this can you can basically have any kind of model and you just need to write a predictor function which will give the output from the model so this could even be a set of rules potentially that you have kind of handcrafted uh and and just written a logic to predict some kind of output so as long as the function gives a certain output certain probability or in this case a certain number uh it's fine right and we give the feature names the target names and the categorical variables so yeah so the first thing in partial dependence plot is you would see this parameter called kind and here we basically have given average as the kind so I will come to this later as to why this is average um but let's plot the explanations first right so this is the entire data that we receive after running the partial dependence uh explainer and if we look into the Json right so we have this uh key called Data where you see a list of arrays here and the feature names here we have basically we are basically trying to draw the partial dependence plots for four features which is temperature humidity wind speed and the season right so if we look at what these values are right uh the like we are querying the data key from the explainer output so you would see that there are there are basically four elements in this list right which I have highlighted just now so the four arrays that are present in the list they correspond to each of the features um that we have given as input right for which we want the partial dependence plot now if we see the size of the array you would see that each of the variables the array corresponding to each of the variables have hundred elements in it and if you go slightly on the top there is something called a grid resolution equal to 100. right so let's take a break here uh for from the Jupiter notebook and see what exactly is going on and how do we actually plot the partial dependence plots right so I'll just pull out an Excel sheet so I hope the excite sheet is visible if not please let me know right so what exactly is the partial dependence plot so say we have a data set right you can see the data set at the top here it contains of three features okay and uh the model basically outputs this particular Target right so here we have three rows with three different values for each of the features and three different predictions given out by the model now in order to now what what what do we mean by the partial dependence plot is that we want to know the average Model Behavior with respect to one of the features right so say we select feature one and we want to draw the partial dependence plot for feature one on the x-axis if you see this these cells here on the x axis you would basically have the values of feature one and on the y-axis you would have the average model output now what do we mean by average model output is basically while finding out the partial dependence plot what it does internally is for each of the instances that are present in your data set it will randomly or basically try out all possible values of the feature that you are trying to find the partial dependence plot for right so if I basically mask the feature F1 for now you see that our data set has three different data points comprising of F2 and F3 which is basically B1 C1 B2 C2 and B3 C3 right so now for finding out the partial dependence plot for feature F1 for A1 we will try out all possible combinations of the rest of the data set so for A1 we would also append B1 C1 for A1 we would also create b2c2 and with A1 we would also append B3 C3 in this way up another permuted data set is created internally right so for every instance we basically try out all the possible values of the feature F1 so we have three rows for A1 another three rows for A2 and another three rows for A3 and then we basically take out the average we Group by this feature and we take out the average of say that uh the model output and we get one point of the graph so if I take the average of Y 1 1 y 1 to Y one three I will get y 1 me similarly if I take this average I would get y to mean and if I get if I take this average I get y 3 Min this basically inverts if I had to put in plain English words it says that when the feature F1 takes the value A1 on an average the model outputs y1 mean similarly when the feature F1 takes the value A2 on an average it outputs y to me so in this way this is a dummy example and hence we just have three distinct values but say we take a continuous feature it would have maybe thousands of different values and for each of the Thousand values we would find the average model prediction in this way right now there are a couple of problems that we can see immediately is that if a data set size is very large and if we have a continuous feature which has thousand unique values the permuted data set that it's creating or the intermediate data set that it's creating would explode right because it would have 1000 into the size of the data set values right hence in order to get around it the first thing that it does internally is obviously it doesn't try out all the values it basically finds deciles or say percentiles and tries out certain values for the continuous feature okay and the grid size parameter that we basically saw here basically controls how many points of each feature do we really want to sample right so once uh we sample here 100 points we will get the average model prediction at each of those hundred points and plot them so let's just go ahead and plot them first right yeah so here along the x axis this is let me just zoom in a bit yeah so the first plot is for temperature and you can see that the predicted number of bikes increases till the temperature is say around 25 degree celsius and it then kind of starts decreasing and for humidity initially the model prediction or the average model prediction is kind of flat but at around say 55 68 starts tapering off and wind speed uh like if you see the scale it doesn't have much it it has a very slow Decline and it kind of starts tapering off at 25 so this these plots kind of give us a sense of on an average how the model prediction varies with each of the features similarly this is a categorical variable so it kind of for each value of the categorical variable it gives us the average prediction of the number of bikes right so coming back to the Excel so one problem which we saw is that the intermediate data set could explode but that can easily be solved by just sampling uh not all the possible values but just 100 of the possible values in the data set right but there's another main problem is that when it starts substituting all possible values or at least the entire range of a feature for every possible value of F2 and F3 there is a possibility that it may start creating unrealistic instances right so for example let's take a cricketing example right so where we'll come to this example later where we'll try and analyze uh cricket data set but uh say we have we are trying to predict the probability of win and we have number of balls bold right uh runs scored till now say right runs scored and say something like wickets right these are the main three features that will be present and say runs code it's a T20 match say 140 now balls bold say are basically 10 uh 12 overs have gone so say 72 and 5 wickets have fallen right now balls bold will have values such as 4 10 right it will have uh 120 when the Innings ends so it will have a entire range of values from 0 to 120. now with PDP such a simple kind of plot what it will start doing is that for this combination of runs code and wickets it will also create another permuted data point having run scored as 140 wickets code AS 5 and balls bold as four right now this basically is an impossible data point to get correct but PDP doesn't really care it reads each and every feature as independent and that is one main problem with PDP so the main assumption is that it reads the features as independent and hence starts permitting all possible combinations so let's see how this handicap is basically handled um but these are some of the insights uh that we can draw from the PDP plots for now and then let's come to another problem that the PDP plot has is that it may happen that in one half of the feature the feature behaves a certain way with the prediction so say like let's take this entire block uh say that say for first half of uh feature one or say this value of A1 the target is basically negative and for B3 and C3 the target becomes positive right and let's say that this keeps happening for every value of feature one which is for A1 A2 and A3 so the mean the mean of all the targets would be hovering around zero right but the actual reality is not that there are some data points which have negative correlation some data points which have positive correlation uh and if we take the average we obviously basically miss out on those patterns so that can be handled by basically superimposing another set of plots called the individual conditional expectation right so let's not get much into the names but if we just think logically it will basically start plotting it will not Aggregate and basically give us a plot for each of these lines or each of these data points that we have collected right so why to average out and give it will give us a sense of how each of the data point varies with uh the feature value and for and how the model output varies right so just to have a look of how the uh individual condition expectation plots look like right so you have this orange line which is basically the PDP plot and this is the average of all these lines that you are seeing here right so this is for temperature humidity wind speed and this is first season right now there's a small slide adjustment that we can make to these ice plots IC plots is that if you see first season uh because all the clicks a certain prediction can start from a larger value the small variations are basically lost in the visualization so what is usually done like even if you see for temperature because each of the points would start from a different prediction right so say the topmost line is starting from 3500 and the bottom most is starting from thousand right but what we care about is how much does this vary along the feature along the x-axis so it is better to subtract this value from the entire curve and then view the curve so it will be basically a zero centered curve but it will show us the variations that are happening across different data points correct so I have already run these plots so just in the interest of time I'll just show them so if you see here once we zero Center it um the variations are much more clear right so if you look at temperature um though like we see the temperature or the number of bikes increasing uh till say 25 degree tapering off for some of the data points it doesn't taper off right for some of the values uh similarly for here there are values for which so on an average it looks like the uh the highest is in fall and the lowest is in Winter um but there are values where all the predicted uh the predicted output for each of the three seasons is constant correct so the PDP Plus uh ICA plots together they at least give us a good initial good sense of how the model is behaving at an overall level at a high level right so just to summarize the PDP plot is a very simple plot it has the feature value on the x-axis and the average model prediction the ice plots are nothing but the other disaggregated plots so we don't take the average we will plot each of the y1 ones y1 to y13 and here like basically a curve would be formed from y one one y two one y three one right and we see the variation of the model prediction with each of the feature value for each of the instances correct so these two plots together give us a good starting point of how to view uh or how to basically get an Insight of how the model is behaving but one major uh Lacuna of both these plots is the correlation that has not yet been handled right so if two features are correlated these basically still substitute the feature not taking into consideration of uh not taking into consideration its relation with other features so now that is solved by let me just get back to the slide so we have looked at partial dependence plots uh then basically to solve the problem of averaging out um and to just see like how uh the interactions um affects each feature uh uh variation with the model output the IC plots kind of solves it to some extent but as we saw here the shortcomings of PDP basically lies in its per or its augmentation of the data set with every possible combination of a feature for each and every instance right so that is a PDP estimation on the left now one obvious way the next most logical step would be that don't permute or don't uh fill in all the permutations of the data point for all the instances just uh fill in or permute the points which are similar to the point under consideration right so say we go back to Excel and if we have B1 C1 and B and A1 B1 C1 we will basically try and permute only the instances in the neighborhood of that point and get an average uh across them correct but this does give a slightly better uh output or a better curve and better representation because it's basically averaging now on the conditional distribution but it still doesn't solve uh the main problem of correlation uh because if there are two features which are correlated then we would still get the model output uh to be the same like if if the if the correlated feature exists um it won't cancel out the coral the the contribution by the correlated feature right so in order to solve this there's something called as accumulated local effects plot now what this does is instead of trying to find out the average model prediction behavior for each feature value it basically tries to approximate the difference in prediction across small intervals of the feature okay so if you see on the x axis let's let's assume that we have this feature one on the x-axis right so early so what it will do is basically it will bin the feature into some n buckets and now for each of these sections it will take all the points right so say it will take all these points line one interval and it will augment the value of feature 1 with the starting point and the end point of this interval okay and what it will try and do is try to find out the difference in the average prediction when starting point was imputed and when the end point was imputed so what will happen if there is a correlated feature which also affects the model prediction the model prediction won't change if even if you augment it slightly with different values and hence you will get basically a value very close to zero as opposed to a PD or a modified PDP plot or the M plot which we saw in this right hand side which would give you a good model output value right so here it basically works on differences uh between the prediction if the end of the interval value was given and as opposed to the start of the interval value and what it does it basically accumulates these differences it starts taking the cumulative difference across the entire range and that plot gives us a sense of how the model output varies with the feature value okay so this is uh just wanted to introduce these three curves because um these are used uh like in in many places and one should I think just be aware of what are the like you now of the PDP plots and why ale plots are used right so just going back to the notebook uh and continuing so here PDP plots now just to end the global methods uh with the pros and cons so the PDP plots we can also have uh multiple feature pdb plots and obviously we cannot have more than two because otherwise we will not be able to visualize but um if you see something like wind speed and temperature right so this is actually a very interesting PDP plot because it shows that when the temperature is between -5 and uh say three or four wind speed doesn't play much of a part uh in the prediction right whereas if you see the green region up here above the yellow region that shows that once the wind speeds go above a 20 or a 25 the temperature value irrespective of the temperature value if the wind speed is too rough people don't go out and rent bikes okay so there are these interesting insights which can be drawn from uh from a combination of features or the PDP plots which had run a combination of features and these are three variants like the The Alibi package or the API of partial dependence plots in The Alibi package uh they can handle all possibilities of um combination of features so here we are seeing two continuous features being plotted here one feature is continuous and the other feature is oh sorry these two are two categorical features and here one feature is continuous whereas the other is categorical right so PDP plots can handle categorical feature with ease because as we saw it is nothing but it's just averaging the model output right as opposed to Ale plots um so with ale plots what we saw is that it relies on taking this creating these intervals taking the difference and then accumulating now the main problem is that when it accumulates there is a certain order that needs to be maintained we can't be accumulating in any order right I mean so for a continuous feature uh the order is obvious but for categorical features it becomes an issue as to what is like what what difference should we be taking right because this this uh the ale or the local effect is the difference between when the end point of the interval and the starting point of the interval is considered so early for categorical values uh is kind of uh tricky uh not very well defined and that is one big Lacuna of uh ale values right so here we see the early values plotted for the same uh features uh and you would see these small small variations of small small jumps and these are basically the intervals that have been created and the local effects being added up right so yeah so just coming to the pros and cons you can see the Le plots they are easy to visualize and understand and they can definitely handle correlations they don't struggle with dependencies but ala of categorical values is not well defined and whereas for PDP and Ice plots together uh they have like a very fast implementation or relatively fast implementation in scikit-learn and uh it's very easily uh for categic can be plotted for categorical variables but as we saw it struggles with dependencies so that kind of briefly ends our overview of these Global methods and just to remind you it's basically trying to answer this question is the how does the model output or the prediction vary with respect to features of Interest now coming to the next set of methods which is basically the local methods right and this is answering the question that what does how much does each feature contribute to every model prediction or every instance of model prediction right so here if you would have already read about it there are two main methods and mainly now dominated by one but the two main methods are lime and shap right so just to give a brief view of what uh the line method is so I'll cover lime and shap together and then kind of go to the notebook so lime basically stands for local interpretable model agnostic explanations so lime actually has a big advantage that we will see later but let's first try and understand how it roughly works right so now we are trying to find local interpretations or local explanations as opposed to the PDP plots and the ale plots where you are trying to just kind of have a broad overview of what the model is doing so say this Red Cross right that you see on the graph is the point of Interest or the instance for which you want a local explanation so this is just one point in your data set so what lime does is that it permutes it like takes small Delta of this observation and you could take as many as you want so say you take n permutations or um uh you take any permutations of this observation then you kind of calculate the distance or calculate how far these permutations are from the original point so you can what you see here is basically a contour plot so the blue region belongs to one class and the light orange region belongs to another class right so now this Red Cross it just lies inside the light orange region so now when we permute we'll get some points which are in the blue region some points which are in the red region some points which are far away because even by permuting one feature uh we may basically land very far from the original point so first thing is to make a lot of permutations of the point of Interest then kind of come up with a similarity score between those permutations and the original point of Interest so these are going to be your weights for the model that you build right now this as you can see the Contour here is completely non-linear right so it say for example this is the output of a very complex model we will try and fit a surrogate simple model simpler model it could be linear it could be any other simple model that we want to fit and try and use the weights of the simple model to identify on or to explain only this local instance right so this I mean there's an option to select the select only top M features based on the complex model output so we create n permutations we score them using the model that we Black Box model that we've already built right then we fit a simple model to the permitted data which which has weights depending on how far it is from the original instance so the distance is basically kind of a similarity score if we are going wrong so our new data set becomes only these n permutations remember that we are just building one model for each point correct so it is only after I say that I want the local explanation for a particular instance line will go and generate another say thousand points it will take that data set and then try to fit a model only for the thousand points and that model we can perform badly on points which are far away from the point of interest but we want to perform very well or we want to have accurate predictions for points which are very close to the point of Interest right so it is weighed by that similarity score now for example if we fit a linear model then the weights themselves basically will be the explanation or the local explanation for that model right so that is the rough kind of workflow that uh lime has now the main takeaway is that we actually don't require any of the training data that was used while training for while building the model so that is one advantage uh that I want to highlight uh that lime has even over shap which we are going to basically see so lime doesn't require any data point from the training data to find local explanations right so that's a huge Advantage like there could be cases in Industry where you don't want to share the training data you just have an API where you can hit the model and get a prediction then lime can be a very very good choice for model explanation as opposed to maybe a better method uh which we look at Sharp right uh so this is kind of the rough uh overview of line coming to the uh slide uh like faults in Lime is um this is like cryptic diagram but if you can see that if the actual model output function uh kind of takes a drastic turn after a certain feature value then depending on the permutations or the observations that we generated we could either we could have any of the three lines uh that fit this local instance right so there are cases where we may go very wrong uh with line uh for these instances which lie in um uh Sudden Change of uh the trend in the function right so I'll not go more I guess into depth of line because uh we still have to cover Sharp but as I mentioned the main takeaway is that it doesn't require the training data so coming to shop right and um I thought that just going through this math would be slightly beneficial because uh in many places so this has been taken from an excellent article that I have also pasted the reference at the end but let's let's try and just understand that in the basics like why shop is maybe a much better option than line correct so sharp comes out of Concepts from Game Theory and uh and the the game theory concept basically shapely values which is the expected marginal contribution right so let's take this simple example that we see here okay and uh what we see here is basically uh the way this uh entire problem has been framed to say that you are entering a competition and there are two people right uh C1 and C2 now if C1 enters the competition or if one enters player one enters the competition uh by himself or herself they earn 7500 rupees or dollars right and C2 or player 2 if they enter they earn five thousand dollars okay and when both of them entered together they earn ten thousand dollars which could be say the first price and obviously if they don't if none of them enter Then they won't win anything okay now the question is that when they do enter the competition and they win say 10 000 rupees or dollars uh how would you basically bifurcate or divide this amount uh between the two players now if if you look at this problem uh I mean many people would say that okay you know 7500 they would have earned individually 5 000 would be 5000 the other player would have earned so let's divide the 10 000 in the same ratio right but uh so let's let's just do that math uh quickly so if I have seven thousand five hundred and five thousand so this is summing up right to be twelve thousand five hundred so this is sixty percent and this is forty percent right all right as you can see here now if I had to divide the ten thousand when both of them were then they would have got six thousand and four thousand rupees right but if we calculate the shapely value and once I think we calculate becomes more intuitive it's not exactly six thousand and four thousand that each one should receive it you will see that the answer basically is that player one should receive 6250 and player 2 should receive 3750. now what is basically shapely values it basically sees that it only sees when you add to a bit add to a current combination or a coalition it's called how much extra value are you bringing to the table right so there are only two possibilities here there are because it's just a two player game you can come by yourself right so the player one could come by themselves and uh earn 7500 correct or the or player one could join with player two and basically earn 10 000. now one when player one comes with player two he's basically adding an additional value of 5000 because C2 would have anyways one five thousand by themselves whereas when they come on their own basically they get an incremental value of 7500 so the total incremental value across all possible coalitions becomes twelve thousand five hundred and if we take the average uh we get this number called six thousand two fifty correct now if we do a similar calculation for two two could come by themselves one five thousand or two could come after one where where they would earn 10 000 but the incremental value is only 2500 right and if we divide this by two we get 3750. so this is one kind of allocation that is possible and there are basically a good theoretical reasons why this allocation is fair uh I'll just State the exams later but let's look at another example where we have three coalitions okay a very similarly just running through the example if you see here uh C1 C2 and C3 if C1 come by themselves they earn 5000 C2 earns 5000 C3 doesn't earn anything and each of the coalitions the kind of a lookup table has been provided to us so c one two three if they all three come together they earn 10 000 right now in order to calculate how much should C1 receive all the possible ways that C1 can enter the combination are basically calculated and the incremental value that it has brought to the table is calculated right so one can come after 2 and 3 have already entered so it brings additional value of 5000 it could come only after two when it then it basically gets an additional value of 2500 similarly after 3 7 500 by themselves five thousand now we have these four values but what is of interest is that if we now have to calculate the shapely value or the fair allocation value the this 5000 is multiplied by 1 by 3 2500 is multiplied by one by six seven thousand five hundreds multiplied one by six and five thousand by two and one by three now the question is why are there different weights for each of the possible correlation entries right hence this table big table that you see on the right uh is all the possible ways or the sequences in which one could have entered uh any kind of Coalition correct so if you see if one enters after 0 post that there could be two more ways right of uh the team being formed so two could have entered after that or three or three could have entered first and two similarly if one enters after two there is only just one way of basically forming the team that is three enters post one enters and then if already two and three have entered still there is just one way right and again if three and two have already entered there is another way so this is basically weighed by giving appropriate weightage as per the number of combinations or the permit or the uh number of combinations in which one could have entered that allocation right so if you see the formula you basically get this big formula and um this part is nothing but this weight that we see here one by three one by six uh one by three and one by six and this is the marginal contribution of player I to correlation s okay so this is this lookup table of five thousand two thousand five hundreds and then when python that we have calculated right now there are some theoretical reasons why this turns out to be share or a fair allocation uh so I'm just listing them down one is symmetry uh so if two players are interchangeable and they make the same contributions to all coalitions okay and then there's obviously a null player property which is basically a player mix zero marginal contribution to all current then they get a null value and Main is the additivity is it if we combine two games then we can basically sum up the contribution for those two individual games uh for any of the player so on these three axioms uh the shapely value is basically kind of derived from and this turns out to be a fair and full allocation that is whatever the value is 10 000 everything gets divided uh amongst each of the players now in this context this may seem very trivial but if we apply it or the way it has been applied in for future importance is that value of the game is basically the model prediction and the feature value are the players right so if we go back to the first example then C12 is nothing but feature one and feature two combined gives a prediction of 10 000. and here feature one by itself or feature one equal to some value by itself is contributing 7500 and feature 2 by itself is contributing five thousand then how important is feature one and how important is feature two is exactly the problem that we are trying to solve right and this is done for every instance so every instance is a game where we try and do a fair allocation of shapely values right and just getting into a small certainty it is not the features but the value of the features for a particular observation that play the game right so when I say feature one one instance would be say feature one equal to 10 feature 2 equal to 20 and the prediction is say 10 000 right so the player becomes feature one equal to 10 when the value of feature 1 is equal to 10 what should be the allocation given to feature one right so this is the way uh the shapely value has basically been applied uh to uh the interpretable model and sharp is just one of the implementation so shapely value was there for a long time but sharp is one of the implementation of safety values which one it has a great set of visualizations to visualize the feature importance and two is that it has a very very optimized implementation for exclusively for tree based methods right so though the entire shapely value concept is model agnostic but it's most optimized implementation is used only for tree based methods right so for other methods there could be like a Time trade-off and that's where lime maybe becomes uh better in some of the cases um because time could be much quicker and would give you a sense of the rank ordering of the feature importance right so I won't go into much depth into these graphs but just to so that kernel shap is something that is model agnostic so it can be used for all kinds of models and Tree Shop is basically the um in the optimized version of uh shapely values that has been developed uh for tree based methods and that is the main main feature of the sharp package right so let's let's just have a quick uh look at the sharp package [Music] so so here I can show you one example quick example for lime and sharp and then maybe move on to the Cricut data set so so for lime there is a package called lime itself uh in Python which we can use so let me just run this node or maybe I'll just go through this notebook rather than learning it so this is another data set of uh shells basically Avalon is kind of shells and where we are trying to basically predict the age of the shells given certain features like length diameter height right uh so we filter aggressor and here we are basically creating a lime tabular explainer right and once the explainer is instantiated like we can see here here we basically mention how many features do we want to use right so if we go back to our slides um we do remember the step that there is an option of basically just selecting the M best features for that local output correct uh and then lime kind of gives this kind of uh an explanation um so positive negative would be uh just that which features have affected in a positive way and a negative way it also now here this is a slightly important piece where you see that the right right here means the actual model prediction is 12.6675 whereas the local prediction is 10.88 right so this is what is uh different in chat uh here the local prediction is 10.88 so your weights whatever linear model it has failed or even uh or it has built or say a tree based model it would add up only to 10.88 and not the complete 12.66 so there is it if first of all it may not be a fair allocation but even the allocation is not full like the entire value of the output is not allocated to all the features right hence lime is a sub optimal or a very in a in some it's actually a very very special case of um uh offshape right so if you actually we use a kernel shap with a linear model we kind of get lime uh interpretations okay but uh just to go through this so lime basically will give you these feature importance and it will also give you a small table of what the future value was now this is one small hack and I will uh this is also taken from another article but I'll post this uh post the GitHub link later on is that shap has these nice plots uh so first just let me show you the sharp plots right all right just maybe let me run the yeah we run this so let me just show you the visualizations available in shap uh so this is an insurance data set just run through this quickly so where we are trying to predict the price of the premium like how much this is a health insurance data set where whether the attributes like whether it's smoker or not uh what is the age female or male and then the BMI and so what would be the eventual uh premium right that would be charged so once we build the model uh so the first plot that shap has uh is basically this bar plot showing how important uh each feature is so in order to now this is a global view as I said that even for local models um what it basically does is it takes the absolute shap value of each feature and it sums it up across all the instances so any feature which has affected in a negative way or a big positive way would basically uh be all added up and the largest value would be the most important feature so here we can see that for health insurance uh smoker being a smoker or non-smoker seems to have the largest impact on the premium then comes the age BMI children like very intuitive rate but it is quantifying the importance uh in an appropriate way so that is something to um uh just keep in mind then it has these nice waterfall plots uh these waterfall plots though are not available for the tree methods uh they are only available for the general explainer but it kind of nicely shows uh how each feature has contributed and what the feature values are to and you would see this that it always gives the prediction or the contribution from the mean value of the entire data set so that is because uh when we go like if we go back right to this Slide the value of the Coalition of say two features is the incremental value over the average okay it is not the final model output because if we didn't have any feature the easiest thing that we could do is predict the average right if we are optimizing the mean square error we just predict the average of all the target values so that is like the base that is c0 with nothing no feature right so with all features combined the value of the Coalition becomes the incremental prediction that it has done it could be in a positive or A negative way hence when the sharp plots are plotted it always keeps the expected value as the reference and then gives how much incremental differences each feature got into the output so here if I had to draw parallel with the game theory Concepts so the value of the Coalition was um it would have been two six seven two six minus one two eight one three whatever that difference is and the players were basically smoker equal to one age equal to 62 BMI equal to 26.29 so there are future values at the players and basically the incremental value becomes the value of the correlation then it has Scatter Plots that you can plot to see the how the shape value varies with uh the feature value uh then similarly there is also something called a force plot which is very similar to the waterfall plot uh waterfall plot uh just like more compact way if you are kind of seeing two three uh explanations so here you can see that we are just seeing one instance right and seeing how a feature has contributed in a particular way then there is something called the decision plot where we can take a slice of the input data and see the decision path decision making path so these are all variations of just the same shape values that have been obtained but different ways of viewing them right and finally is the B swarm plot which is one of the most common plots so just to explain this in more detail uh this would be arranged by the order that we saw in the first bar plot the most important feature on the top second most comes age the coloring is based on the feature value so what this says is when smoker is high now smoker is actually a binary variable we know that so if you see when smoker is One smoker is equal to one this the shap values are usually on the positive side and when smoker is equal to zero the shaft values are on the negative side so the premium becomes lesser if you are a non-smoker right right so similarly with h but you can see that for those smoker is equal to one for different instances it has different shape allocations the shape allocations are not the same because it would depend on how other features have interacted for that particular instance so though they are all high they are basically uh they're different and will add up to the final predicted value that is the additive nature of shape explanations so yeah so this kind of gives you the different plots just to once again go through it we have the bar plot for the global feature importance then we have the waterfall plot for looking at a local importance similarly uh we have the force plot we're looking at one local uh explanation then we have the decision plot where we can choose to select as many instances as we want and plot each of the decision functions uh so it's kind of an ice plot that we had seen right earlier uh where we see the curve for each instance though this is on all the features so it's not same but it's the local version of the global ice plots and the B swarm plot which kind of gives us the sense of correlation of how the feature correlates with the sharp value right yeah so these are the plots in shape so let me just run you through one final notebook and one real data set which uh kind of we created so let's say that we are analyzing a cricketing data set so I've picked this data set from uh it