ML Interpretability: feature visualization, adversarial example, interp. for language models

Key Takeaways

hello guys welcome back to my Channel today we are going to talk about machine learning interpretability let's review the topics of today I will be starting uh by introducing what is machine learning interpretability then we will review deep learning and B propagation because they are needed for us to understand the rest of the videos and the topics then we will see a nice little trick so how to trick a classifier so imagine that you have a classification naral Network like for example a convolutional naral network that can classify pictures into classes for example it will tell you that the picture of a dog is a dog the picture of a person is a person Etc our goal is without touching anything of this model so without touching its weights without touching its parameters its structure or anything related to the model we want the any classifier to be tricked into believing that for example the picture of a dog is actually a person or the picture of a person is actually a dog and we will see how we can trick any classifier of our choice later I will be introducing the an interpretability engine so a library that allow us to make Vision models more inter interpretable uh we will explore the topic of a feature visualization which is very important for interpretability and finally we will apply the techniques that we have learned to language model so how to make language models more interpretable what are the prerequisite for watching this video well for sure that we have a little background in calculus I think knowing what are derivatives and how to calculate them is enough and uh also of of course that you have a background in deep learning so you know what is a loss function or what is the soft Max function for example so let's start our journey what is machine learning interpr interpretability um in 2016 there was a fatal accident between a Tesla car driver and a Truck and as reported by the guardian we can see that the let me use the laser the car sensor system against a bright spring sky failed to distinguish a large white 18 wheeel truck and trailer acrossing the highway so basically the car was going and it failed to recognize this obstacle which was the truck and the car just attempted to drive full speed under this truck uh of course resulting in the crash and it was unfortunately fatle now I don't want to say that it was Tesla's fault or it was the software's fault I don't have enough information for that so let's make a hypothetical case like you are creating a self-driving company and um you want to deploy your car self-driving car how do you make sure that the car can recognize any obstacle uh how do you uh know what do your model has learned because for example if the car uh you the first answer question that you want to answer is what your model has learned for example imagine that you have a model that allow us you to segment the obstacles on the road the first question that you want to answer is what did my model learn so how does my model uh my model recognize a person uh does it recognize a person by its shape or does it recognize its person the person by its shoes or by uh the the color of the clothes Etc knowing this is important why because this allow us to understand what could be a failure mode of our model because if our model is only looking at the color of the clothes for example to recognize a person so only looking at the clothes and not at the face for example then if one day the model will see a person that is wearing strange clothes so something that the model has never seen the model may fail to recognize that person as an obstacle so this is very important so the second question that we want to answer is what features or patterns from the input make the model generate certain outputs and this is very important for example for language models so imagine our language model is a cursing and we want to understand which tokens in the input are being uh used by the model to generate that kind of output knowing how a model thinks and pardon me for this word it's very wrong so let's think about knowing how a model predicts or makes its prediction allow us to debug and fight tune the model which means that during training we can understand why our model is not learning uh something that we wanted to learn or how should we change our hyper parameter that will affect the learning of our model we can identify failure modes before deployment which means that we can understand how what are the things that will make my or are more likely to make my model fail when deployed in production it can increase trust because we can demonstrate that our model is well trained and so people will trust it and this is especially for like some for some scenarios like self-driving cars and also we can discover novel insights from the data because sometimes models learn something that we as humans did not see and this is very important when the models learn pattern for example in image models that humans did not see uh this is for example in the healthcare sector imagine that you are training a model that can recognize a cancer cells from non-cancer cells and we realize that uh the model is performing well and we realize that the model is looking at some parts of the cell that we as human didn't think of checking before that is actually a good predictor for the cell being canc or not cancelers so even the model can teach us something that we did not before when we Define a linear layer in P torch it gets converted into a computation graph for example in this case we have an input that is made up of two features which means that the input is a vector made up of X1 and X2 we for example in this case we have two linear layers one that converts two features into two features so it's taking two feature as input and it's converting it into two features as output and we can see it as a layer made up of two neurons each neuron that is doing um uh weighted sum of the input features multiplied each input feature multiplied by its own weight so X1 is multipied in this case by w11 and X2 is multiplied by W12 and then it performs the sum plus a bias and then we have the application of a um nonlinear activation usually the relo function then we have another linear layer that is going from two features to One features that will produce our output in this case we are trying to model a very simple n network that takes an input two features that represent a features of a house for example the number of bedrooms and the number of bathrooms and wants to predict a price for this house so only one output this is a very simple regression task and we can train it by having a training data with two input features and one target uh pytorch will convert this neural network into a computation graph what does it mean it means that each node will become a operation that is performed on the input subsequently to arrive to the final output which is the price of the house in this case this is actually a simplified version of the computation graph the computation graph usually is made up of more nodes than the one you say here because each kind of single operation is a node and in this case we can can see that X1 and X2 are multiplied by some weights in the first neuron and then they we sum up a bias we apply theu and this becomes the input for the ne neuron at the next layer so we take the two output at the previous layer we multiply by them some weights as you can see here w31 and w32 we add a bias and this becomes our output how do we train train such a network well um usually we have a data set made up of input and output pairs or input input and label pairs the input represents the features of a house and the output the corresponding price uh the price of this house and our goal is to train the neural network to minimize a certain loss function that we can choose for this regression task an ideal uh loss function could be the mean squared error because we want to minimize the error that the model makes on the final price our hope is that the neuron network not only only learns the data that it has seen during training so not only it can predict correctly the price of the houses that it has seen during training but it also learned some kind of pattern that can generalize to unseen outputs inputs so how do we proceed practically we choose the Lost function and we choose for example the mean squar error in this case we run an input through the neural network so we take the input we run it through this Neal Network which is a Feit forward naral Network which means that each input becomes the each output become the input of the next layer so we run our input it will produce an output here we have a loss function that will compare the output of the network with the label that we have assigned to this input label pair it will compute a loss then we calculate the gradient of the loss function with respect to the weight of the network and the weights of this network are also the parameters of these networks and in this case are w11 W12 the bias here by B1 w21 w22 B2 which are the weights and the bias for the first linear layer and also the weights and the bias for the second linear layer um we calculated this gradient because the gradient indicates kind of a Direction so if you remember from high school uh the gradient is basically a derivative so imagine we do it for a single variable so imagine that we have we want to calculate the derivative of the loss function with respect to w11 so we write it here this is the loss function and this is uh w11 imagine that the loss function is doing something like uh this uh we have a kind of a local Minima here and then we have a global Minima here imagine that we are currently here our w11 initially is here when we calculate the derivative of of the loss function with respect to w11 we will get kind we will get the inclination of the tangent line at this points which is this one here and this indicates that the direction in which the function is growing so the function is growing in this direction we usually update our weight to move to the ne to the opposite direction of the gradient so we update our weights to move right so that the loss will diminish so for example we take a little step in this direction so that the loss as you can see will decrease because this will be the new loss two we started from loss one here and this is why we do back propagation so we calculate this gradient so the gradient of the loss function with respect to the parameters of the model and then we update the models to move against the direction of the gradient the first thing that we do during our training is the forward pass which means that we have an input we run it through our computation graph to calculate an output so let's do it here so we have an input that is X1 is equal to 2 and X2 is equal to 4 we multiply for example here in this node each input by the weights of this network and the weights are initialized as follow so w11 is equal to 0.24 W12 is equal to 0.29 the bias 1 is equal to minus 0.70 this will result in some Co activations being calculated so the values of each node are are called activations we use the previous activation to calculate the next one etc etc until we arrive to the output put of this neural network we have a Target because we are training and we can calculate a loss what do we do with this loss we run back propagation which means that we calculate the gradient of the loss function with respect to each of these weights for example to calculate the gradient of the loss function with respect to w11 which is this parameter here we can use the chain rule which means that the derivative of the the loss function with respect to a A6 because we need to watch what are the nodes that connect this parameter to the loss so the nodes that connect this parameter w11 to the loss function are this node here this node here this one and this one what we do in the chain rule is we just go from the loss to the parameter backwards so we do the loss function with respect to the previous node then this node the derivative of this node with respect to the previous node then the derivative of this node with respect to the previous node the derivative of this node with respect to this node and the derivative of this node with respect to w11 which because this node contains the expression w11 this will result in a series of numbers that will give us then when multiplied together will give us the derivative of the loss function with respect to w11 what do we do with this derivative which is a number because we evaluated this derivative on the input points that we have chosen it will give us a number that we will call gradient and we use it to update the value of our parameter so the new value of the parameter w11 is equal to the old value of this parameter minus Alpha which is our learning rate multiplied by the value of the thisg gradient now why do we have a minus sign here because as I saw as I saw before the gradient indicates the direction in which the function is growing with the loss function is growing with respect to the parameter and we don't want to make the loss function grow we want the loss function to decrease so we move in the opposite direction of the gradient so that's why we have a minus sign here now let's trick a classifier so I introduced the gradient descent because it was needed for us to understand how to do this trick so imagine first of all what is a classifier a classifier is a Neal Network that can classify the input into one of the defined classes that we have for example in this case we may have a classifier that can take an input picture uh input picture and then can classify it as a fish or as a dog as a volcano as a car or a pencil for example the resonet network can classify the input picture into one of thousand one of the Thousand classes it has in its output Logics the output of a Neal network of this this kind is called logits because it indicates what is the score that the network assigns to each of these classes we don't usually work directly with the logits we apply a soft Max a softmax is a function that makes the Logics kind of turn them into probability scores because they will sum up to one and then we take the the the class with the highest value of the softback as the um as the prediction of the model so if after applying softmax we see that our network uh indicated that this uh node here is 95% then it means that the network is saying us that this is a fish uh and that also applies to other cases of course so what do I mean by tricking a classifier I mean that I give you a classifier so neural network like this one and you are not allowed to change anything of the network so you are not allowed to change the uh the weights of this network you are not allowed to change the architecture of this network you're not allowed to change anything so the weights are frozen and the architecture and the hyper parameter everything is Frozen when we run a picture of a fish in this network it will probably classify it as a fish but our goal we want to give a picture of a fish as input and we want the network to classify it as something else that we can choose for example as a volcano with very high probability so of course if you think about it the only place where we can work to trick this network is actually in the input and this is what we will do we will change the input in such a way that the network will not see a fish anymore but it will see for example a volcano so this this fish and the previous fish are seem are the same for a human because you can see a fish and I can see a fish but what we did was to add a little bit of noise in this picture so that when the network sees this picture so this one here it will not see a fish anymore it will see a volcano how is that even possible let's see uh we want to so our goal is we want to have a picture as input and we want to change this picture in such a way that the network sees something else how can we proceed well we take uh what we do usually when we um train a network like this is this we have a series of pictures of images of fish of trees of people Etc and the corresponding label so for example we have thousand pictures of dogs and the label saying that they are dogs and then we have a thousand pictures of cats and saying that the label corresponding label is cats Etc so what we do is we give the feed the input picture to the Neal Network the Neal network will calculate some output which is here to which we apply the soft Max then we have the corresponding label because we know what is this picture it's coming from our training data so we know that is a fish for example in this case we can calculate the loss the then we can run back propagation which means that we calculated the gradient of the loss function with respect to the weights of these Networks so the parameters of this network and then we update the parameters to reduce this loss and this is how we train this network so let's try to see how we can trick the network into believing that for example this fish here is actually a volcano now when we do the training as you saw before we calculated the the gradient of the loss function with respect to the parameters but we can actually calculate also the gradient of the loss function with respect to the input picture so what we can do is as follows we can create a new loss function so imagine we have a picture of a fish we know it's a fish but we want to trick the model into believing it's a volcano we can create a loss function with respect to the Target that we want the network to have so we want the network to believe it's a volcano so we can create a new loss function with respect to the Target volcano and then we run this picture in the network and we calculate the gradient of the loss function with respect to the input and later we will show we will see in the code how to do that but let's try to analyze what does it mean to calculate the gradient of the loss function with respect to the input it means that it will indicate a direction in which we should change the input to make the because the this a gradient so it indicates a direction in which we should change the input to make the loss grow so we can run back propagation and optimization on the input to decrease this loss so it because the grent tells us how we should change the input image to make the loss grow we can also change the input image in the opposite direction to make the loss um decrease so that's what we will do uh we calculate the gradient of the loss function with respect to the input image which will indicate a direction we update the image with some noise in the opposite direction so we add a little bit of noise in the opposite direction indicated by this gradient and we keep updating it until the network predicts correctly the output as volcano so in the code it is done as follows imagine we have a model we have an input image and we have a target class for example a volcano what we can do is we take our input image and we create a tensor of it by asking the pych to also calculate the gradient with respect to this tensor because by default pych will only calculate the gradient with respect to the weights so the gradient of the Lost function with respect to the weights but we also want the gradient with respect to this input image then we run for a few steps the following we calculate the output of the model so we are calculating this output here we are creating a special loss function with respect to this target that we want the network to have so we want the network to Output a volcano so we create a loss function with respect to this target class we run backward which means that we calculated the gradient of the loss function with respect to the input and then we update the image so the image is updated just like the update formula for the parameters so it's equal to the old image minus some learning rate here I call it Alpha multiplied by the direction of the gradient at of the loss function with respect to the input and we are moving against the direction of the gradient because we want the loss to decrease if we update the image continuously as follows we will see that the network will predict it as a volcano and this is how we can trick a classifier I made this example because I wanted to show you that models may look at patterns that are completely different from as humans for example in this case the model is predicting this picture as a volcano so the model somehow is seeing a volcano here even if to us humans we we will never be able to see a volcano in this picture it's a fish so under understanding how our model makes its prediction can help us improve our models and thanks to the sponsor of today's video Lip laabs we can get insights into how our model makes their predictions lip laabs is a research lab that is focusing on machine learning interpretability and they have developed this Library the lip laabs interpretability engine that allow us to understand what our model has learned and how we can get insight from our model to improve it for example this uh Library allow us to generate prototypes prototypes what are prototypes well imagine that you have a classification model a computer vision classification model which means that you have some it takes as input a picture and it will classify it as one of the classes in this case it's a food classifier that will classify an input picture as one of the following class for example ice cream or hamburger or pancakes or waffles in this case it looks like that the model is well trained because by generating prototype we can get a the the kind of input that the model wants to see to classify it as a target class so this is the kind of picture that the model wants to see to classify the the input picture as a hamburger this is the kind of picture that the model wants to see to classify the input picture as a pancake and it actually resembles a pancake and this one actually resembles a hamburger which means that the model has learned the correct features from the food to classify it as a given class but we will see later a case in which is this is not true uh another feature of the lip laabs interpretability engine is entanglement entanglement allow us to understand how different classes share features for example for a food classifier like the one we saw before we expect High entanglement between the ice cream class and frozen yogurt class because at least for me as a human they look very similar they both look like ice cream so it's it is expected that these two classes share features but imagine that you have a more broad classifier like the one we saw before that can classify fish and volcano Etc I would not expect for example cheesecake and dog to have high entanglement because at least for me they don't they shouldn't share features uh I mean they are totally different objects so so if they do have high entanglement in the model it means that there maybe the the model is looking at wrong patterns to uh add the wrong features and it also May indicate a more higher chance of misclassification between these two classes uh another feature that is very important is feature isolation so in this case feature isolation allow us for example to understand which parts of the input is being used by the model to make a certain prediction for example for a food classifier imagine we have the following picture the the food classifier will classify it as a fro frozen yogurt with 98% probability but the by generating future isolation we can understand which part of the input is being used to classify the input as a frozen yogurt and it's actually the part that looks like frozen yogurt but also because there is entanglement between Frozen fren yogurt and ice cream the model as you can see is using the similar features to also classify it as ice cream with a low probability because the model is well trained but still they have some shared features as you can see and there is something that you may not have noticed but is the waffles with very low probability the model may also classify it as a waffle why because the model is a seeing some features which are the berries that are in on this um Frozen Yogurt to classify it as a waffle this can happen because in the original picture in the training data the waffles probably had berries on top so the model learned to look at the barriers to recognize a waffle so the lip laabs inter stability engine can understand this and we'll show you this this helps you understand what your model has learned now let's look at a case on how when things can go wrong in our model and how lip laabs inability engine can help us improve it if you look at uh the tutorial link that I have shared in the description uh there is uh if you go to this uh link here to the tutorials at the lip laabs website you will see tank detection case study and if you open it it will open a collab notebook now uh let's run it actually so let me uh change run type we choose T4 GPU and we can run it it will do some Imports now what is the tank detection case study well we are talking about a classification model that can detect tanks or no tanks so it has only two classes that indicates if the picture contains a tank or it does not contain a tank suppose that this is a model that is very important for us and we want to deploy it in the battlefield because it can help protect our um soldiers but before deploying it of course we want to understand what our model has learned so and by understanding what our model has learned we can predict um failure modes so if we run for example a picture of a tank into our classification model we will see that it it classified as a having a tank with a very high probability so in this case the model is predicting that there is a tank in this picture with 98% probability so looks like the model is performing very well but let's try to use the lip laabs interoperability engine to understand what our model has learned so so we install the library then we can use the library to generate prototypes as we saw before the Prototype tells us what kind of input the model wants to see to classify a certain output to give a certain output in this case we need the lip laabs API key which we can generate from the lip laabs website so we go to the dashboard we go settings and it will generate a key here we put our key in the API key and we can generate a prototype in my computer it takes around 25 seconds I think or 1 minute to generate it okay the model has generated the library has generated two prototypes one for the tank class so uh what kind of input the the model wants to see to to to tell us that there is a tank and what kind of input the model wants to see to tell us that there is no tank and uh let's look at this picture which indicates the when the output indicates that there is a tank if we look at this picture we see that actually there is no thing so it means that the model is looking at some stuff that is gray which probably looks like Cloud uh and but there is no tank here I mean I expected to see a cannon I expected to see some wheels or maybe the the gun or a soldier with a gun on top of the tank or something like this but actually there is none of these features so is our model looking at the correct features to actually predict a tank the presence of a tank and let's look at the other class no tank as we can see we have this like green lines here which probably indicates grass so the probably the model is looking at the grass to indicate that there is no tank so if it's opens sees an open field with only grass it will say that there is no tank which could make sense but the problem is why is our model not looking at the tank to indicate that there is a tank so let's try to make a prediction before looking further at what the lip laabs interp the engine can uh tell us what could happen in this case is that imagine that in our training data we have a lot of pictures of tanks and all of them that have tanks happen to have cloudy sky so what our model may have learned is that if there is a cloudy Sky then there is a tank not that if there is a tank there is a tank so let's validate our hypothesis we can use a feature isolation to understand what kind of features from an input picture the model is looking at to make a certain prediction so in this case for example we can feed the um the picture that we uh saw before so this picture as input to see what kind of features the model is looking at to predict a tank let's see as you can see the model is using the entire picture to actually predict a tank but as you can see the the white areas indicate that that feature is not being used and the other areas indicate that the feature is being used so as you can see the tank is here is white which means that the the model is not using the tank to predict the tank but it's using the sky and maybe the ground to predict that there is a tank so the as suspected the model doesn't seem to use the actual tank for classification much at all right it's using the sky the background and maybe the saturation of the picture so how can we fix this model well one way to fix it is to further train the model by using more diverse images of tanks that have maybe some Sunny Sky maybe some cloudy Sky maybe some snowing environment with snow and some environment maybe in the forest etc etc so that the model cannot find any other correlation between pictures of Tanks except for the tank itself so that the model will be forced to learn the presence of the tank itself as um predictor for tanks uh we can run this training and it will for sure improve our model and there is a code here to how to train it again and after training we can run feature isolation again and we can see here at the end that after retraining the model on more diverse pictures the model is actually putting all its attention on the tank itself to predict the tank and not on the surrounding area so all of this TKS to the lip laabs interpretability engine now let's talk about feature visualization so what we saw before with lip laabs interpretability engine is that we can get insights into how our model is making its prediction or what kind of feature of our model has learned and in particular especially for convolutional neuronal networks for computer vision we have of course a subsequent application of layers of convolutions and our goal with feature visualization is to understand what each of these layers or what each of the neurons that are making up these layers ERS what kind of features from the input did they learn that contribute to the final prediction so we want to understand for example imagine that you have a food classifier and you have many layers in your convolutional Neal Network each layer will be looking at a particular kind of feature in the input that will contribute to the final output for the final classification some layers may look at for example lines some some layers may be looking at edges some layers may be looking at certain patterns etc etc so we want to understand what features each of these layers or each of these narrow have learned and we can do feature visualization at many levels we can do it at the neuron level so what features is this neuron looking at or we can do it at the layer level so what kind of features is the particular layer looking at and also at the logic level in the case we are doing um uh we have a classification Network so we want to understand what kind of features the model wants to see in order to uh predict it at that particular class so uh we will model the feature visualization problem as an optimization problem and it's actually how it's done in practice and it's actually also how more or less the lip laabs interpret engine works of course it's much more sophisticated so this is a simplified explanation but I wanted you to understand how such an engine works so that that when you use it you you you also know what's happening inside so what we do imagine that you have a classification Network a convolutional network that is used for classification so as you can see here at the end we have a soft Max and we have subsequent layers of uh convolutions we want to understand what this layer of convolution has learned so in order to understand what kind of features this uh layer has learned we will do um we will treat it as an OP ization problem which means that we will create an input that is a complete noise we run it through our Network we take the activations of this uh layer so all the outputs of this layer and we use it as an objective function so or you can also call it a loss function so it's the same thing so you take the output of this as loss and then you optimize the input to maximize this loss in this case so that's why I'm calling it objective whenever you are maximizing something we call it objective function whenever you are minimizing something we call it loss function but it's the same thing the only thing that changes is that in one case you are doing gradient Ascent in the other case you are doing gradient descent in this case we want to maximize the output of this the the activations of this layer so we treat the output of this nwork as objective function and we run back propagation to maximize these activations and this will modify the input put in such a way that it maximizes this activations this will get get us insights into what kind of features this layer wants to see to uh have um to to contribute to the final prediction um we can also optimize for logits so for example if instead of using a particular layer we use the logits of a particular class for example the class associated with dogs because we want to see what kind of dog our model wants to see to predict it as a dog we can use the Logics corresponding to the dog class we feed as input complete Noise We Run uh we use the logic corresponding to the dog class as an objective and we run back propagation to optimize this input to um maximize this logic uh and yeah this is actually how you can generate kind of a prototype for the class uh dog uh of course you may be wondering is it that simple well not really because when if you do this procedure so if you start from complete noise and you try to maximize a certain logic it will for sure give you insights into what the model has learned so what kind of input the model wants to see to have that logic as output so that class as output uh but it will not look very natural so for example this image here I believe it's taken from resnet and uh from uh in which we can see that for example if we optimize for the class Flamingo we see that the input needs to have something like this long necks here which are typicals of flamingo which means that the model will actually look at these long necks of flamingos to actually predict the flamingo class uh if we look at for example goldfish we can see that we have this eyes of this goldfish here and for example this one looks like the shape of a fish so the model will actually look at the fish to predict um the Goldfish and if we look at for example tarantula we will see this long black legs here like this one like this one which means that the model actually will look at the legs of the tarantula to predict it as tarantula uh but of course you can see that this picture they don't look really natural because if you look at the lip laabs inability engine they look quite natural so for example if we go back and we look at the prototypes generated for pancakes it's actually looks like a pancake and if we look at hamburger it actually looks like hamburger um so how can we make our uh inputs more look more natural well one for once you could use the lip laabs interpretability engine which can do it out of the box but to understand how lip laabs do it they use what is known as regularization let's talk about regular iation so first of all what is regularization when we train a model our goal is to uh run some input through some input through this model calculate an output compare it with the target so that we can calculate the loss and then update the parameters of the model such that we reduce this loss um when we introduce regularization we want this optimization to happen in a particular way so we want to put some constraints in our optimization process for example in when we train a model we can do what is known as L1 regularization with L1 regularization what we do basically is we have our loss function which is our let's say uh cross entropy loss cross entropy loss because we are doing for example classification task then we can add some regularizer which is a constraint that we add to our loss function to have to make this optimization process happen in particular ways for example with L1 regularization we we we we want our models to use the least possible input features from the input so what we do as regularizer we use the L1 regularization which is basically just the um absolute value of all the weights what does it what what happens in this case what will happen is that because we calculate always the gradient of the loss function with respect to the weights of the model this uh the presence of this absolute value on the weights on or the parameters will force these weights to become zero and because the weights will become zero they will use less features from the input and this helps to make the model more sparse which also helps us to then reduce the size of the model so regularizers are particular constraints that we add to the loss function to make this optimization process happen in particular ways to to add some constraints to this optimization problem and this is what we can also do in our optimization problem so what are we optimizing we are starting from Pure Noise for example this is our Pure Noise and we want to transform into some kind of input that maximizes a particular output logic in our classification Network um of course when we train a net neural network the data set that the network was trained upon let's say that in the space is here but this doesn't not mean that the model will not uh activate the output Logic for example corresponding to the class dog for something that is out of distribution so what we want to do is we want our neural network to optimize our input noise so sorry we want our optimization problem to optimize our inut put noise in such a way that we remain close to the distribution of the data that the network has seen so the natural input that the network has seen how to do that well first of all look at my picture do you think it's a noisy picture no because if you look at my t-shirt you can see that Adent pixels they are similar and there is no much variance in the pixel for neighbor for neighboring pixels so we could ask our optimization problem to optimize the input in such a way that it penalizes high variance for uh neighboring pixels and this is known as a frequency penalization so we take our loss function which is our basically just the logic that we want to maximize and we add a penalty to this loss every time we see a very high variance for neighboring pixels another regularizer that we can use is the transformation robustness this is not applied to the loss actually this basically means that we take our input that the one that we are optimizing we transform it some way so we can rotate it we can scale it we can translate it in this case uh this code that I took from the Lucid uh Library which is a very famous library for feuture visualization they apply the um random scaling and random rotation uh which means that they will rotate and randomly scale the input and then pass it through the network and the network because it's a optimization problem will have to the optimization problem will have to modify the input in such a way that even when it's translated even when it's rotated even it's when it's scaled it will still activate that output so it will only affect the pixels the input features that are needed for us to actually activate the um that logic which also means in other words that it will try to for in case we are trying to for example maximize the Lo the logic corresponding to the class dog it will actually try to create a dog because it does not matter if the dog is rotated it does not matter if the dog is scaled it does not matter if the dog is uh translated it's here or it's here it will try to create so it will try to create a natural dog as much as possible of course there are many more regularizers that we need to add to make this uh transformation to to make this optimization problem more robust so that we don't get some out of distribution data but we want to try to generate data that is as in distribution as possible and this is also how lip laabs works so the lip lab interpretability engine can generate prototypes that look natural and the way they do it is described in this paper called the Prototype generation robust feature visualization for data independent inter interpretability in which they describe the process of generating this prototypes and the way they do it is basically they apply all these regularization techniques so for example you can see here random transformation so that the the optimization process produces an input that is as natural as possible uh without ever actually seeing an input so as you remember in the we do a prototype generation with lip laabs inter interpretability engine we never feed an input picture we just give the model and the algorithm will generate a prototype without ever seeing what a natural picture looks like but it's actually generated it can generate very natural uh inputs why because they make this uh optimization process very robust so they penalize for example the the the the high frequency the high variance in the neighboring pict pixels they also apply transformation etc etc so that the resulting input is as close as possible to the Natural inputs that the model is trained upon now let's try to use the knowledge that we have Acquired and apply to language models so as we saw before with the computer vision models we can do prototype generation which is based on feature visualization which means that given a particular for example output Logics we want to understand what kind of input the models want to see to that particular Logics as output can we apply the same techniques also to language model so given a desired output what kind of prompt the model wants to see to generate that output well let's try to answer that question first let's review how language models work so a language model as you know is a probabilistic model that assigns probabilities to sequence of tokens uh for example imagine that the input to the language model is Shanghai is a city in ch in the model will tell us what is the probability of the next token being China or being Beijing or being cat or being pizza or being whatever token is present in our vocabulary one simplification I always do in my video is to associate a token with a word and the word with a token but this is not usually the case so usually a word may not be a token and the token may not be a word and actually most of the cases a word is actually made up of multiple tokens but for our case we will simplify it and see that every token is a word and every word is a token so the language model just tell us what is the probability of the next token given an input prompt imagine we want to understand what our model thinks of the word girl so what kind of um input what kind of prompt the model wants to see as input to predict the word girl as next token well uh let's try to use the techniques that we have seen before so first let's see the results of such an analysis and in particular Jessica who is the founder of lip laabs she did this study so she took some the some tokens for example the word girl and then she optimized the input prompt in such a way that the output girl is maximized so the next predicted token is a girl given this input and she did it also for the word woman and she did it for the word good and for the word doctor this gives us insight into what our model has learned because our model our language model is just a model that models the statistical distributions of tokens based on the training data it has seen so in this case for example the to The Prompt that maximizes the word girl as being the next token is this input here and as you can see it tells us that our model has seen a lot of um bad data that is making our model have a bias against girls for example because we see sexual words we see other girls that are not quite polite and the same happens for well in the case of the woman the word woman it's a little better but still it tells you what are the bias of your model against this particular concept and we can see it also for the word good for example we see that shooting is good or somehow and Jesus and beautiful or um basketball Etc so optimizing uh the prompt to generate a particular output tells you what our model wants to see as input to generate that particular output which gives us insights into the distribution that our model has learned and Jesse guy she ran another experiment which is to because when we optimize The Prompt we will see how it's actually done in practice we start from complete noise and we optimize this prompt to um to to become tokens that are more likely to predict a particular output and of course you can restart this optimization problem from multiple starting points because you can start from complete noise and you have many starting points so many input noises you can have as starting point and she did it many times and she got some input tokens that were more likely to predict the word girl as next token this gives us a map on what kind of inputs the model wants to see with each of um tokens with its frequency what kind of inputs the model wants to see to predict the girl as next token or boy as next token or science or art as next token and this also gives us insight into the statistical distribution that our model has learned for example the word girl um to get the word girl as output the model wants to see some sexual words and some other like not so uh some curse words also but also for example the word dresses or the word boys and in the case of the word boy we can see that it wants to see rebellio monkey girl etc etc but this gives us insight into what our model has seen during its training so now let's try to analyze how to act how to actually generate this kind of map and how this optimization problem Works what we did before with the computer vision models that is we have some output Logics for which we want to find an input that maximizes that logic is exactly what we want to do here except that here we have a language model and we have another complexity so let's do it step by step how we can generate this kind of map imagine that we want to find input embeddings that maximize the m the probability of the next token being girl now the first complexity is that girl may not be one token but it could be multiple tokens so let's suppose that it's actually multiple tokens because this is a real scenario so we have that the output that we want to optimize an input for and suppose that we want to optimize three input embeddings so let's draw three input embeddings to maximize the probability of the next token being girl but we know that girl may not be a single token so let's suppose that it's actually two tokens so one token it's GI and the other token being RL now the job that we did before that is calculating the loss of the output with respect to the input it's something that we cannot do anymore why because the input in the language model is tokens and token are numbers that represent the position of this token in the vocabulary so for example imagine the input could be for example 0 five and nine and these are positions that represent each token in the vocabulary and we cannot optimize for something that is discrete because there is no token 0.5 there is no token 3.2 we cannot change these tokens a little bit hoping that they move towards something that represents that will generate that kind kind of output the only thing that we can optimize are embeddings so we will not be optimizing input tokens we will be optimizing input embeddings so in this case we suppose that we have three embeddings so let me uh delete this part so we we suppose that we have three input embeddings now which three input embedding should we choose well in the case of the computer vision model we started from Pure Noise in this case we can also start from Pure Noise so we can start from three random embeddings one two and three what we can do we can run these three embeddings in our language model and as you know the language model is a transformal model in most of the cases and it's a sequence to sequence model that will generate if the input is three embeddings it will generate three embeddings as output so here we will have three embeddings our goal is to make sure to to select three embeddings that make the likelihood of the next token being GI and the next next token being RL maximized so how to proceed we take these three embeddings we run it through our model it will produce three embeddings as output usually when sampling from a language model we take the last embedding so the last hidden state so the output of a language model here at this point is are called hidden States we take the last hidden State we send it to the linear layer and it will generate what are known as logits logits indicate what is the probability score it's not a probability actually but what is the score that the model assigns to each Tok in the voca