Data Distribution Shift | Covariate Shift | Label Shift | Concept Drift | Explained with Example

Hello and welcome. I hope you are having a great day. In this video, we'll understand what is data distribution shift or in simple terms, the distribution of the data, the input data that we use to train our machine learning models that changes over a period of time and what is their impact on them on the machine learning models. What are their types? Why do they happen? And how to handle them? We'll understand all of it in this video. Let's begin. So first of all, what is distribution shift or that is the distribution means the probability distribution of the data. Probability distribution of the data that we're talking about that shifts. Okay. Earlier say for example this was gshian. So it has shifted the mean has shifted or the distribution itself has shifted that or the data drift. The data has drifted from where it started originally. Okay. So the data a model works with changes over time which causes the model's predictions to become less accurate as time passes. So the data on which the model was trained and over time it has changed many things have changed since the model was trained and this makes the predictions less accurate. Okay. And what are the types in which this distribution can happen? Let's see them. So the first one is called okay. So let's before that let's revise the the base theorem quickly. This is the probability of Y. Probability of Y given X. So probability of the output given my input. This is given by probability of X given Y. This is the likelihood term. Likelihood of seeing the data and this is your prior prior probability of Y without any data that we have seen and this evidence that the data brings. Here's a quick region of of your base theorem. Okay. X and X is the input and Y is the output. This is what is our base theorem. So these terms I will use in this uh video. So that's why I'm giving a quick revision of base theorem. So the coariate shift where the probability of X the inputs probability distribution has only changed. So means that the input distribution seen during training is different from the distribution seen during inference. So over a period of time my input has only changed but the model remains same. the model on which I I have trained using this input. So initially I had some input I trained my model but over a period of time the model was trained and it remained same but my data input has changed. So for example I'm training a self-driving car which was trained on a very bright sunny day but and the model was trained and then the winter came now I'm testing it during a foggy winter. So earier it was very bright all the colors were so separated there were different shades of colors but now it is like all dull grayish everything is dull and gray so the input has changed so the model will not perform so well now obviously okay so that's what it means that the data distribution earlier the colors were different there were a lot of shades of colors that the model learned to make out what is the difference between a person coming in or a car moving and now it is everything is dull so it can't even make out the pixels correctly everything be like dull grayish shade that's what it means so this is the representation in the during the turning phase I had a distribution of the input that I have very bright colors a lot of white spectrum of colors I saw and during a foggy so the distribution itself of the input has changed this is a new distribution so the model will not work well okay so this is the problem next so the label shift or pri prior probability shift label we give label to the output right so the output has labels or the prior the prior distribution in the base we tell prior to the P of Y right this was the P of Y so the P of Y that was that is our prior that has changed over time the labels have shifted H that's what it means the output distribution changed the output the input not the input this time but this time the output distribution has only changed what does this mean for but for a given input the input distribution remains same but for the given input the input distribution remains same that is a condition probability for a given output the input distribution remains same that is the output changes but probability of X given Y the probability of the input given that output remains constant so I'll give you an example then to make sense so for example you have flu detection model whether a person has a flu or not flu or not we try we have trained that model and in summer only 1% it was trained during summer so only 1% of the people had flu so in my probability of y was 1%. But in winter my probability of Y changed to 40%. Because probability of flu it spikes during winter and the model is used during winter. Now so what has changed? The prior probability of flu has changed from 1% to 40%. Prior probability means without any anything. So without even seeing the data this is the probability that a person will have a flu or not. That is a prior probability. Okay. But the symptoms for a person to have a have flu remain same. symptom is say for example if I have fever so the if I have fever then what is the given that I have a fever so what is the probability that I'll given that that I have flu what is the probability that I will have fever so in summer okay the summer if my this probability this is this is the output the summer this will be very low say for example there will be 10% chances that you will have flu okay if you calculate everything x x given y but in winter this value will become the P of X given Y probability of seeing the symptom of fever. Okay. And I can say that you have flu this will increase to a very big number 80%. In summer you can have fever due to many other reasons dehydration or sunstroke or something like that. Right? So you can have many other reasons but in winter that you have got flu that probability increases. That's what it means that probability of seeing the symptom of fever given you have flu. So that was 10% in summer and in winter it maintained to 80% maybe. So that's what it means that this symptoms remain this this had to be calibrated since this remained same. Now the model will not behave well even during win even during winter it will give lower than expected results because this has not changed. The output label given the input that has to be corrected with respect to the change that we have seen over the season that has not been done and because of this the model will not predict well. It will give in winter it will give lower than what is the expected output for a person having flu. Okay. So that's what is this. So this is what it it is giving the low flow rate in summer but in winter you will have high flow rate. But the symptoms remain constant that whether the person is having fever or not etc infection or not. But since the number of flu cases is high so we have not taken into account the change in the symptoms with respect to the output itself. Okay. And that's why the model will not perform accurately and it will give lower than the expected accuracy which was very good in summer. I hope this is clear. And the last one that is concept drift or posterior posterior is your final model or the concept itself that is P of Y given X. This is your final model. This has changed. The relationship between input and output has only changed. So the model that we trained it learned some pattern between X and Y. But the whole idea of that modeling of that pattern has only changed over period of time. So now this model will not give accurate results that has become kind of redundant now. Okay. If the concept drift are cyclic or seasonal in nature. For example, so normal spending behavior in 2019 became abnormal during the 2020. Okay. COVID lockdowns means for example if um in in during COVID so the during normal times the spend on medicine would be very low right this will very low but suddenly there will be this was normal spending it was not much low people didn't spend so much on medicines and sanitizers and etc etc so many but during covid time the spend on medicine will become very high and this is abnormal spending which was normal during normal times only during pre-COVID. So this is what it means the concept of normal spending has only changed the model itself has to be changed now and this is your concept drift or posterior shift. I hope this is clear. So these are the kinds of data distributions that we have seen that how it shifts how to handle them. So what are how to detect whether the distribution has shifted or not over a period of time then there are certain ways to check whether the probability distribution the input that has come in all the all will be a distribution rate so whether they have shifted over a period of time or not for them there are some test which is like case test or cologor smov test this is these are like Russian mathematicians this is a nonparametric test it doesn't assume any thing about the underlying data and it checks if two samples come from the same distribution or by looking at the maximum distance between the CDFs. The CDF is nothing but cumulative distribution function. It is a sum or integration of PDFs. Okay. And there is there is something about population stability index. So it has it qualifies how much of variable has shifted between two snapshots. So how much is the change in the variable between two snapshots. So we have one data and then the next data how much of that has shifted over a period of time. So that is given by this formula. actual percentage minus expected by log of actual percent by expected percent. And if this PSA is greater than 0.25 means you need to act something has changed you need to compensate for that. So either of all the three data shifts we understood earlier right? So either of them have happened and the other one similar to K test is Jensen Shannon divergence. So this is also used to measure the difference between the pro probability distributions two distributions. So it is a smooth version of TL divergence. Okay, it measures similarity between two probability distributions. So case test it is nonparametric. Okay, it uses CDFs and both basically they're measuring the distance difference between two distributions itself probability distributions. Okay, and similarly JS divergence also or we can use scale divergence. It is a smooth version. Okay, so this is how we can detect whether my data has shifted over a period of time or not and we should do this periodically. So that's all for the mean what are data drift or data shift and then how to check whether my data shifted over period of time or not and then to compensate for that we need to data will drift or it will shift. So we need to keep on retraining my machine learning model so that it can compensate for the shift in the data. And what are the ways to retrain them that we'll see in the next video. So thanks for watching this video. Have a great day ahead and bye for now.