How to tackle Overfitting?#machinelearning #overfitting

Key Takeaways

part of data science team at analytics video for those who have joined us for the first time let me give you a brief introduction to the data our sessions the data is a series of webinars conducted by analytics video and led by top industry experts it is the fun way to understand concepts of data science and from the leader players of data technology and as the name suggests it's one hour dedicated to data we are hopeful for that these sessions are going to be a great source of enrichment and value added to our community members now on to our session today which is how to tackle ovd overfitting is a serious issue in machine learning world where a model fits very well in the training data but the performance deteriorates in the test date the session will get around different methods to tackle over fitting in this data the speaker will cover how to reduce overfitting from the data preparation stage and what are the different things to look after a while selecting cohorts then what are the different tips and tricks that can be followed to tackle overfitting in the model building stage like use of regularization covariate shift analysis model and sampling Etc I hope you are excited to attend this data with us before we kick things off I hand it over to us which speaker a quick recap or books housekeeping items so we are recording the session and the recording will be available on YouTube channel please use the qas section for asking any questions you might have during the session and we will do our best to answer them as a data or Progressive towards them lastly we will share a feedback fold towards them you are all requested to kindly fill that up before leaving the session now on to our speaker in this session of data we have subodhi Mukherjee with us subodip is a data science professional with seven plus works work here experience currently working in Amazon as a data scientist for prior to Amazon he has worked in City ITC infotech and Rainman Consulting he has cross-domain experience and experience in fmg CG retail HR reliability spaces and academic voice he has done Masters in statistics from Kolkata University in Kolkata foreign can you guys hear me please let me know in the chat section okay gradual uh can you please uh enable my screen sharing option I have made it a co-host so you can share okay yeah I can hello everyone so welcome to the session and today uh we are going to dig deep on how to tackle overfitting right so okay there are some Bengali comments as well that's that's fine but I want like if you can comment in English so that other can also get to know Okay so yeah so today so as prajwal mentioned that uh overfitting is yeah we can Avail then we can connect later I think you guys can follow in my uh LinkedIn to get about more of your personal things okay so that is fine but I think we can focus because since we have only one hour right we can focus on the topic yeah so this is uh so overfitting is a very serious issue like a lot of the modelers or data scientists who fit different predictive models in their career right if when they fit they fit the model on our data very well but over a course of time in the when they test out their model right they see that the model is not holding what it leads to is reduction in sales of the company it directly may impact the profit and loss of the company and uh the entire strategy sometimes also gets screwed okay so and this is not uh maybe due to sometimes this is not maybe due to the bad quality of the data a lot of data scientists usually call them as the data is not good the data is not of a good quality but that is not the case always so if you take some proper measures while creating uh the model right you can actually reduce the overfitting by huge margin okay and today in the next one are we are we will go we are going to go over that and uh we will see that different techniques and even from before A modeling standpoint means before you start off with your predictive model you will uh know the techniques that how to prevent overfeeding at that stage as well but we will go into more details okay into that so yeah so before we go so this is for the Layman people as well who don't know it has do who does not have any any exposure to predictive models so for them this slide is dedicated to them and also to the others that uh what because these terms I will be going to use a lot in the presentation going forward so please uh so for that I just want to know that what these terms I mean when I will be using this so the first one is called predictive model okay so the first one is the predictive model the predictive model is nothing but it is here you try to find out the relationship between a factor that can be a sales of next year or mail can be the weather the customer is going to default on a loan right based on their previous activity right and after you build a relationship you will try to predict that okay out of my customers who is going to present customers and based on their present attributes right who is going to default on a loan okay or given my present this year's sales what will be the next year sales okay you want to predict and your business will build strategy based on that okay this percent is going to do a default in your loan so I think we can do more checks on him right before giving him the loan or if my sales is going to be this okay so then uh where are the my weak spots which are my strong spots right so these are the different things that that you can get from a predictive model so it will first establish a relationship between what you know and uh a variable of interest which uh which you want to know in future okay so it is just like uh a thing so I don't know how many of you are conversant with astrology I don't know if a lot of people don't believe it as well so what astrologer when you go to an astrologer what he does is first he will look at your hand or maybe on horoscope and he will try to find out that using that data he will try to find out what is going to happen to in future right so using your hands data or your date of birth so that's that will be his input and he will try to Output that what will happen to you in future same thing it can happen in business but here we uh we will more uh rely on the science part right and um so there is no Taboo in it and we will try to establish this relationship mathematical so that is first which is a predictive model the second important thing that is uh that the terminology we will use is dependent variable so the dependent variable this is actually the variable of interest so in case of suppose sales right so the sales for next year uh so this can be uh the sales of next year the variable that you want to predict in your future so this is a event that you want to play in your future so that this can be sales or whether a person is defaulting in your loans so that can so these can be your dependent variable so this is the most important variable which you will use to predict the next part is independent variable so these are the known factors maybe I know your gender I know your age okay I know your previous transaction be aware and based on that I can I I will use this variable to establish my relationship to get uh your future sales predicted or I will use your how many credit cards you have currently okay or how many times you have defaulted in the past right I will use those important variables to predict that whether you are going to uh be defaulting in your loan in future or not so that is called independent variable the next important so what is observation window so this is very important this is the time frame at which you will be observing your independent variables okay so sometimes it depends on your business and I will come to it more in details in the later a few later slides that it can be first uh it can be sometimes two years it can be last two minutes as well okay so suppose uh where uh suppose in my current role that we try to predict that when an equip when a machine is going to break down okay so what they are like one week to last three days data that also becomes important so that if I am running my machine more in the last three days right so my there is a higher chance of breaking down a vis-a-visa for the machines where I'm not running it very high in the last three days so it depends so that is the period that we you use to define the independent variables like your customer Behavior like your previous sales like I can take last one month last two months last one year last two years so this will be my observation window in this time frame again I'm saying in this time frame we will try to observe that our independent variables the next one is a performance window so this is the time frame when we want to see that our when we want to measure our dependent variable so CLC next one year in that case if I am in today right I will take the next one years data as my okay so maybe in last year uh of September I if I build my model based on that cohort right I will take data till this year so that is my one next one year data to predict the sales for sometimes um due to sub Banks usually run predict the faults or uh defaults right or in your loan or a credit card what they used to do maybe they say sometimes they have a 90 days um desperate window so it is called 90 dpd Windows okay so greater than 90 days if he is defaulting continuously for three months then we will call in them a defaulter so and our model will also try to predict in the next three months how many people is going to default so that is called performance window the time window that we uh that we take to Define my dependent variable the next important thing is screen data okay so after you build your uh so this is the data on which you will build your model on the data okay so this is the exact data on which you will build your model and there is uh one data which is a test data which can be from inside your print data a part of our Trend data that also suppose you have a date of 100 rows you can use 80 rows of them for uh your training and 20 rows of them can be for your testing and uh that can be also from some another time frame as well another 100 rows coming from some other another time frame that you can use as well as your test data okay so these are mostly the different terms that we will going to use we will deep deep into test data in the next few slides okay so please let me know guys in the comment section that if you have any uh questions okay okay observation window and performance window right validation data I am coming uh now right uh yes independent and uh okay independent variable Observer performance window and observation window I guess right these are the different thing that is coming okay yes I am coming to this uh okay yes so observation window first of all it is the window so it's for a modeling window right so again I am if I go to your uh past astrologer thing right so Australia needs some data from your past to validate it in future right so observation window is the time frame where you actually create I will come to these questions of L one L2 and all right so at the end of the session you will get all the clarity okay no worries on that okay so for observation window it is actually the time frame to depend your independent variable so suppose you are you have to build a model where you want to predict the sales of next year based on previous year sales okay so you can use previous sales as an input to a machine and it will throw out an output think of a coffee machine okay so coffee machine always what it gives uh it will input take the input of a coffee beans and give you last day or cappuccino right so it is the model predictive models are like that so it will take the input of some inputs and it will give some output and there's an input should be relevant first of all okay so the independent variable uh so the independent variables are those input variables that you put to get the next year sales so it can be last previously a sale that you can put in and it will put out the next year sales that is one independent uh observation window is actually the window in which you will Define your uh Define your input variables so you will obviously take a time right so you can you can take like last one month sales you can take last two months sales you can take last three months since and all is it depending on the effect and yes the independent variable is the cost dependent variable is the effect so independent variable is the input uh dependent so here you have given it uh reverse so dependent is the output yeah an independent I think you have given both as dependent but independent is the input and you will establish a causality relationship and then you will just predict so the observation window it will be the time frame when you depend when you where you define your independent variables so how much sales I should take to prevent to predict last next one yourself it can be one month it can be one year it can be two years it can be three years so this time frame is your uh independent uh uh observation Window performance window is how much you want to predict so what may be a time frame if we are dependent variable so it can be three months and it comes from business usually so it can be three months from now it can be four months for now it can be one year from now so that is your performance window okay and uh the other thing is I guess Trend data and test data so suppose that is the entire topic that we are going to discuss so suppose you build a relationship on on your whole data set okay you define the your relationship on your whole data set okay so now uh what will happen that you you are thinking that okay I have built a very good relationship and I am predicting very well okay yes observation window is the past time frame yes and the performance window is the future time frame okay from in which you want to uh predict build a model can you guarantee me that tomorrow if a new data comes in your model will hold your model will give as good as performance with respect to what uh you are getting in the past no right so that's what that you need another data set which you will use to validate the performance of your data obviously trained data it will never overfit from the test data it will try to overfit and I will come to the later interrupt so the data are variables okay which are the different cases got it yes I guess I have answered the questions so I can move to the next part right okay so before uh going into the uh what is overfitting right we need to analyze what is a bias and what is the variance component I think someone has some questions on uh what is bias and what is variance okay so what is bias right so suppose you have built a model okay and so this is like your yeah so suppose this is like your actual data okay so these dots are your actual data it can so this is uh so these are the case of your actual data right and uh here you have built a relationship which looks like this so here you can see that this difference right between your actual and the build relationship that you have built using the uh the independent variables there's a gap between this okay so this Gap is called so this Gap is called bus on the contrary uh here you see that in this model right here uh this line is actually estimating all the points right so here we can say the bias is very extremely low okay because it is estimating all the points it is touching all the points right so this is called map actually mean square error is also something but it is not exact uh thing uh it is actually a square of this uh of these gaps but maybe something which is uh which is more important you can get it as a percentage and all so the bias is nothing but it is the difference between your actual and your uh uh and your predicted model right yes the second graph is over fitting I will come to that okay so here so in the first case it is a lower bias so it's the high bias model right and the second case it is a low bias model where the actual is that so that the prediction is actually touching all the points means actually you have got 100 accurate model okay and it's a more a style case but one thing is evident that idealism never exists okay so if you see here like this is ideal it also brings a suspicion to your mind that okay if this holds in this data will it hold in any other data set right this relationship is not looking too good two good things are always not very important are not actual real right you will see two good things that never um real Okay so you can't take this too good thing over here so but so this is just like a two good thing right so now it will see that okay here you see if the relationship looks like this right it is look like this but here using this model is also giving you this this value the lines is not correctly predicting all these points very correctly okay over here so this is like in any other data when you just want to present your model the amount of deviation it has right so that is called variance okay so variance refers to the changes in the model when using different portions of the training set even different data set right so when you uh yes so exactly so if the model actually creates it right it means if the model is over is the model over learns right if I want to overlast that will it will happen that on a newer knowledge when you will put it on a newer data set a new knowledge comes in it will just it will know nothing it is just like memorizing your uh when you have given your exams right it is just like memorizing your uh things too much which in Hindi it is called rectumana right so if you if you are focusing on one thing and just learning it learning it learning it I if I ask you another question you just know I don't know what it is so that is that's it if I ask you those questions right uh that what you have uh learned by heart right where you have done that rakta right but when I give you that exam change when I in the exam I just give you some different set of questions you just fail so that is variance and that is called over fitting so when you are in your own domain you are fine when you are just playing very good like it happens sometimes with India right India Works uh when they play on Indian soil they are like super and when when they just move to a foreign country everything goes away the scores goes down right so that is the thing so if you practice more in your own soil right you will have a habit and you might play very well right so that is the thing okay so but when you when I'm giving you a new situation you are not being able to copy it up so your model should be like that it might not be perfect but it should cope any situation that is the ideal case of a model okay I will come to the next one okay so this is the very important diagram that we have okay uh so this is the case where you have so if it's a bullseye right it's in the Bulls I like here all your dots are in one region so it is low bias and low variance this is ideal and this is something you need in your model okay this is a model requirement so this is ideal you need this right what are the other cases that may happen right so here it is high bias right and low variance okay so here the uh here all the dots are in same place but the bias is high right so this happens for a case like simple linear regression okay okay here this is actually called under fitting so this is called under fitting where in any data you will get same performance but a performance is low it is with some of the students right every year they get around 40 to 50 marks right but every year they get it right that around 41 40 50 is lower in terms of 100 if the 100 is the full marks so this is under right it is under performance so this is under fitting okay and okay the next case is where it is low bias and high variance here uh the everything is near the bull side right but low buyers and high variance but all the all the darts are actually spread across all the darts are spread across so this is a case of overfitting okay this is the case of water fitting so this is just like the India team who is uh playing very well in Indian conditions and when they're going to any other Australia or some player they are just true so so this is something that uh we are trying to do like in your own data it is fitting very well you are getting 99 accuracy in the other data right when you just want to stop uh put the model you are getting 20 so that we are trying to work uh reduce because that is not if it if a model shows this then that model is not a model at all okay if it does not hold over time and this is uh where actually nothing is going good so this is actually I will use the term crap this is like it's not a model at all you will definitely don't want this model okay so this is something that you never want this is a reason is that you are a not good modeler okay that you should not get this okay so these are the four things that you so this is the ideal this is this play State you want to attain this state you want to reduce okay under fitting reduction is I think it will take another session but there are a lot of models like execution or machine learning models so that is yeah so high bias High variance it means that your model is either not been able to predict anything or as well as that your model is not holding in any other data set as well so this is crap this is nothing no model at all so this you would never you you don't need it so you if if this is the case then there is a serious problem either as a modeler in you or in your data right you have to go to the business and talk to them that okay this is not possible means this modeling is not possible but ideally this case happens very Less in my I have seven years of my career I haven't seen this either this happens or this happens or this happens yes okay so next I will come to the how to control the overfitting and this is the overfitting case right and we want to reduce this force that you you always you should get a good model and your good model should hold everywhere so that from here you can go to this state which is ideal speed which is the best tip okay so here uh the different phases of controlling over fitting okay so what are the different stages it is the first one is cohort selection so you can so by judicious use of selecting your cohort you will be able to produce overfitting and how I will show it later the next one is data and feature selection So based on your data and using some judicious methods to select your feature you will be able to reduce overfitting as well these are pre-modeling techniques you don't need any model over there the next one is write model selection okay that you have to select a right model and in the modeling stages how we will be able to reduce our fitting so that will be the idea so I will come to the next part which is like chord selection okay so next part is the cohort selection so cohort selection there are two three points that you need to know okay at every step so the first point which is that whenever you are selecting in your cohort you should if there your business has seasonality inherent to it okay this one chord selection data and feature selection and the third one is write model selection okay so the first one is part selection so for the quad selection suppose your business regularization is there dude no worries I will cover okay so um what selection is something which you usually do while selecting your model right which time frame which your performance and window and observation windows so suppose your business already has seasonality because I worked in a Fashion retail for a lot of days okay around six months so I have seen that their summer preferences of customer is very different to your winter preferences of the customer okay so suppose you build a model on your summer data okay or hold it as well and want to use the same model on your winter data it will not hold definitely if there will be overfitting okay so if you want to build a model on your summer data your test data should also be summer right and it means your training data should also be summer and if you want to build a model for Winters your training data should also be winter got it so this is the way the first thing is you if you have to first gear if your data has seasonality or not so that is the risk the risk will be that uh if you use one season's data for model building and use another citizens data for your uh summer right so per winter so in that case your model will not perform them another thing you can do okay I can take full years data but in that data as well it might dilute both in summer both in winter so it's right not over fit a lot but under fitting will be definitely there okay because we are not taking the preferences of summer and winter separately so you should always take your preference of summer and winter separately so what not to do so I'm coming so build one year mod one model taking one season of data and deploy it for full year that is not you should not do and build one more build model one season of data and validate on the other right that that is what the case I have told you build the model on summer validate it on winter never it will not hold so what you should do so build first of all build a model by appending one cohort for each season so if you see more that there is a seasonal impact so if it's a happierly it can be quarterly so take one part from each quarter so take one data frame from each quarter one month from each quarter build your observational performance window according to it and combine them just append them and then build your one that can be one choice the other choice can be build your one model for uh each season right so for each season if you have four season we separate four models and deploy it by running whenever so you have a time frame attached to it so if condition right so if this time frame is there run this model score with this model if this condition frame is there if it's in October run it with the winter one if it's in July run it with the summer one that kind of thing so you can do this thing so this will be uh and while developing as well build fold models in that case or build a fun cohort of data by appending four cohorts for taking one from each season that you can do this will remove your overfitting by huge margin the second one right second one is also like uh it's a subset of the previous one like that so if sometimes for Indian cases more like we want to build models for festivals that in Diwali big billion day lecturing data model will overfitting I will come okay so so when Christopher Christmas for Diwali for Ramadan right make sure that you have chosen the right cohort suppose you have built a model on uh whole year and we're using that same for Ramadan how it will hold how it will hold for big billion days how it will offer the SPL period Christmas period right it will not because a lot of people who have not done any uh any uh other thing right who have not done sales in the whole year they might now come and do the sales funniest festivals right so this will definitely create overfitting right so cohort is date and yes yes recording yeah cohort it means exactly data and time stamp is which month you want to take for your model building that is it okay so that is your quad and data and ramp is important if it depends on fourth season you should take uh the date of Four Seasons separately four different seasons date in any month you can do it and that depends on your business case okay and you can take any date it may be differed within the month as well so if that is the case you can then take three months post month and something like that so that date and time frame is very important timestamp okay the other one is uh like what we should do like so far Ramadan or Christmas or Diwali data if you are using build a model in previous years data on the same time and validate on the data of the same time as well so your test data should also come from the same time frame not by any other and deploy that model in that time frame that's okay in this month I will run this model and on my original model I will not run right so that you should do okay one more thing is don't go too far from the original time frame and the code should be closer to the current time right so here we are taking suppose data of like last 20 years to predict next year's sales don't do it okay don't do it the other one is uh that always uh take right don't go beyond two years first of all that's the thumb Rule and if you are taking like always do a yovi or month on month analysis okay covering the same tension this is no so for that I am saying that okay for Ramadan or so take the ramadan's data only for last year's ramadan's data and build your model on that and deploy it on that so last year's Ramadan it was you can take how much sales has been done in Ramadan after Ramadan maybe one month after uh in the Ramadan time period of one month and take maybe that their previous two uh to three uh months data last year in last one year how many how much data you got so but build it your cohort should be around Ramadan don't take any cohort out of Ramadan got it don't take any cohort out of Ramadan take last year's Ramadan data and based on that build your model and deploy it in that fashion don't take the full year data and deployed when the Ramadan time frame as well the builders separates uh model on this cohort okay and the last Point what we have seen that okay don't go too far from the original time frame and the cohort should be closer to the current time frame don't take 20 years later do a month on month yui analysis here on your analysis that how much your features are changing if the feature has changed a lot right suppose I have taken pre-covered data now to build something on postcode with data it will never hold it might not hold in my cases right so like in city we have seen that the sales have moved from maybe from offline to online so we have seen a lot of companies we have seen Amazon also had a huge uh chunk of sales in the uh covet so after so in the covet pre-covet data if I use in the post covet uh for to predict something in the postcode scenario it might not also you have to be judicious to take the don't go too far into your thoughts okay I will now go to the next part and this is a very important part okay there will be very uh yes geographically as well like if you want to do do an analysis geographical analysis okay so there is a separate uh type of model building which is called panel data okay panel data model building which is if you have time as well you can incorporate that otherwise geography see which geographies are actually holding right your uh have similar kind of data do an analysis then build models for each geography types okay do this analysis first and then uh build your model on that don't build one model for all the geographies and deployed it will be off for some of the geographies so the next part is uh data and feature selection I'm sorry I have not been able to take a lot of questions I will take the questions at the end of the session because it's only one hour and I have to cover a lot of the topics okay so the first of all data is the next part is data and feature selection okay the first thing is uh definitely the how your test data should be so first thing is in time validation in time validation is actually um you should take while building your model right uh uh it should be coming from your own data so suppose you have taken the data of whatever uh training data you have taken for the whole year maybe you have taken a data and keep 80 of it as uh your train and 20 of it as your test now if suppose you have a very small data set okay so then there is something which is called careful cross validation which we will cover uh in the next part uh do that okay that will help you to get that whether your model will hold in in another part of the same data or not and always keep these two data sets that is your the data part or the sample where you will train and the sample where you will test mutually exclusive don't take overlapping observations or overlapping numbers over there always keep both this time frame both these frames separate and mutually exclusive means one should not overlap then the other okay the if you have suppose you have a city or a cross section like uh What uh someone mentioned right if you have a cross-section like City in your data right or a country take a stratified sample so that from each sample in your trade and your test data same proportion of uh observations like suppose in your data in your trained data you have maybe Kolkata Mumbai and Delhi and your test data you have only Mumbai and Delhi that is also not right reverse is also not right you have Mumbai and Delhi in your test data you have you don't have a Kolkata right maybe you have uh in your train data you don't have Kolkata India test data you have Kolkata okay that is also not correct don't do it because in that way your mod data will uh yeah your data will not your model will not see the whole data set or your model will not be able to validate the whole data set right so all this guy is stratified sampling how you should do it I have given a code snap snippet so there is a variable called stratified bar so in this 10 test play so this is a 70 sample I have given dedicated for uh training 30 sample I've dedicated for testing and stratified what will help me to stratify the same thing so always do stratification out of time validation this is just like um you also have to go like one cohort or two cohorts back you should also do get one separate code of data like last one years of data maybe you can take this on this CRS of data you have built your model take last one of your data last years of data to validate it as well along with the trend test validation to see that that whether your model is time I time tested or not okay now if uh in ideal cases two uh cohorts should be selected one forward means from the time so suppose uh this year you are building your data build it on January's data test it test it on maybe um May or June's data once if if the data does not have any seasonality okay and also test it on maybe one year back later so always you should do four so that you will get to know that okay why whether your model will be uh holding going forward as well as going backward or not don't uh rely on one sample try to put two samples if you have that uh thing hold out samples hold up samples is nothing but where you have your training data you can divide it into small parts one two and three so here you can train here you can use your hyper parameters to just test it out okay and here you can uh so this is your holdout sample okay this is called holdout sample this is this you can take into consideration while building your model and here you can test your model okay so this here you will just fit your model right and unless you are fitting you are satisfied with the holdout sample you will not touch this test data unless so when you are satisfied with the performance in the holdout sample then only you should go into the test data so that uh that is another approach that you guys can do this is called holdout sample approach so this is a sample code what I have given here we can do it twice to get the holdout sample no this is not a full strategy fourth strategy is something I'm coming in last next five minutes okay the next one is this is very important this is called character stability index CSI okay this is a very important metric which will help you to get that in case of a data this will give you a threshold okay uh I may create that whether your variables uh are not consistent in your twin and test data okay so this will help you in that so how to do this like uh the steps to do the same is given below I've given like for each one step I have given uh one line okay take one independent variable so the variable you want to see that whether it is going to be overfitting your model or not okay so take one so you have to do it for all the variables in your data set which you want to fit it in your model so first take out the independent variable from your print data take out maybe one divide the variable into 10 bills okay that is deciles means 10 decides 10 percentile cards so there is a python cue cut there is a function which will help you to do that in the pandas Library which will help you to divide your variable into 10 parts if the variable does not have 10 unique values right or we can't do it you can divide in four or five parts as well that is fine now for that variable take and this is we are doing from the training data training sample okay now calculate for each decile calculate the minimum and maximum of it okay so I'll store it in a separate data set the next steps will be use this minimum maximum thresholds on your test data and again create the deciles on those variables so suppose the same variable it can be like age age in your print data you created this 10 uh decides know the cutoffs that okay these are my cutoffs put the same cutoffs and create the same 10 decals in your test data for the same variable H okay now for them calculate this metric so for each now since you have decels right you can just get the how much percentage of observations are there in each design okay so this number percentage of observations in your training sample and percentage observation here in your testing sample and if you create and this formula will create the metric for PSI okay and you have to finally take the mean of it across all the decides to finally get the CSI value okay now if this CSI values is less than 0.1 you do not have to see and do anything if it is greater than 0.1 to 0.2 so if it is between 10 and 20 percent then you can what you can do that okay you might have to look at the variable some more and you can do it and how I will show you and if the CSI is greater than 20 then then this variable is bound to our fit okay then this variable is bound to overfit and now finally you have to repeat the steps for all the variables to get the CSI how it will look it will look like this so suppose this is the variable these are the deciles okay here this is the training percentage so in greater than 720 there are 12 observations in test it has 11 observations and the CSI will look like this okay so here it is like 0.18 within 0.1 within 10 so it is fine so uh here that so this variable not uh this variable will not overfit in our data has a very less chances of work what if CSI fails so definitely you have to see with using other correlation or information values right uh that whether it's an important variable to your model or not or from a business standpoint if it is not definitely drop the variable don't use the variable going forward okay don't use the variable going forward if the variable has some predictive power right and still it is you are seeing the CSI is not holding right then use your outlier treatments I know I don't know whether you guys know about auto advance it is like fluorine capping converting the variables into categorical groups like bin converting that numeric variable into categorical bins or combine the bins see which area which bin actually the CSI is not holding try to combine them or try to make it as a category and use it as a categorical variable so in that way you will not lose your model's efficacy or buy a huge margin you will lose some but not by a huge margin but uh you can use the variable but yes ability will also fold so you need to bin it or tweet it in a better way is this similar to Mi index Matrix method I don't know what is MI index might be you can go through Google's on this okay the next one this is also very important manual information yeah might be but no I think manual information I don't think so it is a bit Mutual information I don't think so I think CSI is a character stability index just follow this this is about the stability of the data not uh how much I Think Mutual information can be due to that how much information it is the proxy of correlation metric that how much it can be helpful for prediction but PSI is a metric which will help you for uh stability of the data okay the next one is covariate shift analysis this is also you guys have seen in the uh thing on in the AV page the okay the uh so in order to and this is specifically uh useful for our uh machine learning model okay so here the first step is take out two equals equal size samples from the 20 tests so suppose you have a data of maybe one one million from test maybe you can take a 50k from uh train okay whatever sample you have you might have a sample size of maybe one lakh of a test that is fine random sampling okay now in the second step for the test so for the trend data create a tag one for the test data Creator tag zero okay and append the two data sets Okay the third step will be run a machine learning model preferably a random first model usually works well right with respect to this one zero tag this will be your dependent variable in this case drop your original dependent variable okay just this is for your overfitting and all your independent variables put it as we are independent variable for this model okay same sample size should be there one zero tagging should be there for train it is one for test it is zero and build a model around it now take the variable importance of the variables see if some of the variable is relatively very high in terms of like 50 or 60 right it is contributing uh to the variable importance variable importance will tell you that what is the order of the variables into segregating these two values right so take the variable importance chart sort it by higher importance to lower importance values if a variable is standing out over there that is causing overfitting reason is what you have done over here like you have created for your trained data and for your test data to taggings if a variable is actually being able to segregate this one and zero die so then this variable is actually different for your train data and your test data okay got it so this is actually the variable as creating a segregation between your Trend it and test data the tag is actually defining that which one is a plane and which one is your test so it will just create the segregation between them okay if that is happening your model is actually overfitting okay and uh you can now uh check that variable of the CSI of that variable now you have not have to run as well for all the variable methods right and you can check the CSI do the treatment your model is done this is a very good method of creating the um of segregating uh we are controlling over fitting the next one is model selection now I think a lot of people will get bit calm right because I will come to their favorite topic okay regularization parameter okay regularization parameter uh works for some classical regression models right uh for linear models as well as for logistic regression models like linear regression and law distributation models so you have an extra parameter and it also has an extra parameter for XZ boost okay extreme boost I don't know whether you have heard the name of this a gradient boost extreme gradient boosting method it also have the pan so what it is does right so this is an underfitting scenario this is an overfitting scenario it will so what happens that if you put lots of variables in the models see that I think I have it in as an um as an uh appendix but I think we need a different session for that okay to explain XP boost if you want you can talk to the AV guys okay I can help you to understand the mathematics of X boost okay a shot I will explain try to explain here okay so what if you push a lot of the variables which might not add a lot of value in the model but it will still Force fit it right what will happen that it will definitely the model is like that it will try to utilize the variable and try to get the variables but whenever and to try to get the relationship but whenever you use that variable right in order to predict because that variable does not have lot of value in the model right so over there it will work so it will over pretty it will not have to be available forcefully imposing into your model okay so what uh so how regularization works okay so so this is a regression estimate this is your data this is your dependent variable this is your sets of your independent variable you will put uh this so on your regression estimates you will put one uh put one condition okay and here so this is called cost function and uh so there are so all so while a linear irrigation fits it will always try to minimize this whole value okay it will try to minimize this whole value because this is the proxy of the error that you are doing okay so it will try to minimize the whole value now and this is a higher parameter okay that you can adjust So based on this it will try to squeeze the values of the variables which is not adding any value anywhere any value okay which is not adding any value which is causing this bias which is causing this bias it will not it will try to squeeze the value of the variables okay of those so suppose you have age you have uh race you have maybe six you have uh so suppose this text variables this is not does not add any value right and it has a maybe a beta okay so it has a coefficient of uh regression so what this will try to do that if it does not have any value right it will try to make its value as 0. so for this one is called uh lasso regression and this one is called Ridge regression okay you guys can definitely get it in SQL they have implemented it okay so the difference between these two is for lasso the values of the weights can be zero and for Rich it will never be zero but it will be very short and it will be close to zero where the variable is not fitting now the important thing is exit boost okay so this executes the parameter actually the parametric form looks like this so this is your first cut I mean after uh the first iteration it will always put your mean or it's a random cut so that is the thing and these are your values that you are getting at each iteration with each decision tree so over here also you have this parameter of uh Lambda which is like um which will reduce this regulation it works similar and it is that it will just penalize your Leaf okay the values that is there in every leaf it will just try to penalize that leaf a detailed calculation and I can show you but I think today it is very difficult but it will definitely try to penalize the leaf okay thus regulate will help you to prune the leaf okay and in that way that if the leaf is of not used for the decision tree for a step it will not use that okay okay when to use lasso when to use Rich so suppose you want that your parameter uh that you don't want the variables right at all right you don't want your variables at all which is causing overfitting then you you should use lasso because it will it will shrink your values to zero and uh suppose you um uh want all the variables to be there whatever is the lower because sometimes business have the interpretation they want to uh deploy suppose on digital media they want to deploy a lot of money on digital media now they want to see what is the importance of uh each variable we want all the variables whether it is small whether it is lag then you can use reach okay okay so also one thing this also eliminates uh yeah yeah definitely speech selection right so it will always remove also remove the variables it means that's like you will get zero it can be used as a speeches selection okay also these two methods has an advantage like if you know about multi-collinearity right that is like if two variables have is highly collinear means high they have high amount of Association like 90 percent if one increases the other also increases by 90 there is a chance uh both of these methods penalizes that and uh you will get one variable and it will remove the other so there is a high chance of that so that will also help device to build your model more judiciously now next one is I think this is also a hot Demand right this is like a fault validation okay so k-fold if you are suppose you don't have a very big data set um and you have a small data secondly you might not take a 80 20 validation data set over there you can't not be able to take it or you might not want to take it so what is the happens is something which is called k-fold cross validation so over there uh a data so this will be their intertraining data okay entertaining data you don't need a test data set over in time validation or test data set over here so what it will do K means that it will take out one sample of the data aside at each step okay and take the remaining some K minus 1 samples right and train your model on the remaining K minus 1 samples and test it at every step test it on the remaining sample and now the choosing of the sample is also random so first step it will choose this second it will be the next some different apart from the first one and it will go on finally the model performance will be uh based on a mean across all the folds I mean across all the folds so here but the caveat of this model is the training time is extremely high for the if your data set size is higher because here every step the sampling process will work third first step it will just take some K minus 1 yeah jackknifing is not the Jeff Nike knife thing is like you will just delete one part here delete one part here from there both sides you are just uh just removing that is called jack knife okay so here it is here it is selection of any part and you are just keeping it aside and using it for your testing set for your testing you will use the whole data set Jack lighting you will just uh remove the data set from both ends to just get rid of the outlier from both ends so that is not uh you are doing this is Crossfire this is called k-fold cross validation yes because cross validating your every step you are cross validating right so here you are just value here you are creating the model here you are cross validating no epochs is the number of iterations in the neural network cross validation is on your training data set suppose this one fold validation okay this is nothing but here it can be our 80 20 validation this is only a data set happening epox is on your neural networks if you want it is the number of iterations you are running your neural network your number of iterations you are forward and backward uh thing is happening okay I've got training is happening right so that is Epoch K first resultation can occur for any models right over here just like every step your uh okay stratified cross validation means here when you just uh do the cross validation you are selecting the samples you are using a straighter over there just like I've shown you right I have shown you guys right that how to create a stratified sample so exactly this sample when you create this sample it will be stratified over there it will be based on straight up so that the event rate is also maintained the data you have all the dimensions or you have all the cross sections in your data so that is the static uh random Forest uh you can fit for make it for anywhere random Forest one thing is there which is called OB four okay OB out of black samples okay but out of back samples does is it's a single kind of a cross validation means you can think about it Epoch is the iteration for neural networks how many forward propagation and backward propagation it will have that is epoch so in the cross validation the data is first broken into K Parts since you are just breaking the return to K uh equal parts on K minus one Parts you will just uh train the model and on the KH part you will change it and this K will also change for every iteration finally you will see that suppose in the K minus this part you are getting maybe 80 accuracy here maybe it can be 70 accuracy here it can be 75 percent accuracy so like this and you will finally take a mean of all the folds to get the final accuracy metric it is not based on one sample for normal current is validation it is based on one sample here you are taking K times it can be five-fold it can be ten fold okay uh so so based on that you are taking a mean and you are so this metric will be much robust in that case but it takes a lot of time okay early stopping criteria this is also very important and is a very good trick that you can use okay and this works for a machine learning models mostly where you are running like Freebase models or uh or a neural network where you are using epochs okay K is a hyper parameter 5 and 10 is the most uh thing okay 5 and 10 is the most thing um most used uh there are other but you have to see that where you are getting good performance that's like a hyper parameter you can just use after CV which model can be test taken to test with the holdout data out of models yes the accuracy so that is the thing rights of accuracy here it takes the mean of the accuracy okay if the accuracy varies we usually take a threshold of twenty percent the in the very in the accuracy varying okay uh you can take a threshold of twenty percent if within it if it's within 20 on the holdout sample it's fine okay if it is more then you can think about it why scoring is negative like negative absolute error so that that is I think I will answer the question later okay that is just like you convert your loss function what is repeated random 10 question so you are just taking different uh samples at each step this is similar to grid search CB and random search so CV it is CV it is cross validation grid search and random search is something on your hyper parameters that you do okay CV is common to both and CV is this CB is your cross validation strategy got it uh the other thing okay any other questions like this so CV is nothing uh yes a grid search is done on top of this yes last I think we will uh okay I think I will have one more site and then we can take all of your questions okay yeah yeah all of them can be done there's a pipeline package please go through it you will get okay fine uh the second one is the early stopping criteria early stopping criteria helps you to uh predict okay that when uh when to stop in terms of a machine learning model suppose you have a random Forest okay so suppose you're a random forest model okay so for the random forest model you have you are training it for thousand piece at each step you are you can notice that what is the performance in your train data or your test data so it starts off like this okay your train data is 50 accurate test data is fifty percent accurate chain data is 51 accurate test data is 51 accurate okay if it is a 53 accurate it is 53 accurate okay so it will go on like this is the first iteration this is your second iteration this will be a third iteration it is gone right a point of time you will see it is 80 accurate okay it is now 80 75 9 Accurate okay now it will go on increasing okay it will now 85 percent accurate it is 70 percent accurate so always while fitting your model always strength is this one means where till the point where your trained data so always test error goes like this so it will always start under fitting and it will start to try to reduce the error and after a point of time it will again increase whereas in a training error right it will always decrease okay you always try to fit it hit it then uh uh yeah see the point from where the test error actually starts to increase okay so this is called early stopping criteria so in a model there is already an oob technique that I also mentioned that also can be used it can be done manually also there is a hyper parameter where you can speed up the steps but okay after five steps if the model does not improve in the test data I will stop the fitting of the model that can also be done use in the model so in the sky cycle learn object you can do that so that can also be done okay uh so that's it and I will come to the last part so this is like model Ensemble right so on Sample it is like combining different models now think of it you have different orders which is maybe overfitting right not you combine the scores so because some uh because some model may be worth fitting for 10 observations some models will be overfitting for uh other 20 observations but you combine them right you can come to composite score right so this is combining different models so you have created different random forests take the Boost and maybe some other Model A linear regression and you can combine them okay uh so that is ensembling comma so that is called Ensemble so in a single sample right a single model using a single model how can you put it on symbols so these are called bagging and boosting so though bagging we uh usually use random forest and we know random first but that is not the thing tagging can be done on any data set backing means that you are taking uh different samples right different samples and fitting same type of classifiers may be fitting same same classifier for different samples of the data and finally you are taking a Consolidated or an average of the scores that the model is generating that you can do bagging random Forest actually is a special special kind of a bagging actually that is a wrong concept those are not same bagging you can do any models you can do it on a linear regression model as well you take different samples so that is also like a bootstrapping we used to learn it like a bootstrap you can bootstrap your data build on different samples and finally combine it random Forest is a example of a background but there are other there can be any bagging techniques that can be used backing it means that you are resampling your data and fitting at the same classifier for all the models that is called okay boosting I think here uh again you are creating weak Learners at each step and uh uh so each step is dependent on their previous step and you can reduce the overfitting over here as well to decision after Pro sampling so that's the hyper parameter right so wrote something it is 80 percent so it means that 80 sample will be taken if you have a column sample by 3 right so over there you mentioned to the point and then again eighty percent of the sample will be taken it is a hyper parameter right so you can just change it based on your data uh based on your requirement that is also hyper parameter that can be done but here we are not talking of boosting or exit boost parameter on the packing says packing is like uh here you will just create different samples of the same data right and um fit the model and finally take a Consolidated of all the model predictions so that is it Ensemble from multiple methods sampling over sampling and yes yes backing can be Rose as a column wise that will as well that can be also that can also occur you can design your own backing okay that is a sampling step Ensemble from multiple methods okay so um so but for random Forest the classifier is decision P for bagging you can take a decision key as well as any classifier or regression any model any technique you can okay that is the main difference okay so the next one is voting okay so here we are trying to learn what are The Ensemble from multiple methods right so the first one is called voting voting it means that you create different models different type of models this can be one linear regression one position one uh you can use one exit boost one you can use one random forest and finally you are combining their outputs okay so for two ways you can do it so first of all if it's a classification model means where you want to just classify your data into one or zero that's like a fraud prediction okay uh so for them uh what you can do like you can create the models like suppose random photos is predicting one uh random uh linear regression is predicting zero three models you have created so here you can take the majority of the votes okay that is one this is called heart voting and there are some where you can just get the probability predictions or for linear equation you can get maybe sales number so it can be 0.9 0.2 0.1 okay and you can now assign weights to it okay so these three weight sum should be one and it should be fractional okay and you can add the weights and finally you can create a weighted combination which will give you soft voting finally you can create a combined using a weighted combination and that is called Soft voting okay the next one is blending and stacking so these are mostly similar yes so this is mostly similar blending and stacking here what you do like uh so suppose you have created one regression model a linear regression model one uh okay maybe the reverse okay maybe you can create one uh XD boost one random Forest two models you have created okay now you use the predictions of these two models okay you use the predict uh so this is very prediction one this is very prediction two right you use the predictions of these two model and you can use one more model where you can use all the variables along with these two prediction as variables so on top of it's a two-stage model in the first stage you create maybe an execution and a random forest and take out their predictions and use it in a velocity regression model that you can also do okay so that is called stacking and blending in the difference between uh the two steps is like uh validation and the in the stacking and the other does uh normal validation right and you always have to do it on holdout sample not on your training data always you have to do it online hold up sample data okay got it I think that's it so these are the techniques I think I could cover in one hour right so now I will go to the questions if you guys have which is over fit and then we can use relevant uh techniques to overcome overfitting um ideally real life data sets are more you can go there and especially UCI machine learning library there are a lot of data sets you can see which one is there make sure please tell me ways to figure out animal detection supervised learning I think that is not a scope of this right there are a lot of ways to detect and that is that will take a different you can talk to me in LinkedIn okay all of your other questions from this you can talk to me in LinkedIn okay so you can talk to me on LinkedIn that is over sampling under sampling or by a good stratified sampling you can you will be able to overcome and uh okay okay I don't use a lot of you can use scale Divergence and all sometimes that can be also helpful I don't use I'd use these methods and that's self-sufficient okay got it okay I think there are some Q A's over here Okay so if you want a Hands-On sessions anyone I think talk to heavy guys I can show you some codes I have given and I don't know if this material is here uh you guys can contact me LinkedIn right I can help you over there as well I have given the links in this um uh PPT so that's okay great search CV okay cross validation is crosswell Regret such is when you are doing any hyper parameter tune okay right okay uh so if you are doing any hyper parameter tuning you can use different combination of hyper parameter and see where your cost validation metric is minimized or where the Gap is actually this right so that is called grid so Stevie and grid search means you will create a grid right you will create a grid of a combination of all possible combination of your parameters right and then try out and finding out the CV at every step where can we find the presentation application I think that is okay got it sort of I think I have answered your question on Twitter CV yes yes well value of hyper parameter using the rate such civilian that is using kaggle a lot okay I was also I have done some positions but that's not okay can you get the model out of series series 5 which Series 5 means it is a combination of five models you can think of five samples okay so and combination of five samples and just uh combining them okay a combination of five samples uh uh you will just repeat the model on five samples and then I'll just combine them on so it's a combination kind of a thing not on one sample you are doing it got it so every step every modeling step is done on five samples accuracy mean can be combined how the model difference can be taken out so see it is only done to validate your model it is not done to take out your model if your model is showing a good uh uh good uh k-fold CV values uh definitely take out your uh then that model is holding those hyper parameters is holding got it so CV is the validation technique it is not a modeling uh thing we are not building different models after model building same model or different samples how R square work okay that is not a part of this course but I guess I just asked her work uh when you have lot of data okay suppose what is the happen happens right if you introduce more variables right R square will go up so you paralyzed the R square right in the adjusted R square you penalize the feature by degrees of freedom or the number of features you have in your model and that will help you to uh build uh much robust so that will help you that that R square will not increase after a point of time and that will be more robust otherwise you may land up to 100 yes yes sure sure please start contact me LinkedIn that because annual rejection is a separate topic I can take a separate data actually on animal detection so so I need to know what kind of animal you use you have right this is the time series anomaly or anything else right so uh any uh it can be solved whether during your supervised learning or unsupervised learning and all those I need all of those I think you will I think that if you guys can do it the LinkedIn profile is the invitation I guess uh in the page that the invitation there uh they have the link yeah yeah so I think if uh so that is a separate uh for animal injection uh you need one R is not enough the guys any more questions because I think it's uh uh mostly I'll have to cover a lot I don't know how much you have understood if you have any doubts right you can talk to me in one-on-one in my LinkedIn right and we can talk over there as well yeah sure I will no worries