End To End Machine Learning Project Implementation With Dockers,Github Actions And Deployment

Key Takeaways

in this video we are going to start our first practical implementation already we have covered up the theoretical part of linear regression we have understood the maths in depth intuition we have understood about gradient descent we have understood about the performance matrix that we are going to use like r square and existing r square now let's go ahead and implement a project uh with the help of linear regression and we'll try to see that okay whenever we have given a data set and this is a very simple project guys just to begin with so that it will actually help you to build up some amount of confidence okay so first of all what we are going to do in this is that we are taking an amazing data set which is called as boston housing data set and the main aim of that particular data set is that we really need to predict the price of the house based on various features now we are going to see what all features are there and what all things are not there which are the important features and all we'll do lot of eda will try to come up with some conclusions and then we'll try to implement a linear regression machine learning implementation all together right so to begin with uh let me quickly make some cells so you can use shortcut like escape a so this is a jupyter notebook over here so let's begin first of all what we are going to do is that we are going to import some libraries like import pandas because pandas are super important in this import and i'm going to write each and every line of code and apparently try to explain you import numpy's np and some more libraries i want is that import matlab layer dot pi plot as plt so this is the library and for inline display of the graphs or any visualization that we do i'm also going to import matplotlib inline right so all these things are definitely there so i have imported all these things now to begin with what we are going to do is that let's load the data set okay let's load the boston house pricing data set now this data set right now if i really want to think about it it is already present in a library but we'll try to see as we go ahead we'll take up different different different problem statements and we're going to make sure that we'll understand about uh you know from different different files or different different sources we'll try to take out the data set and apply machine learning algorithm so today in this video we are going to use from scalar because scale learn also has a lot of data sets all together within themselves so i'm going to import the data set and i'm going to say import load boston okay so there is something called as load boston along with that there are different different data sets which you can definitely explore then this will be my boston underscore df so here uh let's go ahead and write boston's for df i'm just going to call this load boston and quickly print it uh first of all let's see what is the type that we are getting with respect to the boston underscore here that is nothing but scalon.utils.1 so this bunch is nothing but if i probably write just dot ease you'll be able to see that i'll be getting various keys right like data here you can see target feature names description and file name okay everything will make sense uh you know once i go ahead and see all these features okay now first of all i really want to see what is this descr this basically says that this is a description of the entire data set so let's check the and always it is a good way that you should probably write each and every comments whenever you're doing any kind of coding so let's check the description of the data set to begin with okay so here you will be able to see the description of the data set all you have to do is that just write print whatever is your information like this boston underscore df here i'll not say df also let me just name it as boston because uh df we basically say it for data frames and right now i'm taking out the entire data set so no need to write df so print boston dot d e s c r so once i print this here you'll be able to see the entire information about this particular data set so this is a boston housing price data set number of instances that basically how many number of data points are there they have 506 number of attributes there are 13 numerical or categorical predictive median value attribute is usually used as a target here you will be able to see various features that are present in that specific data set like ci crin this basically means per capita crime rate by town so in that the house in which town it basically exists how much per capita crime rate is and obviously if this increases the price will definitely decrease then you have other features like zn proportion of residential land zone for lots over 25 000 square feet then you have indus proportion of non-retail business seekers over town cash charge river dummy variable knocks so if one if uh track bounce river zero otherwise knocks basically same nitric oxide concentration again if this is more obviously basically in that specific town you'll have more amount of pollution right then you have rm average number of rooms per dwelling age this rad tax pt ratio b and stat so you can find out all this information pt ratio basically means pupil teacher ratio by town uh l start basically means lower status of the population median value of owner occupied homes in thousands dollars basically okay so that is giving you the information of medv okay so just check out all this information and if i go forward right how come how do i see uh what are the data with respect to this specific feature and what is the output feature output feature is obviously the price of the house what are the feature names so what i will do is that quickly just go ahead and write print boston underscore data so this will be the data that is given with respect to all this information right so here you will be able to see what are my input features these are all my input features and for this input feature i specifically have my input data that is given in this way right and if you probably try to see this this will be a nested sublist so this basically indicates that all the feature values for the first record will be present over here for the second record will be present over here now this is what data basically says if i really want to find out what is the price so here you can basically print boston dot target so this will basically be the price of the house based on all the data points that i have and this will be a single list that is simple similarly if you really want to go ahead and see what is the features right so here you can basically boston dot feature names so here are all my features that are available over here right so uh these are some of the information regarding the data set it is always important to understand the data set and i hope everybody has got an idea like what all information you are able to retrieve from this and again you can also find out this this is basically also present in this particular repository uh this is the url over there and the data set was taken from so and so library so here you can basically see all the information on regarding the data set itself right so now let's go ahead and let's try to do some eda in probably in the next video but here i think introduction with respect to the data set and understanding of the data set is very much clear for you okay so yes let's go ahead and also you can also convert this entire data set into a data frame which we will do in our next video so yes i'll see you all in the next video thank you [Music] so guys now in this video we are going to prepare the entire data set in our previous video you understood about the entire data set with respect to the boston house pricing data set so with respect to the preparation first of all what we are going to do is that let's go ahead and create my data set altogether and this data set here i'm actually going to take and i'll be using pd.data frame this will actually help us to create the entire data frame altogether with respect to all my independent features so here you can see all my independent features are there right in boston.data so if i probably write pd.dataframe on boston.data here you'll be able to see that i will be getting a data set which will look like this right so yes obviously if i wrote write dataset dot head it will give me the top five records so these are my first record second record and third record fourth fifth like this all the records are available now one thing that i feel over here is that because over here i still don't have my column names right now if i really want to put up all the column names then how should i do it right so there are two ways one is that when we are creating the pd dot data frame here obviously you have something called as a column features now in this column features what you can actually do is that you can just write boston dot features name right so if you write boston dot features name here you will be able to see the entire data set with the column name so here obviously i can see all my column names and uh uh the entire data set but still uh we are missing the uh the output feature these are all my independent features right at any point of time whenever i probably see right in my data set there will be one output or dependent feature and remaining all will be my independent features so how do i put up my output feature in this specific data set my output feature is basically present in boston.target so what i'm actually going to do i will take up this data set and i will create a new field which is called as price and this price will basically have my output feature and this i'm going to assign it to boston dot target right so if i assign this it to boston.target you will be able to see that my data set will now have if i write probably dataset.head head will basically give me the top five records so here is my output feature with respect to price also right so here you can see that we have prepared the data set now this is my complete data set over here now with respect to this data set you know what we have to do is that quickly do some kind of quick analysis you know and that analysis is super super important now first of all what i'm actually going to do is that i'm going to use some of the inbuilt function like data set dot info so if i write data center on info it is basically going to talk about like what are the data types of my all the columns that are present over here and here you can see that all it is floating 64 bits right and it is obviously non null and yes one more important point is that we also need to check whether there is any kind of missing values or not right so here these are some basic information about your data set it talks about like your specific columns what data type it is let's say if it is a categorical kind of data type then we have to do a different kind of pre-processing but here since this is just our first problem statement i've just taken a simple one and we'll try to implement this okay so here is my data set dot info uh with respect to this i have all the information that is present over here this is fine now since this is already in the floating point you know i hope you know about percentiles like 25 percentile what is the medium 75 percentile so if i really want to describe my entire data set so i can also use something called as or i can also say summarizing summarizing the stats right summarizing the stats of the data okay so in order to do this i will just use the data set dot describe so describe is one function and this will probably give you the count mean standard deviation mean 25 seventy five percent and max remember one thing is that suppose if you have features that are like categorical features so in this describe you'll not get tracked category features all the input uh features or all the independent or the output features that are numerical right that will be considered while finding out your entire 25 percent 50 percentile 75 percentile and all the other information so in short here we are summarizing the entire statistics okay now till here we have done this we have prepared our data set we have seen some of the information with respect to how to summarize your data what to how to see the generic information uh whenever you get a data set you have to first of all check the missing values okay so this is a super important step check the missing values okay i'll again say that this step is super super important whenever because the first thing is that whenever i have my data set i really need to check the missing values so in order to do this i'll be using data set dot is null function if i basically use dot is null it is just going to tell me that which rows are true or false if there is any any value that is true let's say with respect to this particular feature this is true that basically indicates that on the record of 501 this particular value is missing right so obviously most of the values are sum but still i'm not able to see all the records over here because some of the records are missing so what i can do i can also apply an aggregate function called as dot sum so it basically says that okay fine there are no missing values as such so since this was a simple problem statement as i took my first first problem statement so i had actually done this later on when we'll be seeing different different problem statements we will try to see whether there is any kind of missing values or not and we'll also see how we can replace this particular missing values so obviously no missing values over here we are good to go okay now uh what we are going to do is that after we have done this you know we are also now going to do some eda right exploratory exploratory data analysis okay so in this exploratory data analysis we are just going to analyze some information of a data now you know that over here i don't have any missing values i've seen okay this is floating point numbers right and obviously when i see this floating point numbers the best thing is that in a case of a regression problem statement the first thing that you should definitely run or first code that you should definitely run is related to correlation correlation is super super important in um in any regression problem statement because we really need to find out how the independent features and my output features are correlated if they are highly positively correlated or highly negative correlated then this basically indicates that our model performance will definitely be high okay this is one of the super important property with respect to linear regression here first of all i'm just going to implement some of the things implement completely and then probably i'm going to make again a theoretical video to make you understand about the assumptions of linear regression super important now in order to do this uh obviously uh what is the technique that i'm actually going to use here i'll just write pandas pd so let's see pd dot so i'm just going to use pd dot okay no need to use pd so if i can write df dot core relation so here if i write df.corr so sorry not df it should be dataset because that is what is the generic thinking right we always try to write df over there so here you will be able to see the correlation now if i probably see the definition with respect to this the by default correlation that you'll be able to see is something called as pearson okay now pearson correlation pearson correlation basically checks the relationship between two uh features x and y in such a way that it will try to say that whether it is highly positively correlated or negatively correlated or it is not at all correlated usually the output of the piercing correlation range is between -1 to plus one the more towards the value towards plus one more positively correlated it is more the value towards negative one the more negative correlated it is and if it is near to zero hardly they are correlated okay so this is super easy way to basically check how the correlation basically exist now here the two types of correlation that you really need to check the correlation between independent feature and the correlation between independent and the dependent feature if there is a high correlation between independent features at that point of time you can remove one of the independent features because that is what is called as multi-collinearity so in linear regression also we really need to check what is multicollinearity and that it basically means that if my two independent features are very highly correlated let's say they're correlated by 95 percent 96 percent you know or negative 90 or negative minus 0.9 5.96 because here you will be getting the value between minus 1 to plus 1 so if that is highly positively correlated or highly negatively correlated what we can do with respect to the independent features we can drop one of the independent feature and we can just take the other one the other correlation that we really need to focus on is that correlation between our independent features and our output feature that is price now here you can see if i probably take this example crim and price these are like negatively correlated by 0.38 like minus 0.38 so somewhere around 38 percent similarly if i see zn with respect to price it is nothing but 0.36 right now what is let's say what is the crim crm obviously and this will probably give you the entire information crime basically says that it is the per capita crime rate by town so obviously if this increases right this increases your house price will definitely decrease right so that is the reason why price is decreasing by minus 0.38 if the crime increases in a specific town it will decrease similarly all the other features are there so here obviously from some of the features here you can see rm is somewhere around 69 percent correlated positively correlated and this age is somewhere on minus 0.37 negatively correlated so we what we'll do in the next video is that we will try to analyze this correlation we'll try to analyze this correlation by constructing some of the important plots okay some of the plots like scattered plots or heat map or we'll also see some of the outliers whether there is some outliers in the price or not but this correlation is super super important two things that you really need to check one is for multicollinearity uh and to check for multiple identity there are also some other methods which we will discuss as we go ahead but definitely multi coloniality is the relationship between the independent features that you have to check and obviously here if i see right there is no much independent features that are very highly positively correlated or negatively correlated right so here if you could see like in this particular example you can say tax and indus it is somewhere around 72 percent correlated but i will not still say that it is very very highly correlated right so i will definitely not drop any of this feature otherwise let's say if this feature was like 96 percent correlated i could have dropped index and probably considered tax or tax or index any one of them and probably used for training my model right so this is a super important part with respect to the correlation now with respect to this information what i will be doing is that one more thing is that based on this correlation right we can also do a scatter plot so for this i'll be using c bond right and import import c1 as sns and i will be using something called as snh dot pair plot now this pair plot will basically give me the how x and y coordinates like basically how this feature crm and price is basically scattered with respect to different different points right so if i probably write df and let's see whether there are any other features that are present over here or let me just explain this sorry this should be data set my mistake so here you'll be able to see that okay it'll take some time time because there's so many features and obviously with respect to different different features there will be a different different plot uh scattered plot in short that you'll be seeing okay so this is getting executed again some five to six seconds max i guess and here you'll be able to see the entire plot okay now obviously this plot that you'll be seeing let me just open uh or zoom in obviously you will not be able to see much information over here but you can see some of the some of the data that is plotted between two features right they have some type of positively or negatively correlation right so uh this information is hardly visible so i would again focus on this correlation itself because this same value how is this distribution right based on that this is entirely calculated right so this is also one more way how you can achieve visually if you have less number of features you can see it much more clearly otherwise i think this is the information let's consider that this specific feature is there like this is crim okay and my last feature is something price so if you see the relationship between these two it is somewhat negatively correlated right somewhat negatively correlated so here you can see crm and price is negatively correlated because my data points is in the inverted manner right so definitely check out this information now in my next video what i am actually going to do i am going to do lot of analysis based on this information that is present okay so thank you i'll see you in the next video [Music] in this video guys we are going to continue the discussion with respect to this uh boston housing price data set and now we are going to analyze the correlated features now already we have seen that this is how our features will look like we have to really analyze with respect to the independent and the dependent feature uh so here let's say that the crime the crim which is the feature that is related to crime rate and if i probably see the relationship with respect to price if i want to quickly create a scattered plot all i will do is that something i write plt dot scatter and here i can basically use df dot ah my feature name that is crim along with the price right so i'll write df dot uh of price and again this is not df in short this is data set okay so i will quickly uh see to it and uh let's see how the graph will look like and here you can basically see the graph okay this feature is wrong data set okay now here you can definitely see that uh the relationship between crime and price is that as the crime keeps on increasing because in the x axis here you can see that it is having crime so here you can also put some x label and y label so here you will be able to see this will be my crime rate i'm just writing something like crime rate and plot dot y label you'll be also able to see the crime rate plus on the y label you'll be able to see the price so now here you can definitely see the relationship between the crime rate and uh the price uh obviously when the uh crime rate is increasing right or the price will also keep on increasing uh decreasing sorry this is inversely correlated here you can see it is like minus point three eight eight uh and that specific percentage is reversing or inversely correlated so that is the reason why you are able to find this kind of scattered plot okay now uh similarly what you can do is that with respect to some of the features let's consider one of the feature over here as rm and here i can see okay it is 69 positively correlated so if i really also want to plot this i can plot it over here and always make sure that you write all the information and observation that you are able to make it okay so here in my x level it will be rm rm and this will be price right rm i think it is nothing but room uh rm what is the rm feature over here let's consider the rm feature average number of rooms per dwelling so obviously as the number of rooms keeps on increasing your price will also keep on increasing right now let's say that if this is my future if i just have one independent feature and one price and if i try to probably create a regression plot how it will look like so for this i'll be using c bond right c bond and sns and here i can write snh dot reg plot which is also called as regression plot and here let's take what all features are required x feature y feature now in this particular case let me say that my x feature is nothing but r okay my y feature is nothing but price and after this i also have to give my data so here you can basically see my data is none over here but i'm going to basically write data is equal to data set which is my entire feature and if i probably execute it so you can here you can see that the same plot is basically displayed over here and here you can see that we are able to create a simple linear regression line and about some amount of variation will always be there that is also given over here right so here clearly you can see that okay there is a positive correlation that is actually between rm and price now similarly you can also take up any other feature that are present so one interesting feature that i can see is lstat and price this is somewhere on negative correlation right so again to analyze it it will always be better that i will just copy this part and paste it over here here i'm just going to use lstat and this will be my price so if i execute this here you can see obviously this is negatively correlation correlated and you can basically create a regression plot like this right so this is some amount of analysis that you can do again you can check out with respect to other features also let's consider some of the features that are not at all correlated let's say cash chas and price right so if i probably see this and if i try to print it so here you will be able to see that chaos and price now here with respect to this the data points look somewhere here it is something like this it is something like this and there we are trying to create a best fit line now it is super important that to understand that linearity should definitely be there in your data set if linearity is not there in a data set linearity basically means when x is increasing y is increasing or if x is decreasing y is decreasing this kind of relationship should be there or inverse relationship should be there to create a better regression model so usually whenever we have this kind of features that are hardly correlated right it uh it reduces the error of the regression model okay now quickly let's go ahead and do some more things and obviously guys here you have to write down the observations okay here also you have to say that as as the stat is decreasing the price is increasing okay or here also you can basically say as the rms that is number of rooms are increasing price is increasing let's take up one more uh and there is something called as pt ratio pt ratio okay so what about the pt ratio pt ratio was something like uh cha parent to tutoring something okay people teacher ratio by down okay so this was the feature over there so let's go ahead and try to do with pt ratio again i'm going to copy this same thing over here and this was something like pt ratio okay so here you will be able to see that okay some amount of negative correlation is there but again there are some types of points that are available and this information it is also saying that as pt ratio increases price decreases or as when the pt ratio increases price sorry as the pt ratio decreases the price increases so there is an inverse relationship okay now uh these are some of the information that i could take out from this particular data set now let's go ahead and quickly do one important thing that is uh create our models and for creating our models we'll do first of all there are multiple steps that we really need to perform uh that is like we need to prepare our independent and dependent features we need to make sure that we need to perform uh you know after we create our independent dependent features we need to do train test split and many more things okay so let's do all these things and let's see that how we are able to do it okay so the first step is basically dividing our features into independent and dependent features okay so uh as you know this is my entire data set uh and inside this data set i know price is my dependent feature and remaining all are my independent features right so let's quickly go ahead and let's try to uh create this particular data set so here uh i will just name all my independent features are x so here i'm just going to use df.iloc and i have to make sure that i take all my input features and i have to skip the price column okay so here it is so colon comma colon minus one if i do colon comma colon minus one that basically means i'm going to skip up all the independent features sorry i'm going to take up all the independent features and just skip the last column right that is price so this will basically give me my independent features similarly with respect to y i'm just going to use df.i lock here i'm going to take all my features and then not all my features all my data points and then i'm going to probably take the last column that is price okay so if i execute this okay this data df again it should be data set i'm extremely sorry about it okay so this should be my data set and here i actually execute it it's okay guys see if i'm making some mistakes it is always good to do some mistakes then only you'll be able to practice more and more and i'm very happy when i make mistakes because that is the reason i try to find out okay where what problem we are actually facing what did a mistake that i do and probably i will not be continuing that mistake i will go ahead so this becomes my x dot head and this becomes more y okay now after doing this the most in the important step that is nothing but train test split okay so here we are going to do something called a strain test split okay now inside the strain test plate what our plan is see whenever we discussed about the performance metrics at that time you know that whenever we get our data set we should definitely do a train test split we should keep that entire test data separately so that we are not going to touch it anytime because once we create our model and then once we test with that new with that touch data set we will try to see the performance of the model then and there other than that we'll use this entire training data set to see that how my model is basically performing okay so for doing the train test plate i am going to use from sk learn dot model selection and here i'm going to basically import drain test split so everything i'm basically using from sklearn here you have and here i'm going to use xtreme comma x test comma y train comma y test so these are all my features that it is going to create with respect to the train test plate so here you have x comma y and then here you basically have test size or i can also push by test size as strain size or test size let's say test size is nothing but 0.3 that basically means 30 percent i'm going to put over there and some random state is equal to 42 i'm just taking any random state if probably you also take the same random state value you may also get the same train test split okay so now you can basically check out my x train so these are my entire data set with respect to x20 and this is your x test and similarly you can check out your white rain and white test so this test data i'm not going to touch it now for the prediction purpose and for the checking the uh how my model is performing i'll be using the test data right now i'll be focusing more on the training data when i use it okay now in the next video we'll start the model training and before doing model training one very important step that we really need to do is something called as standard scaling okay now what is the importance of standard scalar in this particular case understand guys if you have understood about the math intuition behind linear regression problem we always made sure that we use something called as gradient descent right now when we are using gradient descent right one important thing is that uh as we and our main linear regression is to come near the global minima right and one important step is that every feature that i see over here right this is all calculated with respect to different different units now if this is getting calculated with respect to different different units one important step so that our uh you know converging of that specific algorithm of the gradient descent will take place faster we have to normalize or we have to standardize all these data points to the same scale okay and that should be based on something called a standardized scalar okay so that is what we are going to do over here a very important step which is called as standardizing the data set so here i'm going to use from sklon dot pre processing i am going to import standard scala okay and here i'm going to basically use scalar is equal to standard scala and here we execute this now this is the standard scaling object now all i have to do is that i have to pass and write fit underscore transform i have to basically pass my x training data set okay x training data set and transform it completely right to the same scale so here i will probably get my x strain okay now similarly for the x test also we have to do it but in x test whenever we try to apply we don't have to write fit underscore transform because we are going to make sure that whatever information i have with respect to the training data set and whatever techniques i have applied for transforming it the same techniques needs to be applied to the test data set this is done because to make sure that my model does not know much information about the touch data set okay so here i'm basically going to write transform on my x test okay and once we do this and if i probably try to see my x-string here you'll be able to see all the data points similarly if i go ahead and see my x test here you will be able to see all the data points we don't have to do with respect to the output feature but definitely for this specific step right and whenever we get our new data set there also we have to probably apply transform and try to do the prediction now these are the steps that we did it now in the next video we are going to directly train our model this is super super important steps again understand this can be an interview question they'll say that why do you standardize a data set in linear regression you just have to say that internally we use gradient descent our main aim is to come to the global minima and to come to the global media we have to make sure that all our independent features unit should be in the same scale because of that the convergence will happen quickly okay super important super important step so yes uh what we are going to do in the next video we are going to understand how to implement the linear regression algorithm on this particular data set my data set transformation is ready my data set is completely ready all i have to do is that train my model and try to find out a good accuracy and also make sure that see how it is performing with respect to the test data so yes i will see you all in the next video thank you [Music] in this video we are going to understand now how we can actually do the training of this particular model with the help of this data set that we have recently standardized it and split it into basically first of all we split it into train and test and then we standardize the data set so in order to do the model training again we have to probably go and import the linear regression model so for this i will be going from sklearn dot linear regression sorry escalon dot linear model and i'm going to import linear regression okay now one thing that i have seen from many people is that if you find out any confusion with respect to finding the linear regression model all you have to do is that just go and search for scalar and linear regression so here is the model that we are going to use and by default parameters here you can see normalize is deprecated over here so this all parameters you can basically see it is basically first parameter something called as fit intercept whether to calculate the intercept for this model or not by default it will be true we really need to calculate the intercept model and they also some parameters like normalize this parameter is ignored when fit intercept is set to false if true the regressor x will be normalized before regression but we have already normalized it we have done that suppose if you don't want to do the normalization you can basically keep it as full but it is always a good practice to basically normalize it so here you can see that normalize is also getting deprecated so it is better thing that we try to do it by our side itself okay and uh here are all the other features not much features with respect to linear regression but when we go and learn about the other algorithms like ridge lasso and elastic at that point of time will play with some of the parameters so here obviously you have understood how to import it from scalon.linear model i'm going to import linear regression so this is my parameter that i'm actually going to use so here i'm to use regression okay regression is equal to linear regression and i'm going to just initialize this specific model okay now in order to train the model uh we have to just write regression.fit on my extreme and comma y tray okay so once i do this you will be able to see that regression model is created right now this in short once we do fit so on that entire data set we basically create a hyper plane because understand here i have so many features i cannot say that i'll just be creating a 3d planar lock like i'll be creating a hyper plane because i really have so many number of features that are existing over there okay so quickly if i really want to print the coefficients so let's say i want to print the coefficient because whenever we create a regression plane right whenever we solve a regression problem statement two important things will be there one is coefficients and then the other one is intercept okay so in order to print the coefficients i will write print and here i'm going to basically say regression dot coefficients okay and these are my coefficients over here and this will be equal to all the number of independent features because for every independent feature i will be having a coefficient right i'll be having a coefficient and this basically indicates like let's say the first feature is nothing but crime right that is what we have seen already the first feature is nothing but cri crm rate basically means that with the increase it basically with the unit increase in the crim value there will obviously be this much decrease this much unit decrease in the price value something like that right so if suppose let's say thousand crime is increasing at that point of time you know 1100 dollars will be increased decreasing with respect to the price or whatever the unit of the price is there with respect to that specific unit okay so this is with respect to the uh coefficient and similarly one intercept you will definitely be having so here i'm going to write uh underscore intercept intercept and here this is basically the intercept over here okay and i've already explained you what is the importance of this intercept also as we have gone made okay now uh what we can do is that uh if you really want to find out like on which parameters this model has been trained so one more statement i'll write a comment on which parameters the model has been trained okay so here you can basically write regression dot get params and you can execute it obviously this will be a method so just try to execute it like this so here you will be able to see that copy underscore x is true fit intersect is true and underscore jobs is non normalized is equal to false okay so this all information is there and these all information you'll also be able to see in the scala okay now one super important thing because once we have trained our model you know then we are going to do the prediction of a model okay so let's go ahead and do the prediction with test data okay with the new data also i'll try to show you but first of all we'll try to do the prediction with the x underscore dash that is there so i'm just going to use a regression dot there will be a method which is called as predict and here i'm just going to use my extras data okay and this will basically be saved in my regression underscore predict okay so once i execute this and if i probably like to write reg underscore thread here you will be able to see these all my predictions okay now one important thing with respect to predictions is that first of all whenever we get any predictions right we can definitely compare this with the y test information because for the x test y test is the true value right so how do we check like what all information or how whether this model has actually performed or not okay so i'm just going to plot some of the important diagrams uh which will basically help you to understand that okay the kind of predictions that you have got from the model it is it good or bad okay so first of all let's go ahead and let's just plot first plot a scatter plot let's plot a scatter plot for the prediction okay for the prediction so here what i'm actually going to do i'm just going to write plot dot scatter and here i'm just going to use uh you know if i'm probably plotting uh this regression thread okay so what i'm actually going to i'll just take reg underscore red comma y underscore test so now if i plot these two values or i can also change the order let's say i'm going to write y underscore test and reg underscore print okay if i plot this guys here you'll be able to see that the plotting of the values that you'll be seeing it is basically linear right so when this plotting is basically linear that basically means yes your model has actually performed well right and reg thread is basically your prediction value this is your truth values when you see that both the values are almost linear like this right that basically indicates that yes some amount of good information your model are actually done and it is predicting well okay so this is one of the prediction plot that you basically have to do to start with the another plot is that with respect to the residuals so here prediction with uh if i if i say residuals okay first of all let me just go ahead and calculate the residual so here residual is nothing but the error right the error with respect to y test and regression thread okay so if i write y underscore test minus regression production so this will basically give my residual residual basically means errors right so if i probably try to calculate this and try to execute it so here you will be able to see the difference between the y underscore test and regression underscope right now the second plot that i am actually going to do this is the plotting this residuals plotting this residuals now how do i plot this residuals here i am actually going to use plot this residuals okay here i'm specifically going to use c bond so let me use snh dot and i'm going to use something called as dist block now when i plot this disc plot with respect to residuals right let's say i'm just going to plot this residuals okay and here the kind of plot that i'm going to use is something called as kd so here you'll be able to see that i will be getting this kind of plot now here you you can obviously assume that this looks like a normal distributed but here you can see on the right hand side okay there are some kind of outliers and errors the errors are quite high and similarly on this particular graph also you can see that there are some points that are really really uh away from all these things and this looks like an outlier and obviously the difference will be quite high this this is the same graph that basically indicates our main assumption should be that when we are trying to solve we should try to get this as a normal distribution and yes from this right my my error that is my residuals ranges mostly between minus 1 minus 10 to plus 10 in a normalized format and here you will be able to not in a normalized format here you can see the maximum difference is between minus 10 to plus 10 and there are some points that are ranging between 10 to 30 okay so i still feel that yes my model is performing well because whenever i do this kind of plot with respect to the residuals i should be getting a normal distribution so this is the second assumptions that you can probably consider and you can basically find it out okay now let's go and probably assume one more scatter plot and this scatter plot is with respect to with respect to predictions and residuals okay residuals now if i probably try to plot this so let's say i'm plotting plot.scatter with respect to the regression prediction and my residuals that is this error if i plot this here you'll be able to see that my my this this different the when i'm plotting or when it's doing the scatter plot with respect to the output predicted value and the error here you will be able to see there is no such fixed way of telling this is like scattered anyhow right this is scattered anyhow it is not following any kind of distribution or it is not following any kind of format you know uh let's say that i can say that okay this can follow this kind of shape or it can follow these are evenly distributed at every side i can yeah this point will be perfect this points that you can see between the predicted and the residual are evenly distributed here and there some of the points are here some of the points are here but most of the points are scattered uniformly right yes uniformly is the word i'm extremely sorry uniformly is the word so here you can see that uniform distribution is basically there with respect to the test data and residuals super important here i'm going to write uniform distribution right so this still looks good so what i'm actually going to do over here is that i'm just going to consider this uniform distribution and continue so still this gives me an indication that yeah your model is performing well so obviously after doing such a prediction i have plotted many many different different kind of diagrams over here just to indicate that how your model is performing but to be sure to be sure we definitely have to use some kind of performance metrics now what are those performance metrics that we can definitely use first of all i'm just going to use from sklearn dot metrics and i'm going to import something called as mean squared error right i hope everybody remembers this mean squared error that was our cost function right so i'm just going to say mean squared error and then also like mean absolute error root mean squared error we'll discuss about all those things and probably if you really want to also come up with other things like from sk learn dot metrics i'm just going to import root mean absolute error so and let's let's consider this too right now okay and mse i can probably print all these things so my print statement will be having mean squared error with respect to my y sorry mean absolute error with respect to my y test and reg bread this also i'll try to print and i'll also print me squared error and here also i'm going to basically take y underscore test and reg underscope correct so these values will obviously be able to see over here there is also something called as root mean squared error now in order to apply root mean squared error so here i'll be saying np dot i think we have something called a square right square square root okay i will just apply this on my mean squad error of this specific value so here also we'll be able to get the root mean squared error so these are all my values and always remember that now these values will basically indicate how my model is basically performing now apart from this here we have definitely understood okay if i plot the residuals i'm getting a normal distribution yes with some kind of outliers with respect to the differences but most of the points that are having the differences is between minus 10 to plus 10 okay and one more performance matrix that i can definitely use uh is something called as adjusted square r square r square and adjusted square right adjusted r square so this part i will be discussing in my next video this is also another super important performance matrix that we can definitely use uh whenever we are trying to create a linear regression model okay so this is the next thing that we are going to discuss so i hope you have understood till here we have understood about the assumptions so here after the model training i am just going to make some assumptions over here and with respect to the assumptions to just understand whether your model is going to perform well or not you can basically use all these assumptions but at the end of the day see if your data points is highly scattered right one important thing you will be able to understand that obviously just with a single line right you won't be able to do that much good predictions that is a linear line so linear regression is basically creating a linear right obviously with that linear line you will not be able to just create a better model right because your data will be highly distributed and you will be having lot number of features with respect to that right but again we have created it here we have made some assumptions assumptions was basically scattering plot w