Kaggle Data Science Competition Course – Solve Three Challenges Step-by-Step

Key Takeaways

improve your data science and machine learning skills by completing kaggle competitions in this course Rohan Kumar will teach you how to solve three different kaggle problems step by step providing valuable insights and practical knowledge in this tutorial we will be showcasing three different kaggle projects that will give you a brief idea on how to approach kaggle problems which closely correlates to giving you experience of how to solve real world problems according to me kaggle is a premium tool when it comes to data science and Mission learning and it is available for free and speaking of mission learning and data science I run a channel called smds where I primarily post videos about Mission learning and data science itself in fact this entire tutorial is an illustration of our ongoing kaggle Series where I post videos about taking on kaggle competitions starting from the beginning and going all the way up to the submission file to have a look at our leader vat and that's how we will be doing all of these three kagle projects and I also want to mention that at the time of recording this I'm focusing on giving you a perfect computer vision playlist inside of our Channel and that also contains a lot of gagle competitions as well so you definitely do not want to miss out and make sure to leave a link to that in the pin comment now with all that said let's just get into the first project itself all right so here I am in the main page of the competition itself the kagle competition I'll make sure to link this now in the description box below and over here what you have to understand is we will have to basically predict the housing prices based upon some given variables and for this case and since this is Advanced regression techniques as the name suggest we're going to have 79 explanatory variables describing each and every aspect that we will need to make predictions right so as I get to the data tab here they have given us the information of the files that we're going to be downloading there will be a training set test set and then there's some samples Mission CSV file as well and then they have given us information of each and every single column or the feature that we have so if you're going to spend three to five hours in this project this is what I would recommend you to do just understand each and every single column and try to create interesting features with these that assist perform extensive feature engineering and then take it to the next level in terms of predictions as well but for our case in this video just for the sake of Simplicity I'm going to deal with the nonv values just perform basic data analysis and manipulation and then go to prediction directly all right so here that we got information of every single column and then we have the actual visualization of the data itself this you will only get to see once you join the competition so when you click on join competition it's going to ask you for confirmation and as you do this you will be able to be a scop Edition but once you do that it will refresh and then you can go down to the data tab once again and then now you'll be able to see the example of the train CSV itself now there is only one thing that is not going to be present in the test CSV that is going to be the sale price this is the one that we have to find ourself I don't think so it's available here you just have to find the um sale price column all right now this is not going to be available in the test set because that is the one that we have to predict and then give it as a submission and they've also given us a sample submission CSP pile the things that they're expecting they expect a sales price column and then they also expect an ID all right so after you've gone through all this just click on download all that will give you a zip file you will extract it to a specific folder now I have come to my vs code and this is my J notebook here I've written all the code down I'll be explaining it line by line but basically after extracting all of those four files to a specific folder I recommend when you create your python file in that same folder so that you don't have to sensely copy the part inside I'll let you know what I mean can I get right so first thing that I'm going to do here is import all so M LIF and I've also imported C1 this is another L package again if you haven't have just go to your terminal type it install and library right what after that's done I am looking what's my data so now I'm going to use the me CSN function from handers the plain set and all right so I'm going to call now inside of I have created a right column and the reason why I'm doing this is I'm going to La these two columns initially before I per C oration and the reason why I'm doing this is you always have to make sure that your test set and your train set come along same and the main reason why I'm saying this is let's say you do some Eda or let's say you do some data manipulation or train set alone and some of the features that might not be available some of the nonv values that pops up in the test set what might not pop up in the train set will not be dealt with if you just work on the train. PSV so I always recommend you to merge these two put them in the same domain and then start with your exory data analysis and later everything is done you can split them again into the one at the two all right so first one of done here is loaded those two in you just have to copy the part of the train CSV and the test CSV and put it over here and then create a dummy sales price column just initialize everything to and then use the pd. ncat function with them inside a list df1 and df2 and then set access equal to zero this does mean that we're going to vertically stack those we're not going to do that horizontally they just have the same amount of features it's just the number of records that is going to increase all right and then I've set the index to ID because that is going to have basingle unique value now when you run this C up that's going to create our main data frame and then when you type in DF do head as you see over here we get our first five records of the data set itself now here we have set the ID as the index so that pops up a little bit down and then it shows 80 column 79 features the sales price column and the ID all right so after that is done the next thing that I'm going to do is find all of the nonone value columns that is basically if a specific column contains nonone value then I will extract it and and put it inside a new data frame all right so let me show you what I mean so this statement wres here I just want you to look at the one that I'm highlighting right now now this basically means this will show us all of the nonone values in each and every single column and I'm setting a condition that if it is greater than zero that if at least one none value is present in a specific column this will show each and every single none value I can show that ef. SN do sign and need just done this code so you have to put in it like that all right now it shows that each and every single column the amount of n vales present now if it is zero then we're not worried about it so I've said in the condition that if it's greater than zero let me just show you what happens when I do I did this it gives us a true or false value a conditional value just like a moth and I'm going to do that put that inside of the slicing condition and do the same commment once again so this time we're going to get all of the columns actually let me just copy this and show you all of the columns which contains at least one n value so when I paste this and the sco going to get all of the columns that contains at least one none value that is what we're going after and this only displays the nun values in each and every single column what I want is the index of that so I'm going to the index and then extract that from our data frame and store it in a data frame called BF n all right so now when I run this code s specifically and then do this we're going to see the 34 columns that contain the N value and those will pop up right around here all right and then now I'm going to check the heat map that is the amount of nonone values this will give us a good understanding of the none values present in each and every single column of the DF note so when I run this all it gives us the exact heat map that you just saw and as you see over here let's take the AL column for example there is just too many nonv values and so is for the pool QC so we can safely remove all of these columns where all of the non values are just extensively High all right so what I've done here I've gone on ahead and created another data frame and it's called DF objects all right what I mean by objects is all of the columns that contain only just string values or other values that is not numbers for example this doesn't come under object this can comes under float right but this comes under object and this comes under object so I'm going to create a data frame which contains all of these object columns alone and the way I'm going to do this is by using the DF do select D types function and then I'm going to include only the object data type we retrieve all of the columns out of it and then put that inside the DF entirely and then store it in a data frame objects all right now so this does mean that I'm first dealing with the nonone values of the objects column that is I'm dealing with the nonone values in the columns which are only objects and right over there I have just done the same thing what I've done over here like basically just setting the condition to More Than Zero that is I've set the condition over here to 1,100 that is if there is 1100 unv values then it's not going to give us much of a value so I'm just going to drop it directly all right so use the DF dos and a. Su and then mention the condition just like a mask retrieve the index just like we did above and then retrieve all of those columns mention aess equal to one as you do DF do drop all right hopefully that made sense so we have created DF objects and then I have also dropped off all of the columns in the objects where the none values is just greater than 1100 so let's just run this C out perfect that is done all right I've done the same thing this is done on the main data frame I've done the same thing on the DF objects data frame just basically removing all of the columns where the none values is just greater than 1100 and then we are going to fill na values all right so basically now our dat DF objects let me just actually cut this and type in DF objects. head to show you do head you run this goes itself now this objects data frame will contain only those columns where the nonone values are just less than 1100 that is we can at least get some value out of it so those are the columns that we have in this DF objects data set and what I'm going to do here let me just B that once again in those columns I'm just going to fill all of the nonone values and what I mean by that is let me let me just see if I can find some um nothing all right so I'm going to fill all of the none values by a string named not let me just show you why I do this all right so let's say I take this utilities column for example all right so I'm going to access those utility columns and then when I type in value counts and then come over here and see there is only two categories of values in this utilities column it's either all pu or no SAA and no SAA is just one value apart from this there might be none values in this C column so what I've done over here is actually let me just cut this and run this codes I'll show you what I've done over here is fill all of those none values let me just do that and come over here and do this and now we see that we have a new category null category so now we will have three different categories and why did I create this category is because I'm going to do one hot encoding so basically what this means is like right as you see over here the utilities column will have three different types of values it's either all pu or n or sa so what I will do is create three new columns for each and every single record and for that specific record if they belong to all Pub category then off of those all Pub null and sa all Pub will have one and then null will have zero No Sa will have zero but then if the null value is true that is if null seems to be the uh category for a specific record then null will be one and the other two will be zero and this is known as one hot encoding you can just directly do that by using the get Dums function in pandas all right you just have to pass the data frame and that will create a specific new data frame just going to call that DF objects. encoded so let me just run this Cod cell once again and after I've done that I'm going to come over here before we get to this code cell let me just type in DF encoded objects encoded do head I just want to show you the first five records of the encoded objects all right so right as I see over here we've got 247 columns and creates different C categories for Ms zoning like Ms zoning might have in fact let's just go ahead and type in Ms zoning and then as you see There is five different categories right including null it's six different categories and that's what is created here so for the first record if it belongs to the RL category then RL will be true but then all other everything else in terms of amazoning is zero so now we have six new columns out of this Ms zoning column alone and that's what I've done here with every single column in the DF objects right so that's how you deal with it and then what I've done here is I want you to notice this column over here this Ms zoning null only if other five of them are false then this one will be true and we don't have to extensively mention this to the model while training so what I've done is I have deleted all these null categories column so what I've done here is gone on ahead and looped over the entire encoded data frame column and then if null is present in those columns then what I've done here is just dropped off those null column mentioning X is equal to one and I also made sure that I'm printing in so need one this Cod so you're going to see all of the dropped off column and it all has got no so you don't have to extensively mention this to the model but if you do wish to do that it's actually okay to do so all right so this is kind of an optional step and I've done this um the new data frame that I've created over here this is the new DF and over here it has got 304 columns and I only want the columns which are not objects because we have dealt with it and we have created too many bullan columns as well so what I've done is access the main data frame object columns and then drop that off from the new data frame and then now I have set the condition once again checking for null values in each and every single column and if it turns out to be true these are the ones so basically what we've done here is our new data frame contains only number numerical columns that is the columns which only contain numerical values it's either one zero or any floing point numbers so those are the columns that we have and over there we've got so many non values in these columns so we have to deal with all of these and what I've done over here is again let's just R this so and what I've done over here is created two new list I've named it mode columns and mean columns this basically means I've created a list of columns for which I'm going to fill all of the none values with just mode that is I'm going to find the mode of those columns and use it to fill the N values all right and then when it comes to mean columns I'm just going to find the mean value of a specific column to fill in the non values for that the way I've decided uh if I want to find the mode or if I want to find the mean it's by just looking at the column how priz is and then go to the data to look for a specific column and then see what kind of U definition have they given for example in our case let me just go ahead and see for garage cars so show you all right so garage cars the size of garage in the car capacity um so that means that we can't find a specific mean value for this what I've done here is gone on ahead and I'm just going to decide that I will have mode values for this so that's how I have decided for each and every single column that goes under mode and mean all right so just looking at the columns trying to identify what it means and then I have looped over all of the columns and every single columns I have used the fill ni function that is just again you just have to see that new DF of I I'm again accessing it with the fill na function and then finding the mode of it just make sure that you're accessing the zero element that only gives you the mode and similarly I have done that over here as well I've just used the numpy round function just making sure that I don't get a floating Point number and finding the mean for each and every single column to fill in the N values over there all right now when I put in this condition it should not give us any columns and it gives us an empty list so there is no nonone values whatsoever inside of our entire data set and I've called this the new DF so the final data frame that we have after after performing pre-processing is new DF now try to remember that I uh split the training date that I mer the training data and the testing data together before performing any type of exploratory data analysis so now it's time for us to again split them up into df1 and df2 and I'm calling this the testing data the training data the training data will be the Len of df1 that is it's going to start from zero and then find the length of df1 and then get the exact length of all of those and but done the training data and onwards that is from the length of df1 till the end will be the testing data because remember that we have vertically stacked the training data and the testing data so on top of the testing data is going to be the training data so the first initial set of columns will be of df1 and then from df1 till the end will be to the testing data that is df2 all right now our testing data we don't need the sales price column anymore so I'm going to drop that off as well all right so let's just run this and now we come to the model training section all right here we specifically select a few algorithmic models and then train the model based on the given data and then we decide how the model performs based upon some metrics and then we will go on ahead to create a submission CSV file and give it and submit it on the competition so here a lot of inputs are happening and the first dependency that I'm going to show you is the train test split this will split the data let's say I'm taking the training data over here I want to split this into train and validation I will be providing around 80% of the data to train the model and then 20% of this training data because we have the labels for these training data to test the model on the training data itself that is the rest of the training data I'm going to call that validation set all right and then I have imported metrics mean squar eror this is the metric that I will be using to see how well a model is performing and or how dumb a model is performing all right you can choose any other metric as well there is F2 score there is many others so I've just gone on ahead with the mean square error for simply see and then I'm going to be using three different models and see which one performs the best all right so I've gone on ahead to import the linear regression from the linear model there's also polinomial regression you could try out which I recommend you to do so and then from XG boost this one turns out to be my favorite from XG boost this is just gradient boosting again just make sure to bip install XG boost so go to your terminal and type in bip installing that will install the XT boost algorithm where it's got XT boost regressor and XT boost classifier as but we have talked about that in our previous video and then I have also gone on ahead from Psy learn and sub I have imported a random Forest regressor there's also a run for classifier which you can do for classification tasks all right so import all of these and once that is done now I'm going to split the training data all right so I will be dropping off the sales price column just temporarily just to set this as the training data and then the output label y label will be the training data's sales price all right now here I have haven't mentioned in place equal to true so this is not going to just directly delete the sales price column off of the training data but then it will extract that out of it and then put um everything else in terms of X and then as I'm accessing it on the Y label it stores that on the Y all right then I have created X train X test y train and Y test and this X test and Y test will be out of our main training data only see as I want you to notice over here we've created and Y all right we've used the train test split function you pass in X Y label and then you want to mention the test size that is the validation size if you mention 0.2 that means that you will get 20% of your entire data for testing all right so X train will contain 80% of X and Y train will contain 80% of Y and the rest goes on to X test and Y test and then I have used the reshape function because generally when you see white trains shape before uh doing this step it's just going to be a one-dimensional rather than being two-dimensional and our model expects always a two-dimensional array in terms of understanding data so I've done that for white train and then I've also done that for white test and then now when I run this code so if I type an X train. shape and Y TR on shape it gives us this shape and then we want it to be two dimensional and make sure that the number of Records match all right this one basically means the output value that we trying to predict all right so the first model that I'm going to have is linear regression we have imported that above and then you just have to type in actually first as you define the model the basic format is always going to be to Define that model and then use the fit function and then use do predict function all right so first you create the linear regression model and then you pass in X and Y of course you could pass in the X train and the Y train but we also have a specific testing data which is what we're going to try to predict on so what I've done here you could try out the X train and the Y train which I actually did and then see how the model performs on the test set so let's just right put that X train and Y train and then run this codo we have to put this in capital now we get a pretty high mean square error when I compare that to our model two now model two has a few hyperparameters first thing is going to be n estimators how many estimators do you want in terms of grading boosting and then you also want to mention the learning rate again and random state is an optional step let me just remove that to set a random seed I'm just going to have that learning rate is basically how fast you want to take the next step in terms of learning again if you're familiar with the grading boosting algorithm uh directly then you will understand what learning rate is so basically you just have to mention two parameters two hyper parameters as you mentioned you create xtb regressor and then use the model. fit again pass in the train data first to see how the model is performing over there capital Y right all right and then um see the predictions for the X test that is the 20% off for our training data you use the mean Square Ed function you first pass in the actual label and then you also pass in the prediction label which is we made over here model to predict actually does that all right right as we see over here the model actually performs better than the linear regression model itself directly but again this gives it a very very high value so as I try for the random Forest regressor again you have to mention the number of estimators that you want in terms of this algorithm and as you get the regressor function again you just perform the same steps and see how the model is performing over here again just make sure to put in the train right make a prediction all right now we get the random forest mean square error as well and this seems to be higher than the xtb regressor and in terms of mean Square eror we want to make sure that the number that we get is always lesser so this model turns out to be better than the other two that we have decided so what I'm going to do now is just rather than giving the train set alone I'm just going to give the entire training data set because we want model to train on this and then now see how the model is performing again we see that the values drop the mean Square ER values drop significantly that's because we're giving in the entire data we just going to do the same thing for our second model that is xtb regressor and then on this codes some all right and here as well we see the value drop off significantly and then I'm just also going to do that for random Forest because we just had to provide in the entire data set in terms of training to train our model perfect L and we make predictions on the testing data alone and what I mean by testing data is this one and what I mean by X test or Y test is the test set that we created from our training data all right so as I see over here once the model is done training we get the mean square error and again this value does drop off but again it seems to be very very high so when you compare this to other two x un aggressor does significantly better so that's the model that I'm going to take to make predictions and also this is just for visualization you don't have to actually understand this this is kind of an optional step you could do any type of visualization if you wish I'm just going to delete this and I have selected model two as I saw it has the lowest meain square error and then I have used the predict function passed in the testing data to get predictions and I want you to remember the sample submission CSV file in fact let me just go over there our sample submission CSV file should contain the ID and then the specific sales price that we are predicting all right so I'm just going to go over here and then to get the index or the idid itself the testing data. index that is our final ID because since we have set the index initially to ID over here in the very first line in the very first cell of our data analysis we have set the index as ID so this column will be set as our index and right as we split that over here we can just directly access that index that index will be the ID column from the testing data and then I've called that column as ID so first after creating an empty data frame in and calling it final the final of ID that is creating a new column ID I said that with the index and then I've made predictions accordingly with the testing data and set that as the sales price column now we don't need a specific index column we have the ID itself so we have to mention index equal to false when you're typing in final to CSV that is it creates a CSV file I've just called that output. CSV and then you don't need a specific index so many you Le this code itself I going to create an output CSV file as I open it we get only two columns ID and sales price whereas when you don't mention this let me just show you what happens um UND once again and go over here just do this now we get another additional column which is the index which is not what we need so I'm just going to close this and make sure that I type in index equal to false like that and then this go once again so now we have an output CSV file which we can submit for prediction and the way you have to do this is just go to your homepage of the competition itself and over here just click on submit prediction and navigate through to your folder where you have the python file and then from there you can just select the output CSV file but let me just do that quickly the output CSV file I'm just going to track and drop that over here and then click on submit and there we go our submission is successful we've got a score and this score has to be as low as possible so when I go to my leaderboard and then when we jump to our position it put us at 1458 which is pretty decent and we got many others at the same number as well and the number goes down further as well so you want to make sure that you have minimum amount of error if you can achieve this type of error that is actually really really nice and if you wish to do that and what I recommend you to do so is just go over here and identify and understand every single data column trying to create feature engineering and trying to just manipulate the data much more it's all about manipulating the data rather than finding a perfect model or an algorithm all right let's have a quick overview of the problem statement itself here they have given us a brief introduction about the problem but basically what you have to understand is around 13,000 people are traveling in this spaceship Titanic and we have to identify for any given person if they have been transported to the desired location or if they have been put into the alternate Dimension and here we have all of the data columns or the features if you will and the transported column is what we have to predict so if the person has been transported which in this case will give us true and if they have been put to another dimension then we will get the output as false now I recommend you to look at all of the information given about the columns it's just very important when it comes to feature engineering so the way I'm going to approach this problem is by two things I'm going to be using K and imputer to pre-process the data fully and then we'll be testing out different models to see which one performs better again if you don't understand what I mean I'll make sure to show you how I implement it line by line but with that said now we just have to download all of this data there's three files there is strain test and CSV the test file will obviously not have the transformed column and the submission CSV it's just the expected format of which we have to submit we will have to have in the passenger ID and then we will have to say if the person has been transported or not it should be a Boolean value all right so after going through all of this just make sure to click on download all it will install a zip file and then you have to extract that to a specific folder now here I am at vs code where I've created a new folder CLE not2 and that is where I have all my three files right over here I'm just going to create a jupyter notebook again make sure to have the Jupiter extension installed of vs code and then just type in main. iy and B now this is going to create a jup notebook and make sure to select your default Kel by default it's just going to have the base python installed onto your local machine and now after we've done all of that we can just get into the main code itself I'm going to be starting off with importing all of the packages that I need we'll be needing the standard pandas num La lib then I'm also going to be importing seon as SNS thank you copilot and then the main tool that I mentioned earlier we will be importing K andn imputer I'll just tell you how it works as we get there for now I'm just going to import it like that and now make sure I run the scell right these are the ones that we will be needing to perform the pre-processing step after this is done we will get to the machine learning step where we will use multiple different algorithms and test out different models and see which one does perform better now after that is done we can now load in the data and then name it a data frame so I'm going to First Call in the train data frame which will be pd. read CSV and then just the train CSV because as we look over here they've already boughten the train CSV and the test CSV together so let me just load in test CSV as well like that and once that is done I'm going to create a dummy column inside of the test set because that is a column that will be absent in the test set and that is what we will have to predict so what I will be doing is fill it all with false like that and then now before performing the pre-processing step this is the most important thing that you have to keep in mind you have to concatenate these two files to put them in the same domain just make sure that whenever you're doing a machine learning problem you just have to put all of the data sets all different data sets into a single domain before pre-processing before performing any sort of data manipulation all right now to do this I'm just going to call it a main data frame and then use the pd. concat function to katate the train and the test set I just make sure to put sort equals to false nor want to sort it like that and then let me just see the actual data column itself now when you do this we have to get the top five records of main data frame and it shows us how is it available all right now just to make sure that we have concatenated property let me just type in DF do shape must be equal to actually D do shape of zero must be equal to you trains DF train shape and DF shape now this should return true and if it's true that means that we have concatenated perfectly all right now as we look into it we just see that we have all the different columns now the immediate next step is just going to be identifying all of the nonone values in each column so main function that we're going to be using to identify this is just typing in BF dos sna do sign like that and when I run this Cel you get to see the number of nonone values in each column so I recommend you to deal with all of these none values column by column rather than using a simple algorithm like K and imputer or simple imputer but for the time being I be using K and imputer it's a much efficient way as well so now for this one I'm just going to go back to our overview page where we have to understand the data itself now over here I want to see the cabin column because as we see on the data set the cabin column has values that looks like this and the information that they've given us here is the values are in the format of the deck number and the site so we have to separate them into three different columns and we just copy this so we will have to separate this into three different columns one as deck one as number and the other one as side so in order to do this what I will be doing is using regular Expressions to extract the exact string based upon the forward Flash and to do this let me just show you our DF which will get three new columns deck comma number which will be first access the DF cabin and from here convert this to string and then use the split function and then we want to split it in terms of the forward slash like this then type in expand equal to true now this will create three new columns just like that and those three new columns will be named as deck number and side respectively and then we can make sure to drop our cabin column columns equal to cabin that's about it now let's just run this and then go over here just to see our DF do head great okay just typee in DF and and now we should be having three new columns while the cabin column should be gone yep so we get deck number and side now the next thing that I'm going to do is deal with these nonv values first now if you run this Cod cell once again you'll be seeing that you get three new columns over here and I'm going to deal with these three columns none values first and the reason why I'm doing this is because at the end I realize that this gives me much better accuracy but then you can try this with simply imperi as well as I show you how to I'll be here PF of De will be PF of deck fillna this function will help us to fill all of the nonone values with the value that you're mentioning then I'm just going to use U so all of the nonone values and the reason why I'm doing this is actually show you DF of de. value counts is a function they can use to identify the number of values in a column for each and every single category actually this is a little bit bigger let me just show the side yeah so when you use the value counts function it will show you the different categories available in each column and then their values and this will be sorted from top to bottom all right so let me just try that for the home planet show you a much better example again home planet is the column which is the the original location of the passenger itself there is a bunch of Europa Earth and many more and when you run this code cell oops you just have to put in this now when you run this it gives you that there's three different categories and then their respective values available so now when I fill the none values with u there is a new category created which is you so let me just run this codo once again actually we already extracted the cabin column so what I could do is let's just run all of these Cod cells once again click on run all all right now we have filled all of the nonone values with u and now when I put in the value counts back we get a new category which has all of the none values and now when you run the is na. Su function it will give us no none values which is what actually we want so deck is dealt with like that and this is numbers column so we have to fill it with a number then what you could potentially do is just have the number column filled fill it with something like minus one and then I'm just going to do the same thing for side as well just add a new category EF of side and just type in new like that just going to cut this actually other than doing that let's just click on run all once again all right now as we see now let me just run this codel once again to show you that there's no none values for the three columns that we had and then now what I'm going to do is something called as label encoding all right there is nine different categories in deck and I'm just going to convert all of these categories into numbers let me just get to the Scot cell the way I'm going to do this is first access this deck column then over here use the map function DF de. map and then you can use a dictionary inside of it basically going to start with G now G will be termed as zero and then put a comma f as one and you basically get the idea you're just going to convert all of these categories into numbers and those numbers will have these values then finally T will be8 like that then after this is done I'm also going to do this for the side column as well side as only I guess three so let me just see three different categories yep so there's s p and U so as we access the side column after mapping these two let's just run this codo and then now we can basically get to K and imputer now what we can do here is first Define a set of columns which will have all of numbers so let me show you actually now if you look at all of the columns available now here we see that the name column will never be numbers it's only going to be object again let me just get to the data frame itself show you and then this home planet is object this destination column is an object and rest all will be numbers and then what I'm going to do with the name column is just basically drop it it's not going to give us any sort of value towards getting to the final output itself so I'm going to just drop the name column from the data frame itself and then we're not going to consider destination and home planet for now we're only going to take all of the numbers and the Boolean values together and then use the KNN imputer because KNN imputer Works only best with numbers what going to do here is actually first just type in DF do drop let's just do that over here DF do drop name is the only column that we're going to drop and get over here then I'm going to create a list of imped lists which will be all of the columns that contains numbers so the columns that contain the numbers will be age all right here I've included all of the columns just basically ignoring the home planet and the destination those two are going to be objects and then we have already dropped the name column and then you will not be needing transported column so other than that every other column has been included over here and we're going to be giving this to the K and imput to get the updated data set then the rest is going to be let's just create another list list of let's access all of the columns that is available in the data frame and then take off the inut list like that and convert this to a list so this is the rest of the columns that is not inside of in list and then let's create a data set will be BF of rest like that now I'm going to create a new variable imp and I'm just going to name this to be can and imputer algorithm imputer now G and imputer works based upon the the amount of neighbors that it is going to consider before updating the data set so I can just type in neighbors and basically what this will help us do is fill off all of the none or the missing data based upon looking at the other data set using the nearest neighbors algorithm Now by creating this as an algorithm with n neighbors set to five we have now an imputer which will take in our data set which will only take in the columns that we mentioned over here and then based upon looking at that it will fill all of the missing data I hope that made sense and the way we can implement this is by using let's just create an imputed data frame just type in fit transform then send the entire data frame with our imputed column well just like that this is the data set that we're going to give to our imputer and it's going to fill all of the missing data that is inside of that and then I'm just going to update this data frame to a data frame Itself by using pd. dat frame by using the pd. dataframe function you can just convert the imputed transform data to a data frame itself and then you just have to mention the columns as well because you have to have a name for each column and that will be as same as information over this inside of this list all right so after that is done what you could do is just update our main data frame all right so now what I'm going to do is take this data frame and this data frame and combine it to get back our initial data frame so to do this let me just redefine our data frame again use the concat function and basically we're going to be con catting the BF imputed and the DF rest this is the rest of the columns that is not inside of this imputed data frame so to make sure and get back the original data frame itself we have to make sure that we don't give index to this imputed data frame and this rest data frame so to do this what we could do is just put this inside of square brackets first if of rest then type and reset index now we don't need this index and then just mention drop equal to true and then do the same thing for DF imputed as well just type in reset index and then type and drop equal to true then as copilot suggested you have to mention access to be one because we're combining it horizontally we're getting all of the columns back to our initial data set all right so let's just click on run all all right we get an error saying that end neighbors is not an argument which we get to mention I guess they have updated that but it just works on the nearest neighbors and traditionally we should be mentioning the number of neighbors but for this game is not working so let me just remove that and try this time it should hopefully work all right now it worked so if you go to this command over here and run this Cod cell once again now this time we should see all of the nonone values vanished except for the one and two column that is the destination and the home planet and the F this none values it's just basically the same we're going to just fill it with u just like we have done for the deck and the side column let's just get to the Cod cell then type in DF of home planet just update it will be instead of unknown let's just have it to be U and then do the same thing for Destination as well fill it with you and then run this codo now this should deal with all of the none values instead of every single column so when you run this this should give us all zeros this is what is the initial step of data cleaning and if you've done this it's really well done and as I said earlier what I recommend you to do is just look at all of these columns try to identify what value it provides to the final output and then try to fill the N values accordingly but then I've used Canon imputer it's just much faster and efficient way to do so and it's actually much reliable as well so after we have done this now we can get to feature engineering which is just basically creating new features for our model to better understand the data now for this again let's just get back to for this again let's just get back to our data over here and basically I just want you to look over here all of these five columns just relate to the amount spent by the passenger to get the luxury amenities all right so what we could do is these columns are just numbers of the passengers in terms of money so what we could do is just try to add up all of this and create a new column saying this is the total amount spent by the customer and we can also find the average of it and the way I've done this is first defining a list with all of these column names these are the amount columns and now I've created a new column DF amount spent will be first give the Bill calls and then use sum function AIS equal to one now this will calculate the entire values amount and then give us a sum back and I'm also going to find the standard deviation now that also gives us good value in terms of providing it to the model so what I can do is just use the standard deviation function and get the standard deviation for all five these columns and then I'm also going to find the mean as said earlier now these three have been created solely based upon this Bill calls all right and then I'm also going to create two new columns based upon trying to identify the correlation of each of these columns with the transported column now if that doesn't make sense then what I recommend you to do is use this function actually let's just go over here ef. core and then when you do this it's going to give you a heat map kind of numbers but then it's not going to be in the map format it gives us an error oops I forgot to mention one thing so as we filled all of these none values now we are still not left with numbers for these two columns as we look at the data frame head let's just go over here and type it out now as we see this destination column is still object and this home planet is still object again passenger ID is not something that you're worried about so what I can do is again these are categories of values so let me just use the value counts function once again to show you as we go to the home planet then run this we have got four different categories now what if I create creates four new columns out of this and this is called the one hot end coding now if I just show you for the destination as well destination then run this there is again four different categories so what I'm going to do is convert this destination column which is turned as one which has got four categories and we going to convert that into four new columns and the way I'm going to do this is let's just get to this Cod so and then let's just say I'm going to create a category calls list and this will be left with home it and destination just like that and then now going to use a for Loop to Loop over that for in category calls we just use the copal a statement and explain it our updated DF will be we're going to concatenate the initial DF and then what this dummies function does actually let me just take this alone separately when you do this now this will create the thing that I mentioned earlier it's just going to convert all of those categories into new columns so when you just try to do that let me just actually print all right now as you see there is a new column created a new column and then a new column and then a new column and this is for the destination and then this is for the home planet now look at the prefix is going to be the home planet itself along with the category name and this is what this pd. dummies function will be doing and what we're doing here is all of these new columns are being put onto a main data frame with the suffix of all these four new columns will be added to our main data frame using the BF concat function hopefully that makes sense I'm just going to remove this and then take off the do string like that and then also make sure to mention one because we're going to be combining that horizontally all right so now when you're on this this should combine four new columns from home planet and for new columns from destination to our main data frame so let's just on this all right not this one yep so as we get over here it says that the code is run successfully let's just try to give thef head now I feel like our entire data set should only contain numbers except for these two columns because we don't need these two columns anymore because we have created four new colums for each of those two so what we could do is just drop piece two and let's just get over here and since we have it already def