Insurance Premium Prediction - Machine Learning Python Project

Key Takeaways

what is going on guys welcome back in this video today we're going to go through a full machine learning project and we're going to try to predict the health insurance premium given certain factors like age whether someone is a smoker and more so let us get right into it not [Music] a all right so our goal today is to build a machine learning model that predicts the health insurance premium given certain information like the age of a person whether they're a smoker or not the body mass index and stuff like this and we're going to get this data from the US Health Insurance data set on kaggle you're going to find a link to it in the description down below and here we have the following features we only have six features in one target variable we have the H of a person the sex of a person the body mass index of a person uh the number of dependents or children and uh whether someone is a smoke or not and the region they live in and then we have this target variable charges which is basically the insurance premium so this is a classic regression task and I would say that this project is quite beginner friendly so if you are new to uh machine learning if you're just getting started and trying to you know work through a practical project I think this is a good start because it doesn't have too complicated features the pre-processing is going to be quite straightforward and simple uh and we're going to go through the full process from exploration to pre-processing to training to evaluating and then to also hyperparameter tuning um and evalua waiting again so yeah go to the link in the description down below you're going to find this data set download it as a zip file for example or load it with kagle Hub if you want to and then we're going to work in a jupyter notebook I prefer working in a jupyter notebook when it comes to projects uh like this because we can run individual cells we don't have to run all the code uh all the time so I recommend working in a jupit to notebook for this and we're going to start simply by loading the data set but before we can do that we need to install the packages uh that are necessary for this project today so if you don't have them open up a command line and type pip or pip 3 install and then we're going to need numpy we're going to need pandas we're going to need matte plot lip we're going to need uh scikit-learn and we're going to need caborn so these are the packages that we're going to use today make sure you have all of them installed and we're going to start by just importing panda s PD and we're going to take a look at the data set this is always what you want to do first you want to load the data by just saying read CSV and uh you want to just take a look at it so that you can see what it looks like and in this case we already saw it in kagle we have some uh basic features here there are not a lot and we have the target variable so this is a number this is not a class this is not a label so this is a regression task we're going to try to predict this value as accurately as possible and you can see that some of the features that we have here can be used right away for example the H or the number of dependents or children here can be used easily also the BMI is a numeric value but then we have things like female male so the sex which is a binary feature in this case um and we have uh things like a smoker which is also binary feature and we have region which is a categorical feature because we have more than two values if you're not sure if something is a categorical or a binary feature you can just go ahead and say um give me the feature for example smoker and then look at the value counts so call the value counts uh method and you're going to see we have no and yes nothing else uh with the sex as well if I go and say value counts you're going to see we have male female um you know you could have different values like uh not sure or undefined or uh rather not specify or something like this you could have different values and then it would make this a categorical feature uh if we look at the region we're going to see that we have four different values Southeast Southwest Northwest and Northeast so we have these uh four different regions this makes it a categorical feature and depending on what kind of feature it is we need to process it in a different way or pre-process it uh in a different way so in this case here we're going to just turn this here into zero and one we're going going to turn this here into zero and one and we're going to turn this here into multiple features we're going to SoCal we're going to use the so-called uh one hot encoding the basic idea of one hot encoding is that you take the categories and you turn them into features so instead of having a feature region that has the value Southeast South Southwest and so on we're going to turn them into four different features Southeast Southwest and so on and they're going to be either true or false so we're going to turn them into binary features uh this makes sense if the feature that we have here is not on a scale for example if you had something like um if you had a feature in a different data set which is education level for example and you have certain things like uh let's say um high school and then maybe you have bachelor's degree or something like this and then you have maybe master's degree and then maybe you have PhD in this case it would make sense to turn this into a a numerical feature because you have a scale you have something like 1 2 3 4 where actually a higher number means a higher education level but in the case of a region it doesn't make a lot of sense because calling Southeast one and calling Northeast 4 wouldn't make a lot of sense because they're not on a scale they are geographical features in the same way that it doesn't make sense to uh I don't know call if someone has a favorite programming language it doesn't make sense to represent this as a number on a scale because maybe the number one and four are more similar than the number one and two so in this case we're going to encode this as um we're going to one heart encode this feature so let's start with the sex and the smoker because these are quite simple what we're going to do in this case is we're just going to make them binary so we're going to say DF uh sex is going to be equal to apply Ling the following function to it so we're going to apply the Lambda expression for a given input X we're going to say one if x is equal to male and otherwise zero and for the smoker we're going to say the same [Music] thing we're going to say apply Lambda X 1 if x is equal to yes else zero if you don't know what Lambda expressions are basically you get an input X and depending on the input X you're returning something in this case one if x is male otherwise zero one if x is yes otherwise zero um and this is then applied this function is then applied to every single instance so this is going to turn into one this is going to turn to zero and so on we can see what this looks like by just printing the result and now you can see I have zero 0 1 and 1 Z here instead of uh text value so we can use this right away now with machine learning models now the region needs to be one heart encoded and for this we're going to use the get dummies method from Panda so we're going to say PD get dummies and I can pass DF region here and you're going to see what this looks like now I have four features Northeast northwest southeast Southwest West and they are binary so now each of these features uh is either true or false which means that per row only one is going to be true because if the value was Southwest before now I have true in the southwest column and false in all the other columns so I turned this into four binary features now to make this uh more compatible I'm going to say dtype is equal to uh can I just say int yeah so this is how I can do this I can say int and now I have uh zeros and one here uh ones here now what I want to do now is I want to join this with the already existing data frame so I'm going to say DF join um this stuff and now you can see I have these columns in addition to all the other columns and of course what I want to do now is I want to drop the region column so I want to say drop uh region axis equals 1 and this is now my data frame and I just have to apply this now and I can uh look at all these features they're all numeric or numerical uh what I can do now easily is I can plot histograms I can say DF hist and I can get a histogram for every single feature if I want to and of course I can do something like figure size is equal to 15 10 to make this larger um but I don't see anything too bad here so I don't see anything that is like SK very skewed or very problematic so we can just keep the data as it is we don't have to necessarily drop anything another thing that we can look at is we can call the info method to see if we have some missing values but here we can see we have uh 1,338 non null values and we don't have any missing values here for any feature which is perfect we don't need to drop or fill any Nan values we don't need to drop or fill any rows or features uh which means we can just proceed here another interesting thing would be to look at correlations do we have in the data already some correlation with a Target variable do we have correlation uh among features so do we have for example one feature that almost perfectly predicts another feature that would be interesting and maybe we should um remove it or take care of that uh so what we're going to do here is we're going to say import matplot matplot lip. pip plot splt import caborn as CNS or SNS uh and then we're going to say SNS heat map DF and we're going to call the correlation function so DF core if you don't know what DF core does it basically just gives you the correlation in a sort of table structure here uh but we can visualize this with Colors by using a heat map so DF correlation annotation is true and then we're going to say uh the color map is going to be cool warm I also want to specify the range to be from -1 to positive one for the correlation here um and before that I want to do PLT figure and let's use here a figure size of 108 and this produces a heat map like this so now I can see here if something is very red it means that it has a positive correlation if something is very blue it means it has a negative correlation uh obviously the diagonal is going to be one because it's always the same feature so H and H are perfectly correlated obviously uh but besides that the only strong correlation that we see is this here 0.79 which is quite high for the charges being connected to Smoker so it seems right away that smoker um is highly correlated with paying more if you smoke you probably pay more in terms of uh insurance but Cor coration isn't always what's the most uh important for the prediction what we're going to do here is we're going to train a random forest model which means that we can look at the feature importances and they might actually differ from the correlations so what the model considers to be most important for making a decision is not the same as the feature that is the highest that has the highest correlation with the uh with the charges so that's quite interesting uh we also see that there is some correlation here between Ag and charges we see that we have some correlation interestingly here between Southeast and BMI seems like in the Southeast people have more weight not sure the BMI I mean as far as I know the higher the value the worse your uh weight I'm not sure or your obesity or whatever um but yeah there's some correlation here there's also some negative correlation here with other areas and obviously this here is blue because if you are in the Northeast region this is going to be negatively correlated with all the other regions so exactly 0.3 is what we would expect here because they're mutually exclusive so that doesn't tell us much but interesting is that we have a strong correlation here between being a smoker and paying more so that's already something to take a look at um and what we're going to do now is we're going to say uh let's train a simple random Forest regressor on this so we're going to say from SK learn. model selection first of all we're going to import a train test split function this is going to allow us to take our X data our input data and our Y data and split it into a training set and a testing Set uh because of course usually what we want to do is we want to train a model on one set we want to evaluate it on another set to make sure that uh we're evaluating the performance on data it has never seen before and what we want to use here is from sklearn on soft we want to use random forest regressor and uh then we also might want to take a look at different metrics so from SK learn metrics we want to import the root mean squared error the RMS e and also the mean actually let's go with the mean absolute error and then we're going to say x is equal to everything in a data frame except for the charges except for the prediction so we're going to drop that and Y is going to be equal to only the charges so we're going to select this and then we're going to say xtrain X test y train y test is going to be equal to a train test split of X and Y with a test size of 20% so 0.2 which means 20% of our data set will be used for evaluation and 80% for training so I can uh execute this and now we can just say the model I want to train is a random Forest regressor I want to use all the CPU cores I have so n jobs is going to be equal to -1 and then I'm going to say model fit on xtrain and y train this is going to be done quite quickly because we don't have a lot of data we only have 1,300 38 rows or actually for the training even less than that and now we can see okay how well does this model perform out of the box so if I just say model. score X test y test I get 0.78 is an R2 uh scored here so this is something between zero and one so one would be the best quote unquote but I don't like this measure often times for regression task because it's not like accuracy in classification tasks uh it is not the Perfect Score so I like to also look at the rmse and the Mae so the root mean squ error and the mean absolute error to get a feeling of how off we are in terms of an actual value so we can say here that I want to make predictions with a model model pred predict the test data and I want to see how these predictions differ from the actual results so I want to say rmse is going to be equal to the root mean squared error if we consider the test data and the prediction data and that is going to be in this case 5,590 this is going to be uh in the same unit as our Target variable so what we're going to do here to see how bad this is or how good this is is we're going to say give me from the data frame or yeah give me from the uh should we use the whole data frame or just the test um let's just go with the whole data frame here to get a feeling of the range so let's say we have data frame dot uh charges and I want to have the standard deviation and in this case here the standard deviation is is 12,000 so that is not uh very high even though it's 5 ,000 and 5,000 is not a small number considering that the standard deviation is quite high that is not a huge value we can also just look at the test data set so we can say y test standard deviation and here we also have a very high value uh we can also take a look at the median so we can say charges median and that is also a large number so that is not horrible uh let's look at the mean absolute error so if I say Mae mean absolute error y prediction and Y test I get a score of 3,000 so you know is this good or bad depends but we are going to see if we can improve that metric here so maybe before we move on to the hyperparameter tuning I would also like to show this visually so what we're going to do is we're going to display um the predictions against the charges so against the actual truth um and we're also going to look at the feature importances so what I'm going to do here is I'm going to say import numpy SNP and I'm going to say that I want to scatter so PLT scatter the predictions or actually the test data here is on the x- axis and the predictions on the y- axis and um then I also want to have the identity function as a line or actually uh is it the identity function yeah the identity function is a line just so I have the ideal line and then I can see my actual data spreading around it so I'm going to say here PLT uh plot and I'm going to use the Lin space function to say plot from zero to Max of Y test on the x-axis and plot from zero to the maximum of the predictions on the um on the y axis and then we're going to to say PLT dox label is going to be equal to is going to be equal to the actual charges and then the Y label is going to be equal to the prediction and then we're going to say PLT title is going to be prediction versus truth and then we're going to just display this now one thing that I thought about this why I Had a Brain lag for a second is I think that it might make more sense to just use test and test um because I'm not sure that we're actually this is not the identity function is it because we're using different different values but I'm not sure about this um one thing is definitely we have to turn this into a red color so color is equal to Red okay now we should change that to also be y test because otherwise we're not having the identity function it won't make a huge difference visually but we should do this for the sake of correctness and what you can see here here now is the identity function this is the ideal line which means that if all data points were on this line it would be a perfect prediction because this is where the charges match the prediction so that is where you want your data points to be now you can see that we do have some points that are not very well classified or well predicted here which means that these are the actual values and our predictions are actually down here so they are for example in this case here it's something around 20,000 something and we predict it's not even uh 5,000 so this is a pretty bad prediction we have some data points down here that are not um predicted very well but most points you can see are close to the line so we have uh pretty decent prediction for most points not for all of them and um our goal now is to improve this our goal now is to see can we get better at this but before we do that I want to also look at the feature importances so what does the random Forest regressor considered to be the most important um aspects of the data in order to predict the charges so we're going to say here feature importances is going to be equal to and what we want to do now is we want to zip together the model feature names with the model feature importances this is something that you can do with um that you can do with random Forest models you can just get the feature importances and what we're going to do then is we're going to sort them so we're going to say sort it we're going to say that the key for sorting is going to be the important so the second part so we're going to say Lambda X X1 and reverse is going to be equal to true to have it in descending order uh feature importances does not exist yeah because it's called feature importances underscore and uh now we're going to visualize this we're going to say PLT figure figure size is going to be equal to let's say 106 PLT bar we want to have a bar plot we're going to say here x0 so the feature name for X in feature importances and then X1 for X in feature importances and then we want to add the title feature importances and then we get this we get the different features and we can see that the smoker seems to be the most uh relevant thing here when predicting uh when predicting the charges this aligns also with the correlation so in this case there is no mismatch here it means that the smoker variable is actually the most important thing we can see the BMI here is also quite important the H uh we can actually look at the correlations here what is the BMI correlation with the charges 0.2 H is actually 0.3 so H is more correlated with the charges but in the feature importances it seems to be less important than the BMI maybe because it's used in combination here with other features but it seems to be the case that the smoker variable is the most important one the location is basically irrelevant the gender as well and the BMI and AG seem to be quite important so let's try to do some hyperparameter tuning now hyperparameter tuning means that we're adjusting things about our model that could um influence how it works so for example I'm going to open up here the documentation random Forest SK [Music] learn uh actually regressor so here I have the random forist regressor and here we can see what kind of parameters we can specify about the model and what they mean so for example we can uh Define the number of estimators now a random Forest is a collection of decision trees so you can specify how many decision trees you want to use and um the default is 100 so you have 100 decision trees which together make a decision if you want to decrease or increase that you change the N estimators variable or parameter here uh we also have max depth which means means that how deep can the tree go um so how granular can it get we also have Min sample split which means how many samples are required to make a split in a note and also Min samples leave which is the number of samples required to be at a leaf note so when can we um yeah when do we allow for a leaf note um to happen and what we can do now is we can adjust default values to see if we can maybe end up with a more um with a with a model that performs better so what we're going to do for this is we're going to use something called grid search so I'm going to say here from SK learn. model selection import grid search cross validation so grid search CV and I'm going to define a parameter grid so I'm going to say param grid is going to be a dictionary and for specific variables for specific parameters hyperparameters uh I'm going to to specify certain values we want to try so I'm going to say for example the max depth which I think was none by default I'm going to also provide none as an option here so try the default value but also try the value two also try the value five and um then I also want to try for the Min samples split I want to try the values what was the default again I think the default was two so let's try uh two let's try four 6 and 8 and then I want to also try Min samples leave which I think was one by default so let's try 1 2 four and six and the idea of a grid search now is that I'm going to try all the configurations I'm going to try to train a tree with no max depth with two men sample split with two Min samples Lea then I'm going to try one with no max depth uh two here but two here this time then none two 4 none 2 six then none4 1 none 42 I'm going to try all the combinations here uh and I'm going to just decide or I'm going to determine based on Cross validation which of these configurations performs best considering cross validation now cross validation means instead of splitting into a training and testing set I uh split up the data in so-called folds so I basically have um fold one fold two fold three for example if I have a three-fold CV a three-fold cross validation uh we're going to do a fivefold which means I have five folds here and these are just portions of the data and what I'm going to do is I'm going to train on four of them and evaluate on the fifth one then I'm going to train on the other four I'm going to evaluate on the other fifth one I'm going to train then maybe on these and this and then I'm going to evaluate on four I'm going to do that for all the combinations here with all the different configurations which means that what I'm going to end up with here is I'm going to have 3 * 4 * 4 * 5 different models or different uh trainings and evaluations that I'm going to do you can calculate this this is 3 * 4 * 4 * 5 that is 240 different models that I have to train here or actually it's not different models it's uh that many different models and this many different training sessions so yeah depending on how complex your data set is how complex your model is you want to keep this down but in this case it will work because we can train this quite quickly so what I'm going to do here is I'm going to say model is equal to random forest regressor and jobs is going to stay -1 and then I'm going to say grid search is equal to grid search cross validation model the parameter grid is going to be equal to the parameter grid CV is going to be equal to five because we do a fivefold cross validation and then I can just say grid search fit X train y train and now this is going to take a while and once it's done we're going to get our best estimator all right so the training is done and what we can do now is we can say grid search Dot and we can see a bunch of things here best param will give us the best parameters as you can see here it seems like the max depth of five the Min sample Leaf of four and the Min sample split of six is the optimal setting now one thing that you could do here is if you notice that you have some values on the boundaries you can try more values so for example if I see that max depth 5 is the best maybe it would be even better to try seven or eight or nine so you can try to see how far you can go with that uh if you have something like four and you have values around it then four is probably a good value so I would keep that um with a Min samples sleeve you could try something else so you could try again since this is a boundary you could try eight 10 and so on but for now we're just going to go with this one and in order to get the model with these parameters the trained model with the parameters we can say model is equal to grid search best estimator so this is going to give us the best estimator which is this random forest regressor with these settings and also already trained so we can see now easily by just saying model score if this outperforms the previous model on the test data it has never seen before this is also very important you don't want to use your test data for hyperparameter tuning you never use your test data to decide how to design a model because then you cannot evaluate it on other data so what you do is you do either cross validation as I explained before or you split your training data into a training and validation set the idea being you want to have one set for training the model you want to have one set either the validation set or you do it with cross validation um but you want to have certain data configurations to see which parameters work best which hyperparameters work best but you don't want to have this be biased by the actual evaluation data because it's important that once you decide on something like here the hyperparameters they were not decided by looking at a test data so now I can still evaluate the model on the test data and it's not biased so I I don't have any information about the test data and here now I get 0.81 before we got 0.78 so that's an improvement here even though it's not tuned on the test data so let us see if we get a better rmse as well we're going to say here y predictions are going to be equal to model predict X test and then we're going to say rmse is going to be equal to root mean squared error y test y prediction and here I get 5,200 what did I get before I got 5,500 so this is lower this is good and now let's look at the mean absolute error so Mae is going to be equal to mean absolute error y test uh y test and Y [Music] prediction here we get 2,700 before we got 3,000 so this is an improvement let us also do the visualization again I'm going to copy this I'm going to paste this down here and you can see not too much it still looks fine I don't think that we're going to see a huge change visually I don't think that we're going to be able to get these points here um we would have to do some more analysis on how they happen maybe we would use some fancy technique here but we're not going to do this for this video now so we're going to just accept that we're not going to uh yeah catch these points we're not going to be able to make super awesome predictions for them um besides that maybe another thing that we could try I don't think that we necessarily will end up having a better performance but I'm going to copy all of this here I'm going to copy all these cells and paste them down below I'm going to try to train here the grid search with a different metric I'm going to say that the scoring is going to be done on based on the negative mean absolute error which means that uh we're going to optimize for this metric so let's see if we're able to by doing this get at least a better Mee score maybe this is going to negatively influence these two scores but maybe we can get a better score for this metric here if you're interested in that if you want to optimize for this so we're going to do the exact same process but the evaluation method is going to be a different one one so now let's see what the best parameters are we have different parameters uh now the score is lower quite a bit lower the rmse is higher which is also bad but let's see what about the Mae even the Mae is higher so this could also just be a problem with a random um iteration here so maybe if I run this again I'm going to get a different result but this was Now worse than before so this was not an improvement in any way the question is now did this happen just because of Randomness or did this did this happen because I specified a different scoring method it's hard to know we can just try again and see what happens in this case here I get again different parameters and I get again uh different scores but not better scores so this could also just be due to Randomness there is some some chance in here especially because we don't have a lot of data but yeah you can play around with that and this is the full pipeline here this is the full machine learning process we get the data we look at it we explore it we look at histograms we look at correlations we pre-process the data we start training a model we look at the model at the performance we look at the feature importances to understand what determines the most how the insurance charge is going to be um and then we just try to optimize the model using hyperparameter tuning and yeah then we end up with the quote unquote best model by doing that and we evaluate it on the test data The Next Step would then be deployment I have videos on this on this channel as well but that is basically how you do that so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting a like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this Channel and hit the notification Bell to not miss a single future video for free other than that thank you much for watching see you on the next video and bye for